System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving information input by a user; means for generating a customized conversation simulator based on the information; means for collecting conversation data with the conversation simulator; and means for analyzing the conversation data and non-verbal data and providing feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional communication skill improvement tools often require a real conversation partner or require expensive use of specialized instructors and classrooms. Another problem is that users are often embarrassed about meeting others face-to-face, preventing effective conversation practice. There is a need for appropriate practice tools that can solve these problems and enable more people to effectively improve their communication skills. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including a means for receiving information input by a user, a means for generating a customized conversation simulator based on the information, a means for collecting conversation data with the conversation simulator, and a means for analyzing the conversation data and nonverbal data and providing feedback. This system allows a user to practice with a conversation simulator customized based on a specific situation and the attributes of the other person, and also provides comprehensive feedback by analyzing the nonverbal data. This allows users to improve their communication skills effectively and economically.

[0006] A "user" is someone who wants to use the system to improve their communication skills.

[0007] "Information" is data entered by the user regarding a specific situation or the attributes of the other party.

[0008] A "conversation simulator" is the creation of a virtual conversation partner that is customized based on information entered by the user.

[0009] "Conversation data" refers to the content of the conversation that took place between the user and the conversation simulator.

[0010] "Nonverbal data" refers to data that includes non-verbal communication such as a user's facial expressions and tone of voice.

[0011] "Feedback" refers to advice or evaluation provided to the user based on analyzed speech and non-verbal data.

[0012] A "system" is a set of devices or programs that receives information input by a user, generates a conversation simulator based on that information, analyzes conversational and non-verbal data, and provides feedback. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] System Overview

[0035] This invention provides a conversation practice tool for users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and non-verbal data, and provides feedback. This invention uses three entities: a server, a terminal, and a user.

[0036] Server Roles

[0037] The server is primarily responsible for performing the following functions:

[0038] 1. Receiving and analyzing information:

[0039] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[0040] 2. Creating a conversation simulator:

[0041] Generative AI (e.g., a GPT model) is used to generate a customized conversation simulator based on the received information. The received information is used as a prompt to create an appropriate dialogue model.

[0042] 3. Model Transfer:

[0043] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[0044] 4. Data analysis and feedback generation:

[0045] The system analyzes conversational and nonverbal data sent from the device and generates feedback to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[0046] Device Role

[0047] The terminal is responsible for the interaction with the user and the communication with the server:

[0048] 1. Information input interface:

[0049] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[0050] 2. Submit your request:

[0051] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[0052] 3. Interfacing with the conversation simulator:

[0053] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[0054] 4. Nonverbal Data Collection:

[0055] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[0056] 5. Viewing Feedback:

[0057] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[0058] User Roles

[0059] The user typically performs the following actions:

[0060] 1. Enter your information:

[0061] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[0062] 2. Execute the conversation:

[0063] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[0064] 3. Review your feedback:

[0065] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[0066] Specific examples

[0067] For example:

[0068] 1. User Input:

[0069] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[0070] 2. Server Processing:

[0071] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0072] 3. Conversation session:

[0073] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[0074] 4. Data Collection and Analysis:

[0075] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to a server, which analyzes them to generate feedback and send it back to the device.

[0076] 5. Viewing Feedback:

[0077] The device displays the feedback received from the server to the user, such as advice like, "Your answers are accurate, but your facial expression is stiff. You'll make a better impression if you relax."

[0078] In this way, a system is realized that allows users to efficiently improve their communication skills.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[0082] Step 2:

[0083] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[0084] Step 3:

[0085] The server receives the request sent from the terminal and analyzes the situation and attribute information.

[0086] Step 4:

[0087] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[0088] Step 5:

[0089] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[0090] Step 6:

[0091] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[0092] Step 7:

[0093] The user interacts with the generated conversation simulator via a terminal using text or voice.

[0094] Step 8:

[0095] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[0096] Step 9:

[0097] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[0098] Step 10:

[0099] The server analyzes the received speech and non-verbal data and generates feedback regarding the user's communication.

[0100] Step 11:

[0101] The server sends the generated feedback to the device, which displays it in the user interface, allowing the user to receive specific advice on how to improve their communication skills.

[0102] In this way, the system provides an environment in which users can actually attempt a dialogue, evaluate their performance from multiple angles, and provide feedback, allowing users to efficiently improve their communication skills.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Communication skills are extremely important in modern society, but there are limited tools available to effectively acquire and improve them. It's particularly difficult to objectively identify one's weaknesses and areas for improvement, making efficient training difficult. Furthermore, existing tools lack a system that comprehensively analyzes a variety of data, such as the content of conversations, facial expressions, and tone of voice, and provides feedback, limiting the ability to fully improve users' communication skills.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for receiving information input by a user, means for generating prompts using a generative AI model and generating a customized conversation simulator, means for collecting conversation data and nonverbal data, means for analyzing the conversation data and nonverbal data and generating feedback, and means for displaying the feedback to the user. This allows the user to have their communication skills comprehensively evaluated and be given specific indications of areas for improvement, thereby enabling them to efficiently improve their skills.

[0108] "Means for receiving information entered by the user" refers to the function of the terminal providing an interface for the user to input information such as the situation and the attributes of the other party, and transmitting that information to the server.

[0109] "Means for generating prompts using a generative AI model and generating a customized conversation simulator" refers to a function that sends prompts to a generative AI model based on user information received by the server, thereby generating a conversation simulator that is suitable for a specific situation or the attributes of the other party.

[0110] "Means for collecting conversational data and non-verbal data" refers to the function of the device to record and save in real time using a camera or microphone voice data generated during a conversation with a user, as well as non-verbal data such as the user's facial expressions and tone of voice.

[0111] "Means for analyzing conversational data and non-verbal data and generating feedback" refers to a function in which the server uses analytical technology to evaluate collected conversational data and non-verbal data and generate specific advice and areas for improvement to improve the user's communication skills.

[0112] "Means for displaying feedback to the user" refers to a function that allows the terminal to display feedback information sent from the server on the user interface, allowing the user to check it and use it to improve their own communication skills.

[0113] A "situation" refers to a specific situation or scene that a user inputs to the conversation simulator, and includes specific situations such as an interview or a presentation.

[0114] "Partner attributes" refer to the characteristics and personality of the conversation partner that the user envisions in the conversation simulator, and include, for example, age, gender, and personality tendencies.

[0115] A "generative AI model" is an artificial intelligence model that provides generated text or dialogue based on input prompts, and includes, for example, a natural language processing model.

[0116] A "prompt" is a string of characters or data given to a generative AI model as an instruction or input, and is basic information used to determine the content of the generated conversation simulator.

[0117] A "conversation simulator" is a program or system that simulates dialogue based on specific situations and the attributes of the other party, generated by a generative AI model.

[0118] This invention is a conversation practice tool for users to improve their communication skills, and is a system that generates an AI conversation simulator based on information input by the user, collects and analyzes conversation data and non-verbal data, and provides feedback, thereby enabling users to improve their communication skills efficiently.

[0119] Server Roles

[0120] The server uses the following hardware and software:

[0121] Hardware: High-performance processors, memory, and storage devices

[0122] Software: Generative AI models (e.g., GPT-3 (registered trademark))

[0123] The server first receives information about the situation and the other party's attributes sent from the device. For example, it receives information such as "Situation: Interview, Attributes: Middle-aged male, Strict personality." Next, it references the generative AI model based on the received information and generates an appropriate prompt. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality." This prompt is sent to the generative AI model, which generates a customized conversation simulator. The server then transfers the generated conversation simulator to the device.

[0124] The server also collects and analyzes speech and non-verbal data sent from the device, using voice and face recognition technology, and generates feedback to be provided to the user based on the analysis results and sends it to the device.

[0125] Device Role

[0126] The terminal is responsible for the interaction with the user and the communication with the server:

[0127] Hardware: Camera, microphone, display, input devices (keyboard and touchscreen)

[0128] Software: User Interface Application

[0129] The terminal provides an interface where the user can input information about the situation and the other person's attributes. For example, using text boxes and drop-down menus, the user can select "interview" and input "middle-aged male, strict personality." The information entered by the user is sent to the server, which then requests the generation of a conversation simulator.

[0130] The device initiates a dialogue session with the user using the conversation simulator received from the server. It provides an interface for the user to converse via voice or text, and uses a camera and microphone to collect the user's facial expressions and tone of voice in real time. The collected data is sent to the server for analysis and feedback generation.

[0131] Once the feedback is sent from the server, the device displays it to the user, and specific advice is provided in the user interface, allowing the user to analyze their performance and take action to improve.

[0132] User Roles

[0133] Users will use the system to improve their communication skills by:

[0134] First, a user inputs a specific situation and the attributes of the other person through the device interface. Then, the user engages in a dialogue with the generated conversation simulator. For example, in an interview simulator, the user responds to questions from the interviewer via voice or text. After the session ends, the user can review the feedback provided by the server and use it to improve their communication skills.

[0135] Specific examples

[0136] The user starts up the device, selects an interview situation, and enters "middle-aged male, strict personality" as the attributes. The device sends this information to the server, which uses a generative AI model to generate a prompt sentence: "Situation: Interview, Attributes: Middle-aged male, strict personality," and a conversation simulator is created.

[0137] The user begins a conversation with the simulator displayed on the device. The simulator acts as an interviewer, asking questions, and the user answers. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server. The server analyzes this, generates feedback, and sends it back to the device. The device displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your expression is stiff. You would make a better impression if you relaxed" may be displayed.

[0138] Through these steps, users can effectively improve their communication skills. The system also provides users with objective feedback that can be used for self-evaluation.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] System program processing flow

[0141] Step 1:

[0142] The user uses the device interface to input the situation and the attributes of the other person. For example, they might select "interview" and input "middle-aged man, strict personality" as the attributes. This input is done using text boxes and drop-down menus. The input is the user's selection and input data. The output is the input information being saved on the device.

[0143] Step 2:

[0144] The terminal sends the situation and attribute information entered by the user to the server. This transmission is performed via an HTTP request. Specifically, the terminal sends the information "Situation: Interview, Attributes: Middle-aged male, Strict personality" to the server in JSON format. The input is the situation and attribute data from the user. The output is the server receiving this data.

[0145] Step 3:

[0146] The server generates a prompt for the generative AI model based on the received information and generates a conversation simulator. Using a generative AI model (e.g., GPT-3), it generates a prompt sentence such as "Situation: Interview, Attributes: Middle-aged male, Strict personality" and sends it to the AI ​​model. The input is the situation and attribute information sent by the user. The output is the generated conversation simulator.

[0147] Step 4:

[0148] The server transfers the generated conversation simulator to the device. This transfer is also performed via an HTTP request. The input is the generated conversation simulator. The output is the device receiving the conversation simulator. Specifically, the server sends the conversation simulator to the device in JSON format.

[0149] Step 5:

[0150] The terminal provides the received conversation simulator to the user and starts a conversation session. The user converses using voice or text. The terminal interface displays the dialogue content to the user and provides an input field. The input is the dialogue input between the generated conversation simulator and the user. The output is the user's dialogue history and dialogue content generated in real time.

[0151] Step 6:

[0152] The device collects conversational and non-verbal data. It uses a camera and microphone to record the user's facial expressions and tone of voice, and saves them as a log. The input is the user's facial expression data and voice data. The output is the collected log data.

[0153] Step 7:

[0154] The device sends the collected data to the server. The data includes conversational content and nonverbal data and is sent to the server in JSON format. The input is the collected conversational data and nonverbal data. The output is that the server receives this data.

[0155] Step 8:

[0156] The server analyzes the received data, using natural language processing and facial recognition technologies to evaluate the content of the conversation and non-verbal expressions. The input is the collected conversation data and non-verbal data. The output is feedback as the analysis results.

[0157] Step 9:

[0158] The server generates feedback based on the analysis results and sends it to the device. The generated feedback is sent in JSON format. The input is the analysis results. The output is that the device receives the feedback.

[0159] Step 10:

[0160] The terminal displays the feedback received from the server on the user interface. The feedback includes an evaluation of the user's conversation content and advice on non-verbal expressions. Specific areas for improvement and advice are highlighted. The input is the feedback from the server. The output is the user being able to visually confirm the feedback.

[0161] This realizes a system that allows users to efficiently improve their communication skills.

[0162] (Application example 1)

[0163] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0164] In modern retail stores, sales associates are expected to consistently provide high-quality customer service. However, for new sales associates or those with uncertain skills, simulating actual customer interactions and improving their skills is a difficult task. In particular, because they are required to respond to diverse customer attributes and situations, real-time feedback is lacking. This can result in a decline in customer satisfaction, which can have a negative impact on the store's reputation and sales. There is a need for a system that can solve these issues and effectively improve sales associates' customer service skills.

[0165] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0166] In this invention, the server includes means for generating a customized conversation simulator based on a situation and attributes of the other party input by a user, interface means for collecting interaction data and nonverbal data with the conversation simulator, and means for analyzing the interaction data and nonverbal data and using the generative AI model to provide feedback, thereby enabling store clerks to simulate customer interaction situations in real time and improve their customer service skills based on the obtained feedback.

[0167] A "situation" is information that a user inputs to simulate a specific situation or scene.

[0168] "Partner attributes" is information that indicates the characteristics and personality of the person you are likely to be speaking with in a given situation.

[0169] A "customized conversation simulator" is a dialogue program that is generated based on the situation and attributes of the other party input by the user.

[0170] "Interaction data" refers to voice and text data exchanged between the user and the conversation simulator.

[0171] "Nonverbal data" refers to non-verbal information such as a user's facial expression, tone of voice, and posture.

[0172] A "generative AI model" is an artificial intelligence model that generates customized dialogue simulators based on user input information, or analyzes collected data and provides feedback.

[0173] "Customer service" refers to the customer service provided by retail store staff to actual customers.

[0174] "Feedback" is information for improvement or evaluation provided based on user interaction and non-verbal data.

[0175] This invention provides a system for improving the customer service skills of retail store staff. The system is composed of three main components: a server, a terminal, and a user.

[0176] Server Roles

[0177] The server is responsible for performing the following main functions:

[0178] 1. Receiving and analyzing information: The system receives information about the situation and the other party's attributes entered by the user on the terminal. For example, it receives the situation "serving an elderly person at a store" and the attributes "elderly woman, hard of hearing."

[0179] 2. Conversation simulator generation: A generative AI model (e.g., GPT-3 / 4 model) is used to generate a customized conversation simulator based on the received information. The AI ​​model uses the received information as a prompt to create an appropriate dialogue program.

[0180] 3. Analysis of interaction data and nonverbal data: Analyze the conversation data and nonverbal data sent from the device. Generate automatically generated feedback based on the analyzed data and send it to the device.

[0181] Device Role

[0182] The terminal is responsible for the interaction with the user and the communication with the server:

[0183] 1. Information input interface: Provide an interface for users to input information. Specifically, provide text boxes and multiple-choice input fields to allow users to enter the situation and the other person's attributes.

[0184] 2. Sending a request: The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[0185] 3. Interface with the conversation simulator: Start a dialogue session with the user using the generated conversation simulator received from the server. Provide an interface that allows the user to have a conversation via text or voice.

[0186] 4. Non-verbal data collection: A camera and microphone are used to collect the user's facial expressions and tone of voice in real time, which are then analyzed on the server.

[0187] 5. Displaying Feedback: The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[0188] User Roles

[0189] The user typically performs the following actions:

[0190] 1. Information input: A specific situation and the attributes of the other party are input through the terminal interface. For example, a situation involving dealing with an elderly person at a store, and the attributes of the elderly woman and her hearing impairment are input.

[0191] 2. Execution of conversation: Dialogue is carried out with the generated conversation simulator. At this time, the conversation is exchanged with the simulator via voice or text. For example, specific conversation content is confirmed as practice for dealing with elderly people.

[0192] 3. Check the feedback: Check the feedback provided by the server as an analysis result and improve your customer service skills based on it.

[0193] Specific examples

[0194] For example:

[0195] 1. User input: The user starts up the terminal and inputs the situation as "serving elderly people at a store" and the attributes as "elderly woman, hard of hearing."

[0196] 2. Server processing: Receives the information sent from the device, and generates a prompt for the generation AI based on the following: "Situation: Serving elderly customers at a store, Attributes: Elderly woman, hard of hearing," and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0197] 3. Conversation session: The user begins a conversation with the conversation simulator displayed on the terminal. The simulator plays the role of an elderly customer, prompting the user with questions and responses, and the user responds.

[0198] 4. Data collection and analysis: The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the conversation to the server. The server analyzes them and generates feedback, which is sent back to the device.

[0199] 5. Displaying feedback: The device displays the feedback received from the server to the user. For example, advice such as "You should speak more clearly. Also, it is important to treat the elderly with a kind expression" is displayed.

[0200] Example prompt sentence:

[0201] Situation: Assisting elderly customers in stores

[0202] Attributes: Elderly woman, hard of hearing

[0203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0204] Step 1:

[0205] The user inputs the situation and the other person's attributes.

[0206] The user inputs a specific situation (e.g., "serving an elderly person at a store") and the other person's attributes (e.g., "elderly woman, hard of hearing") through the terminal interface. The terminal then receives the input data and prepares it to be sent to the server.

[0207] Input: Situation and partner's attribute information

[0208] Output: The request data sent to the server

[0209] Step 2:

[0210] Sending a request from the device to the server

[0211] The terminal sends the information entered by the user to the server, which then starts generating a customized conversation simulator. Specifically, the terminal sends the situation and the attributes of the other party as prompts to the server.

[0212] Input: Request data

[0213] Output: Prompt data received by the server

[0214] Step 3:

[0215] The server generates a conversation simulator.

[0216] The server uses the received prompt data to generate a customized conversation simulator using a generative AI model (e.g., GPT-3 / 4). The AI ​​model generates a dialogue program based on the prompt data and returns the generated conversation simulator to the device.

[0217] Input: prompt data

[0218] Output: Generated conversation simulator

[0219] Step 4:

[0220] The terminal receives the conversation simulator and starts the conversation session.

[0221] The terminal starts a dialogue session with the user using the generated conversation simulator received from the server. The user converses with the user using voice or text. Specifically, the terminal displays the conversation simulator and the user responds.

[0222] Input: Generated conversation simulator

[0223] Output: Interaction data

[0224] Step 5:

[0225] Nonverbal data collection

[0226] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and sends this non-verbal data to a server after the conversation ends.

[0227] Input: User's facial expression and tone of voice data

[0228] Output: Collected nonverbal data

[0229] Step 6:

[0230] Transmission of interaction data and non-verbal data

[0231] The terminal transmits the collected interaction data and nonverbal data to the server, which receives and analyzes the data.

[0232] Input: Interaction data, nonverbal data

[0233] Output: Data used by the server for analysis

[0234] Step 7:

[0235] The server analyzes the data and generates feedback

[0236] The server analyzes the received interaction data and nonverbal data using generative AI models and data analysis tools. Based on the analysis results, it generates feedback for the user and sends it back to the device.

[0237] Input: Interaction data, nonverbal data

[0238] Output: Generated feedback

[0239] Step 8:

[0240] The device displays feedback

[0241] The device displays the feedback received from the server to the user, who can review the feedback and receive specific advice on how to improve their performance.

[0242] Input: Generated feedback

[0243] Output: Feedback that is displayed to the user

[0244] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0245] System Overview

[0246] This invention provides a conversation practice tool that allows users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Furthermore, by combining it with an emotion engine, it has the ability to recognize the user's emotions and provide feedback based on the results. This invention uses three entities: a server, a terminal, and a user.

[0247] Server Roles

[0248] The server is primarily responsible for performing the following functions:

[0249] 1. Receiving and analyzing information:

[0250] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[0251] 2. Creating a conversation simulator:

[0252] It uses generative AI (e.g., a GPT model) to generate a customized conversation simulator based on the information it receives, including prompts entered by the user.

[0253] 3. Model Transfer:

[0254] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[0255] 4. Data Analysis and Emotion Recognition:

[0256] The system analyzes conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[0257] 5. Feedback Generation:

[0258] Based on the analysis results and emotion recognition results, feedback is generated to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[0259] Device Role

[0260] The terminal is responsible for the interaction with the user and the communication with the server:

[0261] 1. Information input interface:

[0262] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[0263] 2. Submit your request:

[0264] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[0265] 3. Interfacing with the conversation simulator:

[0266] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[0267] 4. Nonverbal Data Collection:

[0268] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[0269] 5. Viewing Feedback:

[0270] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[0271] User Roles

[0272] The user typically performs the following actions:

[0273] 1. Enter your information:

[0274] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[0275] 2. Execute the conversation:

[0276] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[0277] 3. Review your feedback:

[0278] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[0279] Specific examples

[0280] For example:

[0281] 1. User Input:

[0282] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[0283] 2. Server Processing:

[0284] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0285] 3. Conversation session:

[0286] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[0287] 4. Data Collection and Analysis:

[0288] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[0289] 5. Providing Feedback:

[0290] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[0291] In this way, we realize a system that allows users to efficiently improve their communication skills. By using an emotion engine, we can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[0292] The processing flow will be explained below.

[0293] Step 1:

[0294] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[0295] Step 2:

[0296] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[0297] Step 3:

[0298] The server receives the request from the device and analyzes the situation and attribute information. For example, it extracts the situation "interview" and the attributes "middle-aged male, strict personality."

[0299] Step 4:

[0300] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[0301] Step 5:

[0302] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[0303] Step 6:

[0304] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[0305] Step 7:

[0306] The user interacts with the generated conversation simulator via a terminal, either through text or voice. For example, the simulator playing the role of an interviewer asks, "Please introduce yourself," and the user responds.

[0307] Step 8:

[0308] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[0309] Step 9:

[0310] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[0311] Step 10:

[0312] The server analyzes the received speech and non-verbal data and uses an emotion engine to recognize the user's emotions, for example, by analyzing facial expressions to determine whether the user is smiling.

[0313] Step 11:

[0314] The server analyzes the collected data to generate feedback about the user's emotions, such as providing specific advice like, "Your answers are accurate, but your facial expression seems stiff, so you should try to relax."

[0315] Step 12:

[0316] The server sends the generated feedback to the terminal, which displays it on the user interface, allowing the user to receive specific feedback to improve their performance.

[0317] Example 2

[0318] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] In modern society, improving communication skills is considered important, but there are limited opportunities to practice them effectively. It is particularly difficult to practice nonverbal communication (facial expressions, tone of voice, etc.). It is also difficult to objectively evaluate the effectiveness of practice and obtain specific feedback. To address these challenges, there is a need for a system that provides practical, individualized practice and feedback.

[0320] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0321] In this invention, the server includes means for receiving information input by the user, means for generating a customized conversation simulator using a generative model, means for transferring the simulator model to the user's terminal, means for collecting conversation data with the conversation simulator, and means for analyzing the conversation data and nonverbal data, performing emotion recognition, and providing feedback, thereby enabling the user to objectively evaluate the effects of their practice and identify specific areas for improvement.

[0322] "Information entered by the user" refers to data that the user enters to the system to specify a particular situation or the attributes of the other party.

[0323] A "generative model" is an algorithm or program that a pre-trained artificial intelligence uses to generate natural language based on specific input.

[0324] A "customized conversation simulator" is a conversation agent that is generated by a generative model based on information input by the user and that corresponds to a specific situation or the attributes of the other party.

[0325] The "means for transferring the simulator model to the user's terminal" is a mechanism for transmitting the generated conversation simulator from the server to the terminal used by the user.

[0326] "Conversation data with the conversation simulator" is text or voice data generated when the user converses with the conversation simulator.

[0327] "Nonverbal data" refers to data that includes non-verbal communication elements such as a user's facial expression and tone of voice.

[0328] "Emotion recognition" is a technology that analyzes speech and non-verbal data to identify a user's emotional state.

[0329] "Feedback" is information that is generated based on analyzed data and includes advice and suggestions for improving the user's communication skills.

[0330] This invention is a system that provides users with a conversation practice tool to improve their communication skills. The system generates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Three entities are involved: a server, a terminal, and a user.

[0331] System Overview

[0332] Server Roles

[0333] The server mainly performs the following functions:

[0334] 1. Receiving and analyzing information:

[0335] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[0336] 2. Creating a conversation simulator:

[0337] The server uses a generative AI model (e.g., GPT-3) to generate a customized conversation simulator based on the received information. The prompt includes the information entered by the user. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality."

[0338] 3. Model Transfer:

[0339] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[0340] 4. Data Analysis and Emotion Recognition:

[0341] The server analyzes the conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[0342] 5. Feedback Generation:

[0343] Based on the analysis results and emotion recognition results, feedback is generated to be provided to the user. The feedback is advice based on the content of the conversation and nonverbal expressions. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is provided.

[0344] Device Role

[0345] The terminal is responsible for the interaction with the user and the communication with the server.

[0346] 1. Information input interface:

[0347] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or an input field with multiple-choice options.

[0348] 2. Submit your request:

[0349] The terminal sends the information entered by the user to the server, which starts the generation of the conversation simulator.

[0350] 3. Interfacing with the conversation simulator:

[0351] The terminal uses the generated conversation simulator received from the server to initiate a dialogue session with the user, providing an interface through which the user can converse via text or voice.

[0352] 4. Nonverbal Data Collection:

[0353] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[0354] 5. Viewing Feedback:

[0355] The device displays the feedback received from the server in a user interface, allowing the user to receive specific advice on how to improve their performance.

[0356] User Roles

[0357] The user mainly performs the following actions:

[0358] 1. Enter your information:

[0359] The user inputs a specific situation and the attributes of the other person through the terminal interface. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[0360] 2. Execute the conversation:

[0361] The user interacts with the generated conversation simulator by speaking or texting, for example, answering hypothetical questions as practice for an interview.

[0362] 3. Review your feedback:

[0363] The user checks the feedback provided by the server as an analysis result and improves their own communication skills based on it.

[0364] Specific examples

[0365] Specific examples are shown below.

[0366] 1. User Input:

[0367] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[0368] 2. Server Processing:

[0369] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0370] 3. Conversation session:

[0371] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[0372] 4. Data Collection and Analysis:

[0373] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[0374] 5. Providing Feedback:

[0375] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[0376] Technology used and data processing

[0377] This system includes the following technologies and data processing:

[0378] Hardware: The device is equipped with a camera and microphone, which allows for real-time collection of the user's facial expressions and tone of voice.

[0379] Software: Generative AI models (e.g., GPT-3) are used to generate the conversation simulator, and an emotion engine is used to analyze nonverbal data and employ algorithms to recognize user emotions.

[0380] Data Processing: Techniques are used to collect and analyze speech and non-verbal data. The analyzed data provides important information for providing feedback to users.

[0381] Through this system, users can effectively improve their communication skills. By utilizing the emotion engine, users can receive more detailed and useful feedback.

[0382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0383] Step 1:

[0384] The terminal provides an interface for users to access the system. Here, users select a situation (e.g., an interview) and enter the other party's attributes (e.g., middle-aged man, strict personality). The input data is saved in text format and sent from the terminal to the server in JSON format.

[0385] Input: Situations and attributes entered by the user

[0386] Output: JSON format data (situations and attributes)

[0387] Step 2:

[0388] The server receives and analyzes the JSON-formatted data sent from the device. As a result of the analysis, situation and attribute information is extracted. Based on this information, the server generates a prompt for the generative AI model.

[0389] Input: JSON format data (situations and attributes)

[0390] Output: Prompt sentence for the generative AI model (e.g., "Situation: Interview, Attributes: Middle-aged male, Strict personality")

[0391] Step 3:

[0392] The server sends the prompt sentence to a generative AI model (e.g., GPT-3) to generate a customized conversation simulator, which is returned to the server as text data.

[0393] Input: Prompt sentence for the generative AI model

[0394] Output: Text data of the generated conversation simulator

[0395] Step 4:

[0396] The server transfers the generated conversation simulator to the terminal, which displays the conversation simulator on its interface and starts a dialogue with the user.

[0397] Input: Text data of the generated conversation simulator

[0398] Output: Interface showing the conversation simulator

[0399] Step 5:

[0400] The user interacts with the conversation simulator and inputs responses. The device collects the conversation data in real time. It also uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice.

[0401] Input: User responses, facial expressions, and tone of voice

[0402] Output: Speech and non-verbal data

[0403] Step 6:

[0404] The device transmits the collected speech and non-verbal data in real time to a server, which receives the data and performs data analysis using NLP technology and an emotion engine.

[0405] Input: Speech and non-verbal data

[0406] Output: Analysis results and emotion recognition results

[0407] Step 7:

[0408] The server generates feedback to provide to the user based on the analysis and emotion recognition results, including specific advice based on the conversation content and nonverbal expressions.

[0409] Input: Analysis results and emotion recognition results

[0410] Output: Generated feedback

[0411] Step 8:

[0412] The server transmits the generated feedback to the terminal, which displays the feedback on a user interface for the user to confirm.

[0413] Input: Generated feedback

[0414] Output: Feedback displayed in the user interface

[0415] (Application example 2)

[0416] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0417] To improve their communication skills, users need to be able to practice effectively and practically based on specific situations and the attributes of the people they are interacting with. However, conventional methods have faced challenges in providing practical training for specific situations and in obtaining effective feedback. This problem is particularly evident in training salespeople in brick-and-mortar stores to improve their customer service skills.

[0418] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information input by a user, means for generating a customized conversation simulator based on the information, means for collecting conversation data with the conversation simulator, means for analyzing the conversation data and nonverbal data and providing feedback, means for collecting the nonverbal data by face recognition means, and means for supporting the improvement of the user's customer interaction skills through the feedback. This enables the user to undergo practical training based on specific situations and the attributes of the other party and receive effective feedback.

[0419] A "user" is a person who utilizes the system to input information and receive feedback to improve their communication skills.

[0420] The "means for receiving information" is an interface or device for acquiring information input by the user about the situation and the attributes of the other party.

[0421] The "means for generating a conversation simulator" refers to a technique or device for creating a conversation simulator that corresponds to a specific situation or the attributes of the other person, based on information input by the user.

[0422] The "means for collecting conversation data" refers to a device or technology for recording the content of the conversation that takes place between the user and the generated conversation simulator.

[0423] "Nonverbal data" refers to data that includes information other than words, such as a user's facial expression or tone of voice.

[0424] A "means for providing feedback" is a device or technology that analyzes collected speech and non-verbal data and indicates areas for improvement in the user's communication skills.

[0425] "Facial recognition means" refers to technology or systems that use devices such as cameras to identify and analyze a user's facial expressions.

[0426] "Customer service skills" refers to the abilities and techniques that sales staff in physical stores and other locations have to respond appropriately to customers.

[0427] "Means for supporting improvement of users' customer service skills" are technologies and systems that include feedback and advice provided to enable users to provide better customer service.

[0428] This invention provides a training system for sales staff in brick-and-mortar stores to improve their customer service skills. The system operates using three entities: a user, a server, and a terminal. The roles and specific operations of each of these entities are described below.

[0429] Server Roles

[0430] The server performs the following functions:

[0431] 1. Receiving and analyzing information:

[0432] The server receives the situation and the other party's attributes sent from the terminal. For example, the server receives the situation "handling a complaint" and the attributes "elderly woman, angry."

[0433] 2. Creating a conversation simulator:

[0434] The server uses a generative AI model (e.g., OpenAI® GPT-3.5) to generate a customized conversation simulator based on the received information, including the prompts entered by the user.

[0435] Example prompt sentence:

[0436] Situation: Complaint handling, Attributes: Elderly woman, angry

[0437] 3. Model Transfer:

[0438] The server transfers the generated conversation simulator model to the terminal so that the user can use it.

[0439] 4. Data Analysis and Emotion Recognition:

[0440] The server analyzes the conversational data and non-verbal data sent from the device and recognizes the user's emotions using an emotion recognition engine, which includes facial expression analysis and tone of voice analysis.

[0441] 5. Feedback Generation:

[0442] The server generates feedback to provide to the user based on the analysis results and emotion recognition results, including advice based on the content of the conversation and nonverbal expressions.

[0443] Device Role

[0444] The terminal is responsible for the following functions:

[0445] 1. Information input interface:

[0446] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or a multiple-choice input field.

[0447] 2. Submit your request:

[0448] The terminal starts generating a conversation simulator by transmitting the information input by the user to the server.

[0449] 3. Interfacing with the conversation simulator:

[0450] The terminal initiates an interactive session with the user using the generated conversation simulator received from the server, and provides an interface through which the user can conduct a conversation by text or voice.

[0451] 4. Nonverbal Data Collection:

[0452] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[0453] 5. Viewing Feedback:

[0454] The terminal displays the feedback received from the server on a user interface, allowing the user to receive specific advice on how to improve their performance.

[0455] User Roles

[0456] The user performs the following actions:

[0457] 1. Enter your information:

[0458] Through the terminal interface, specific situations and the attributes of the other party are input. For example, the situation of handling a complaint and the expected attributes of the customer at that time (elderly woman, angry) can be input.

[0459] 2. Execute the conversation:

[0460] The user interacts with the generated conversation simulator by speaking or texting. For example, the user can practice how to respond to a complaint.

[0461] 3. Review your feedback:

[0462] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[0463] Specific examples

[0464] As a concrete example, consider the following flow:

[0465] 1. The user starts up the terminal, selects the situation "Handling a complaint," and enters "elderly woman, angry" as the attribute.

[0466] 2. The server receives the information sent from the device and sends a prompt to the generative AI model with the following content: "Situation: Handling a complaint, Attributes: Elderly woman, Angry," and generates a customized conversation simulator.

[0467] 3. The user starts a conversation on the terminal using the conversation simulator returned from the server. The simulator plays the role of a customer and makes a complaint, and the user responds.

[0468] 4. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them to the server along with the content of the conversation.

[0469] 5. The server analyzes the collected data and recognizes the user's emotions using an emotion recognition engine.

[0470] 6. The device receives the feedback generated by the server and displays specific advice to the user, such as, "The content of your complaint response is good, but it would be better if you spoke with a calmer expression."

[0471] In this way, the system allows users to efficiently improve their customer service skills in physical stores. By utilizing an emotion recognition engine, the system can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[0472] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0473] Step 1:

[0474] The user starts up the terminal and inputs information about the situation and the other party's attributes. For example, the situation "handling a complaint" and the attributes "elderly woman, angry" are entered into the text boxes. This starts the information input interface.

[0475] Input: situation (e.g., handling a complaint), other party's attributes (e.g., elderly woman, angry)

[0476] Output: The information entered

[0477] Step 2:

[0478] The terminal transmits the information input by the user to the server, where it arrives as a customized prompt.

[0479] Input: Information entered by the user

[0480] Output: Request to server

[0481] Step 3:

[0482] The server analyzes the received information and generates prompt sentences. The prompt sentences are used to generate a customized conversation simulator for a generative AI model (e.g., OpenAI GPT-3.5).

[0483] Input: Prompt sentence (e.g., Situation: Handling a complaint, Attributes: Elderly woman, Angry)

[0484] Data processing: Creating a conversation simulator using a generative AI model

[0485] Output: A customized conversation simulator

[0486] Step 4:

[0487] The server transfers the generated conversation simulator to the terminal, which then starts a dialogue session with the user based on the received simulator.

[0488] Input: A customized conversation simulator

[0489] Output: Transfer of conversation simulator to terminal

[0490] Step 5:

[0491] The user initiates a dialogue with a customized conversation simulator displayed on the terminal, and the user converses with the simulator using voice or text.

[0492] Input: A customized conversation simulator and the user's conversation content

[0493] Output: Conversation progress

[0494] Step 6:

[0495] The device uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice, which is then sent to a server in real time along with the content of the conversation.

[0496] Input: User's facial expression data, voice data

[0497] Data Processing: Collection and Transmission of Nonverbal Data

[0498] Output: Nonverbal and conversational data

[0499] Step 7:

[0500] The server analyzes the received non-verbal data and conversation data, and recognizes the user's emotions using an emotion recognition engine. It also analyzes facial expressions and voice using a face recognition means.

[0501] Input: Nonverbal data, conversation data

[0502] Data Calculation: Analysis by Emotion Recognition Engine

[0503] Output: Analysis results (user emotion recognition results)

[0504] Step 8:

[0505] The server generates feedback based on the analysis results, including specific advice such as, "The content of your complaint handling is good, but it would be better if you spoke with a calmer expression."

[0506] Input: Analysis results (user emotion recognition results)

[0507] Data Calculation: Feedback Generation

[0508] Output: Feedback content

[0509] Step 9:

[0510] The terminal displays the generated feedback on a user interface, and the user can check the feedback and understand areas for improvement in their communication skills.

[0511] Input: Feedback

[0512] Output: Feedback display to the user

[0513] As a result, this system allows users to effectively practice conversation skills in specific situations and receive feedback, thereby improving their customer service skills.

[0514] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0515] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0516] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0517] [Second embodiment]

[0518] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0519] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0520] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0521] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0522] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0523] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0524] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0525] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0526] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0527] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0528] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0529] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0530] System Overview

[0531] This invention provides a conversation practice tool for users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and non-verbal data, and provides feedback. This invention uses three entities: a server, a terminal, and a user.

[0532] Server Roles

[0533] The server is primarily responsible for performing the following functions:

[0534] 1. Receiving and analyzing information:

[0535] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[0536] 2. Creating a conversation simulator:

[0537] Generative AI (e.g., a GPT model) is used to generate a customized conversation simulator based on the received information. The received information is used as a prompt to create an appropriate dialogue model.

[0538] 3. Model Transfer:

[0539] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[0540] 4. Data analysis and feedback generation:

[0541] The system analyzes conversational and nonverbal data sent from the device and generates feedback to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[0542] Device Role

[0543] The terminal is responsible for the interaction with the user and the communication with the server:

[0544] 1. Information input interface:

[0545] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[0546] 2. Submit your request:

[0547] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[0548] 3. Interfacing with the conversation simulator:

[0549] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[0550] 4. Nonverbal Data Collection:

[0551] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[0552] 5. Viewing Feedback:

[0553] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[0554] User Roles

[0555] The user typically performs the following actions:

[0556] 1. Enter your information:

[0557] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[0558] 2. Execute the conversation:

[0559] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[0560] 3. Review your feedback:

[0561] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[0562] Specific examples

[0563] For example:

[0564] 1. User Input:

[0565] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[0566] 2. Server Processing:

[0567] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0568] 3. Conversation session:

[0569] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[0570] 4. Data Collection and Analysis:

[0571] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to a server, which analyzes them to generate feedback and send it back to the device.

[0572] 5. Viewing Feedback:

[0573] The device displays the feedback received from the server to the user, such as advice like, "Your answers are accurate, but your facial expression is stiff. You'll make a better impression if you relax."

[0574] In this way, a system is realized that allows users to efficiently improve their communication skills.

[0575] The processing flow will be explained below.

[0576] Step 1:

[0577] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[0578] Step 2:

[0579] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[0580] Step 3:

[0581] The server receives the request sent from the terminal and analyzes the situation and attribute information.

[0582] Step 4:

[0583] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[0584] Step 5:

[0585] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[0586] Step 6:

[0587] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[0588] Step 7:

[0589] The user interacts with the generated conversation simulator via a terminal using text or voice.

[0590] Step 8:

[0591] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[0592] Step 9:

[0593] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[0594] Step 10:

[0595] The server analyzes the received speech and non-verbal data and generates feedback regarding the user's communication.

[0596] Step 11:

[0597] The server sends the generated feedback to the device, which displays it in the user interface, allowing the user to receive specific advice on how to improve their communication skills.

[0598] In this way, the system provides an environment in which users can actually attempt a dialogue, evaluate their performance from multiple angles, and provide feedback, allowing users to efficiently improve their communication skills.

[0599] Example 1

[0600] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0601] Communication skills are extremely important in modern society, but there are limited tools available to effectively acquire and improve them. It's particularly difficult to objectively identify one's weaknesses and areas for improvement, making efficient training difficult. Furthermore, existing tools lack a system that comprehensively analyzes a variety of data, such as the content of conversations, facial expressions, and tone of voice, and provides feedback, limiting the ability to fully improve users' communication skills.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0603] In this invention, the server includes means for receiving information input by a user, means for generating prompts using a generative AI model and generating a customized conversation simulator, means for collecting conversation data and nonverbal data, means for analyzing the conversation data and nonverbal data and generating feedback, and means for displaying the feedback to the user. This allows the user to have their communication skills comprehensively evaluated and be given specific indications of areas for improvement, thereby enabling them to efficiently improve their skills.

[0604] "Means for receiving information entered by the user" refers to the function of the terminal providing an interface for the user to input information such as the situation and the attributes of the other party, and transmitting that information to the server.

[0605] "Means for generating prompts using a generative AI model and generating a customized conversation simulator" refers to a function that sends prompts to a generative AI model based on user information received by the server, thereby generating a conversation simulator that is suitable for a specific situation or the attributes of the other party.

[0606] "Means for collecting conversational data and non-verbal data" refers to the function of the device to record and save in real time using a camera or microphone voice data generated during a conversation with a user, as well as non-verbal data such as the user's facial expressions and tone of voice.

[0607] "Means for analyzing conversational data and non-verbal data and generating feedback" refers to a function in which the server uses analytical technology to evaluate collected conversational data and non-verbal data and generate specific advice and areas for improvement to improve the user's communication skills.

[0608] "Means for displaying feedback to the user" refers to a function that allows the terminal to display feedback information sent from the server on the user interface, allowing the user to check it and use it to improve their own communication skills.

[0609] A "situation" refers to a specific situation or scene that a user inputs to the conversation simulator, and includes specific situations such as an interview or a presentation.

[0610] "Partner attributes" refer to the characteristics and personality of the conversation partner that the user envisions in the conversation simulator, and include, for example, age, gender, and personality tendencies.

[0611] A "generative AI model" is an artificial intelligence model that provides generated text or dialogue based on input prompts, and includes, for example, a natural language processing model.

[0612] A "prompt" is a string of characters or data given to a generative AI model as an instruction or input, and is basic information used to determine the content of the generated conversation simulator.

[0613] A "conversation simulator" is a program or system that simulates dialogue based on specific situations and the attributes of the other party, generated by a generative AI model.

[0614] This invention is a conversation practice tool for users to improve their communication skills, and is a system that generates an AI conversation simulator based on information input by the user, collects and analyzes conversation data and non-verbal data, and provides feedback, thereby enabling users to improve their communication skills efficiently.

[0615] Server Roles

[0616] The server uses the following hardware and software:

[0617] Hardware: High-performance processors, memory, and storage devices

[0618] Software: Generative AI models (e.g., GPT-3)

[0619] The server first receives information about the situation and the other party's attributes sent from the device. For example, it receives information such as "Situation: Interview, Attributes: Middle-aged male, Strict personality." Next, it references the generative AI model based on the received information and generates an appropriate prompt. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality." This prompt is sent to the generative AI model, which generates a customized conversation simulator. The server then transfers the generated conversation simulator to the device.

[0620] The server also collects and analyzes speech and non-verbal data sent from the device, using voice and face recognition technology, and generates feedback to be provided to the user based on the analysis results and sends it to the device.

[0621] Device Role

[0622] The terminal is responsible for the interaction with the user and the communication with the server:

[0623] Hardware: Camera, microphone, display, input devices (keyboard and touchscreen)

[0624] Software: User Interface Application

[0625] The terminal provides an interface where the user can input information about the situation and the other person's attributes. For example, using text boxes and drop-down menus, the user can select "interview" and input "middle-aged male, strict personality." The information entered by the user is sent to the server, which then requests the generation of a conversation simulator.

[0626] The device initiates a dialogue session with the user using the conversation simulator received from the server. It provides an interface for the user to converse via voice or text, and uses a camera and microphone to collect the user's facial expressions and tone of voice in real time. The collected data is sent to the server for analysis and feedback generation.

[0627] Once the feedback is sent from the server, the device displays it to the user, and specific advice is provided in the user interface, allowing the user to analyze their performance and take action to improve.

[0628] User Roles

[0629] Users will use the system to improve their communication skills by:

[0630] First, a user inputs a specific situation and the attributes of the other person through the device interface. Then, the user engages in a dialogue with the generated conversation simulator. For example, in an interview simulator, the user responds to questions from the interviewer via voice or text. After the session ends, the user can review the feedback provided by the server and use it to improve their communication skills.

[0631] Specific examples

[0632] The user starts up the device, selects an interview situation, and enters "middle-aged male, strict personality" as the attributes. The device sends this information to the server, which uses a generative AI model to generate a prompt sentence: "Situation: Interview, Attributes: Middle-aged male, strict personality," and a conversation simulator is created.

[0633] The user begins a conversation with the simulator displayed on the device. The simulator acts as an interviewer, asking questions, and the user answers. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server. The server analyzes this, generates feedback, and sends it back to the device. The device displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your expression is stiff. You would make a better impression if you relaxed" may be displayed.

[0634] Through these steps, users can effectively improve their communication skills. The system also provides users with objective feedback that can be used for self-evaluation.

[0635] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0636] System program processing flow

[0637] Step 1:

[0638] The user uses the device interface to input the situation and the attributes of the other person. For example, they might select "interview" and input "middle-aged man, strict personality" as the attributes. This input is done using text boxes and drop-down menus. The input is the user's selection and input data. The output is the input information being saved on the device.

[0639] Step 2:

[0640] The terminal sends the situation and attribute information entered by the user to the server. This transmission is performed via an HTTP request. Specifically, the terminal sends the information "Situation: Interview, Attributes: Middle-aged male, Strict personality" to the server in JSON format. The input is the situation and attribute data from the user. The output is the server receiving this data.

[0641] Step 3:

[0642] The server generates a prompt for the generative AI model based on the received information and generates a conversation simulator. Using a generative AI model (e.g., GPT-3), it generates a prompt sentence such as "Situation: Interview, Attributes: Middle-aged male, Strict personality" and sends it to the AI ​​model. The input is the situation and attribute information sent by the user. The output is the generated conversation simulator.

[0643] Step 4:

[0644] The server transfers the generated conversation simulator to the device. This transfer is also performed via an HTTP request. The input is the generated conversation simulator. The output is the device receiving the conversation simulator. Specifically, the server sends the conversation simulator to the device in JSON format.

[0645] Step 5:

[0646] The terminal provides the received conversation simulator to the user and starts a conversation session. The user converses using voice or text. The terminal interface displays the dialogue content to the user and provides an input field. The input is the dialogue input between the generated conversation simulator and the user. The output is the user's dialogue history and dialogue content generated in real time.

[0647] Step 6:

[0648] The device collects conversational and non-verbal data. It uses a camera and microphone to record the user's facial expressions and tone of voice, and saves them as a log. The input is the user's facial expression data and voice data. The output is the collected log data.

[0649] Step 7:

[0650] The device sends the collected data to the server. The data includes conversational content and nonverbal data and is sent to the server in JSON format. The input is the collected conversational data and nonverbal data. The output is that the server receives this data.

[0651] Step 8:

[0652] The server analyzes the received data, using natural language processing and facial recognition technologies to evaluate the content of the conversation and non-verbal expressions. The input is the collected conversation data and non-verbal data. The output is feedback as the analysis results.

[0653] Step 9:

[0654] The server generates feedback based on the analysis results and sends it to the device. The generated feedback is sent in JSON format. The input is the analysis results. The output is that the device receives the feedback.

[0655] Step 10:

[0656] The terminal displays the feedback received from the server on the user interface. The feedback includes an evaluation of the user's conversation content and advice on non-verbal expressions. Specific areas for improvement and advice are highlighted. The input is the feedback from the server. The output is the user being able to visually confirm the feedback.

[0657] This realizes a system that allows users to efficiently improve their communication skills.

[0658] (Application example 1)

[0659] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] In modern retail stores, sales associates are expected to consistently provide high-quality customer service. However, for new sales associates or those with uncertain skills, simulating actual customer interactions and improving their skills is a difficult task. In particular, because they are required to respond to diverse customer attributes and situations, real-time feedback is lacking. This can result in a decline in customer satisfaction, which can have a negative impact on the store's reputation and sales. There is a need for a system that can solve these issues and effectively improve sales associates' customer service skills.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0662] In this invention, the server includes means for generating a customized conversation simulator based on a situation and attributes of the other party input by a user, interface means for collecting interaction data and nonverbal data with the conversation simulator, and means for analyzing the interaction data and nonverbal data and using the generative AI model to provide feedback, thereby enabling store clerks to simulate customer interaction situations in real time and improve their customer service skills based on the obtained feedback.

[0663] A "situation" is information that a user inputs to simulate a specific situation or scene.

[0664] "Partner attributes" is information that indicates the characteristics and personality of the person you are likely to be speaking with in a given situation.

[0665] A "customized conversation simulator" is a dialogue program that is generated based on the situation and attributes of the other party input by the user.

[0666] "Interaction data" refers to voice and text data exchanged between the user and the conversation simulator.

[0667] "Nonverbal data" refers to non-verbal information such as a user's facial expression, tone of voice, and posture.

[0668] A "generative AI model" is an artificial intelligence model that generates customized dialogue simulators based on user input information, or analyzes collected data and provides feedback.

[0669] "Customer service" refers to the customer service provided by retail store staff to actual customers.

[0670] "Feedback" is information for improvement or evaluation provided based on user interaction and non-verbal data.

[0671] This invention provides a system for improving the customer service skills of retail store staff. The system is composed of three main components: a server, a terminal, and a user.

[0672] Server Roles

[0673] The server is responsible for performing the following main functions:

[0674] 1. Receiving and analyzing information: The system receives information about the situation and the other party's attributes entered by the user on the terminal. For example, it receives the situation "serving an elderly person at a store" and the attributes "elderly woman, hard of hearing."

[0675] 2. Conversation simulator generation: A generative AI model (e.g., GPT-3 / 4 model) is used to generate a customized conversation simulator based on the received information. The AI ​​model uses the received information as a prompt to create an appropriate dialogue program.

[0676] 3. Analysis of interaction data and nonverbal data: Analyze the conversation data and nonverbal data sent from the device. Generate automatically generated feedback based on the analyzed data and send it to the device.

[0677] Device Role

[0678] The terminal is responsible for the interaction with the user and the communication with the server:

[0679] 1. Information input interface: Provide an interface for users to input information. Specifically, provide text boxes and multiple-choice input fields to allow users to enter the situation and the other person's attributes.

[0680] 2. Sending a request: The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[0681] 3. Interface with the conversation simulator: Start a dialogue session with the user using the generated conversation simulator received from the server. Provide an interface that allows the user to have a conversation via text or voice.

[0682] 4. Non-verbal data collection: A camera and microphone are used to collect the user's facial expressions and tone of voice in real time, which are then analyzed on the server.

[0683] 5. Displaying Feedback: The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[0684] User Roles

[0685] The user typically performs the following actions:

[0686] 1. Information input: A specific situation and the attributes of the other party are input through the terminal interface. For example, a situation involving dealing with an elderly person at a store, and the attributes of the elderly woman and her hearing impairment are input.

[0687] 2. Execution of conversation: Dialogue is conducted with the generated conversation simulator. At this time, the conversation is exchanged with the simulator via voice or text. For example, specific conversation content is confirmed as practice for dealing with elderly people.

[0688] 3. Check the feedback: Check the feedback provided by the server as an analysis result and improve your customer service skills based on it.

[0689] Specific examples

[0690] For example:

[0691] 1. User input: The user starts up the terminal and inputs the situation as "serving elderly people at a store" and the attributes as "elderly woman, hard of hearing."

[0692] 2. Server processing: Receives the information sent from the device, and generates a prompt for the generation AI based on the following: "Situation: Serving elderly customers at a store, Attributes: Elderly woman, hard of hearing," and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0693] 3. Conversation session: The user begins a conversation with the conversation simulator displayed on the terminal. The simulator plays the role of an elderly customer, prompting the user with questions and responses, and the user responds.

[0694] 4. Data collection and analysis: The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the conversation to the server. The server analyzes them and generates feedback, which is sent back to the device.

[0695] 5. Displaying feedback: The device displays the feedback received from the server to the user. For example, advice such as "You should speak more clearly. Also, it is important to treat the elderly with a kind expression" is displayed.

[0696] Example prompt sentence:

[0697] Situation: Assisting elderly customers in stores

[0698] Attributes: Elderly woman, hard of hearing

[0699] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0700] Step 1:

[0701] The user inputs the situation and the other person's attributes.

[0702] The user inputs a specific situation (e.g., "serving an elderly person at a store") and the other person's attributes (e.g., "elderly woman, hard of hearing") through the terminal interface. The terminal then receives the input data and prepares it to be sent to the server.

[0703] Input: Situation and partner's attribute information

[0704] Output: The request data sent to the server

[0705] Step 2:

[0706] Sending a request from the device to the server

[0707] The terminal sends the information entered by the user to the server, which then starts generating a customized conversation simulator. Specifically, the terminal sends the situation and the attributes of the other party as prompts to the server.

[0708] Input: Request data

[0709] Output: Prompt data received by the server

[0710] Step 3:

[0711] The server generates a conversation simulator.

[0712] The server uses the received prompt data to generate a customized conversation simulator using a generative AI model (e.g., GPT-3 / 4). The AI ​​model generates a dialogue program based on the prompt data and returns the generated conversation simulator to the device.

[0713] Input: prompt data

[0714] Output: Generated conversation simulator

[0715] Step 4:

[0716] The terminal receives the conversation simulator and starts the conversation session.

[0717] The terminal starts a dialogue session with the user using the generated conversation simulator received from the server. The user converses with the user using voice or text. Specifically, the terminal displays the conversation simulator and the user responds.

[0718] Input: Generated conversation simulator

[0719] Output: Interaction data

[0720] Step 5:

[0721] Nonverbal data collection

[0722] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and sends this non-verbal data to a server after the conversation ends.

[0723] Input: User's facial expression and tone of voice data

[0724] Output: Collected nonverbal data

[0725] Step 6:

[0726] Transmission of interaction data and non-verbal data

[0727] The terminal transmits the collected interaction data and nonverbal data to the server, which receives and analyzes the data.

[0728] Input: Interaction data, nonverbal data

[0729] Output: Data used by the server for analysis

[0730] Step 7:

[0731] The server analyzes the data and generates feedback

[0732] The server analyzes the received interaction data and nonverbal data using generative AI models and data analysis tools. Based on the analysis results, it generates feedback for the user and sends it back to the device.

[0733] Input: Interaction data, nonverbal data

[0734] Output: Generated feedback

[0735] Step 8:

[0736] The device displays feedback

[0737] The device displays the feedback received from the server to the user, who can review the feedback and receive specific advice on how to improve their performance.

[0738] Input: Generated feedback

[0739] Output: Feedback that is displayed to the user

[0740] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0741] System Overview

[0742] This invention provides a conversation practice tool that allows users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Furthermore, by combining it with an emotion engine, it has the ability to recognize the user's emotions and provide feedback based on the results. This invention uses three entities: a server, a terminal, and a user.

[0743] Server Roles

[0744] The server is primarily responsible for performing the following functions:

[0745] 1. Receiving and analyzing information:

[0746] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[0747] 2. Creating a conversation simulator:

[0748] It uses generative AI (e.g., a GPT model) to generate a customized conversation simulator based on the information it receives, including prompts entered by the user.

[0749] 3. Model Transfer:

[0750] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[0751] 4. Data Analysis and Emotion Recognition:

[0752] The system analyzes conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[0753] 5. Feedback Generation:

[0754] Based on the analysis results and emotion recognition results, feedback is generated to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[0755] Device Role

[0756] The terminal is responsible for the interaction with the user and the communication with the server:

[0757] 1. Information input interface:

[0758] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[0759] 2. Submit your request:

[0760] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[0761] 3. Interfacing with the conversation simulator:

[0762] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[0763] 4. Nonverbal Data Collection:

[0764] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[0765] 5. Viewing Feedback:

[0766] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[0767] User Roles

[0768] The user typically performs the following actions:

[0769] 1. Enter your information:

[0770] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[0771] 2. Execute the conversation:

[0772] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[0773] 3. Review your feedback:

[0774] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[0775] Specific examples

[0776] For example:

[0777] 1. User Input:

[0778] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[0779] 2. Server Processing:

[0780] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0781] 3. Conversation session:

[0782] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[0783] 4. Data Collection and Analysis:

[0784] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[0785] 5. Providing Feedback:

[0786] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[0787] In this way, we realize a system that allows users to efficiently improve their communication skills. By using an emotion engine, we can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[0788] The processing flow will be explained below.

[0789] Step 1:

[0790] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[0791] Step 2:

[0792] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[0793] Step 3:

[0794] The server receives the request from the device and analyzes the situation and attribute information. For example, it extracts the situation "interview" and the attributes "middle-aged male, strict personality."

[0795] Step 4:

[0796] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[0797] Step 5:

[0798] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[0799] Step 6:

[0800] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[0801] Step 7:

[0802] The user interacts with the generated conversation simulator via a terminal, either through text or voice. For example, the simulator playing the role of an interviewer asks, "Please introduce yourself," and the user responds.

[0803] Step 8:

[0804] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[0805] Step 9:

[0806] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[0807] Step 10:

[0808] The server analyzes the received speech and non-verbal data and uses an emotion engine to recognize the user's emotions, for example, by analyzing facial expressions to determine whether the user is smiling.

[0809] Step 11:

[0810] The server analyzes the collected data to generate feedback about the user's emotions, such as providing specific advice like, "Your answers are accurate, but your facial expression seems stiff, so you should try to relax."

[0811] Step 12:

[0812] The server sends the generated feedback to the terminal, which displays it on the user interface, allowing the user to receive specific feedback to improve their performance.

[0813] Example 2

[0814] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0815] In modern society, improving communication skills is considered important, but there are limited opportunities to practice them effectively. It is particularly difficult to practice nonverbal communication (facial expressions, tone of voice, etc.). It is also difficult to objectively evaluate the effectiveness of practice and obtain specific feedback. To address these challenges, there is a need for a system that provides practical, individualized practice and feedback.

[0816] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0817] In this invention, the server includes means for receiving information input by the user, means for generating a customized conversation simulator using a generative model, means for transferring the simulator model to the user's terminal, means for collecting conversation data with the conversation simulator, and means for analyzing the conversation data and nonverbal data, performing emotion recognition, and providing feedback, thereby enabling the user to objectively evaluate the effects of their practice and identify specific areas for improvement.

[0818] "Information entered by the user" refers to data that the user enters to the system to specify a particular situation or the attributes of the other party.

[0819] A "generative model" is an algorithm or program that a pre-trained artificial intelligence uses to generate natural language based on specific input.

[0820] A "customized conversation simulator" is a conversation agent that is generated by a generative model based on information input by the user and that corresponds to a specific situation or the attributes of the other party.

[0821] The "means for transferring the simulator model to the user's terminal" is a mechanism for transmitting the generated conversation simulator from the server to the terminal used by the user.

[0822] "Conversation data with the conversation simulator" is text or voice data generated when the user converses with the conversation simulator.

[0823] "Nonverbal data" refers to data that includes non-verbal communication elements such as a user's facial expression and tone of voice.

[0824] "Emotion recognition" is a technology that analyzes speech and non-verbal data to identify a user's emotional state.

[0825] "Feedback" is information that is generated based on analyzed data and includes advice and suggestions for improving the user's communication skills.

[0826] This invention is a system that provides users with a conversation practice tool to improve their communication skills. The system generates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Three entities are involved: a server, a terminal, and a user.

[0827] System Overview

[0828] Server Roles

[0829] The server mainly performs the following functions:

[0830] 1. Receiving and analyzing information:

[0831] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[0832] 2. Creating a conversation simulator:

[0833] The server uses a generative AI model (e.g., GPT-3) to generate a customized conversation simulator based on the received information. The prompt includes the information entered by the user. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality."

[0834] 3. Model Transfer:

[0835] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[0836] 4. Data Analysis and Emotion Recognition:

[0837] The server analyzes the conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[0838] 5. Feedback Generation:

[0839] Based on the analysis results and emotion recognition results, feedback is generated to be provided to the user. The feedback is advice based on the content of the conversation and nonverbal expressions. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is provided.

[0840] Device Role

[0841] The terminal is responsible for the interaction with the user and the communication with the server.

[0842] 1. Information input interface:

[0843] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or an input field with multiple-choice options.

[0844] 2. Submit your request:

[0845] The terminal sends the information entered by the user to the server, which starts the generation of the conversation simulator.

[0846] 3. Interfacing with the conversation simulator:

[0847] The terminal uses the generated conversation simulator received from the server to initiate a dialogue session with the user, providing an interface through which the user can converse via text or voice.

[0848] 4. Nonverbal Data Collection:

[0849] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[0850] 5. Viewing Feedback:

[0851] The device displays the feedback received from the server in a user interface, allowing the user to receive specific advice on how to improve their performance.

[0852] User Roles

[0853] The user mainly performs the following actions:

[0854] 1. Enter your information:

[0855] The user inputs a specific situation and the attributes of the other person through the terminal interface. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[0856] 2. Execute the conversation:

[0857] The user interacts with the generated conversation simulator by speaking or texting, for example, answering hypothetical questions as practice for an interview.

[0858] 3. Review your feedback:

[0859] The user checks the feedback provided by the server as an analysis result and improves their own communication skills based on it.

[0860] Specific examples

[0861] Specific examples are shown below.

[0862] 1. User Input:

[0863] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[0864] 2. Server Processing:

[0865] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[0866] 3. Conversation session:

[0867] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[0868] 4. Data Collection and Analysis:

[0869] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[0870] 5. Providing Feedback:

[0871] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[0872] Technology used and data processing

[0873] This system includes the following technologies and data processing:

[0874] Hardware: The device is equipped with a camera and microphone, which allows for real-time collection of the user's facial expressions and tone of voice.

[0875] Software: Generative AI models (e.g., GPT-3) are used to generate the conversation simulator, and an emotion engine is used to analyze nonverbal data and employ algorithms to recognize user emotions.

[0876] Data Processing: Techniques are used to collect and analyze speech and non-verbal data. The analyzed data provides important information for providing feedback to users.

[0877] Through this system, users can effectively improve their communication skills. By utilizing the emotion engine, users can receive more detailed and useful feedback.

[0878] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0879] Step 1:

[0880] The terminal provides an interface for users to access the system. Here, users select a situation (e.g., an interview) and enter the other party's attributes (e.g., middle-aged man, strict personality). The input data is saved in text format and sent from the terminal to the server in JSON format.

[0881] Input: Situations and attributes entered by the user

[0882] Output: JSON format data (situations and attributes)

[0883] Step 2:

[0884] The server receives and analyzes the JSON-formatted data sent from the device. As a result of the analysis, situation and attribute information is extracted. Based on this information, the server generates a prompt for the generative AI model.

[0885] Input: JSON format data (situations and attributes)

[0886] Output: Prompt sentence for the generative AI model (e.g., "Situation: Interview, Attributes: Middle-aged male, Strict personality")

[0887] Step 3:

[0888] The server sends the prompt sentence to a generative AI model (e.g., GPT-3) to generate a customized conversation simulator, which is returned to the server as text data.

[0889] Input: Prompt sentence for the generative AI model

[0890] Output: Text data of the generated conversation simulator

[0891] Step 4:

[0892] The server transfers the generated conversation simulator to the terminal, which displays the conversation simulator on its interface and starts a dialogue with the user.

[0893] Input: Text data of the generated conversation simulator

[0894] Output: Interface showing the conversation simulator

[0895] Step 5:

[0896] The user interacts with the conversation simulator and inputs responses. The device collects the conversation data in real time. It also uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice.

[0897] Input: User responses, facial expressions, and tone of voice

[0898] Output: Speech and non-verbal data

[0899] Step 6:

[0900] The device transmits the collected speech and non-verbal data in real time to a server, which receives the data and performs data analysis using NLP technology and an emotion engine.

[0901] Input: Speech and non-verbal data

[0902] Output: Analysis results and emotion recognition results

[0903] Step 7:

[0904] The server generates feedback to provide to the user based on the analysis and emotion recognition results, including specific advice based on the conversation content and nonverbal expressions.

[0905] Input: Analysis results and emotion recognition results

[0906] Output: Generated feedback

[0907] Step 8:

[0908] The server transmits the generated feedback to the terminal, which displays the feedback on a user interface for the user to confirm.

[0909] Input: Generated feedback

[0910] Output: Feedback displayed in the user interface

[0911] (Application example 2)

[0912] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0913] To improve their communication skills, users need to be able to practice effectively and practically based on specific situations and the attributes of the people they are interacting with. However, conventional methods have faced challenges in providing practical training for specific situations and in obtaining effective feedback. This problem is particularly evident in training salespeople in brick-and-mortar stores to improve their customer service skills.

[0914] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information input by a user, means for generating a customized conversation simulator based on the information, means for collecting conversation data with the conversation simulator, means for analyzing the conversation data and nonverbal data and providing feedback, means for collecting the nonverbal data by face recognition means, and means for supporting the improvement of the user's customer interaction skills through the feedback. This enables the user to undergo practical training based on specific situations and the attributes of the other party and receive effective feedback.

[0915] A "user" is a person who utilizes the system to input information and receive feedback to improve their communication skills.

[0916] The "means for receiving information" is an interface or device for acquiring information input by the user about the situation and the attributes of the other party.

[0917] The "means for generating a conversation simulator" refers to a technique or device for creating a conversation simulator that corresponds to a specific situation or the attributes of the other person, based on information input by the user.

[0918] The "means for collecting conversation data" refers to a device or technology for recording the content of the conversation that takes place between the user and the generated conversation simulator.

[0919] "Nonverbal data" refers to data that includes information other than words, such as a user's facial expression or tone of voice.

[0920] A "means for providing feedback" is a device or technology that analyzes the collected speech and non-verbal data and indicates to the user areas for improvement in their communication skills.

[0921] "Facial recognition means" refers to technology or systems that use devices such as cameras to identify and analyze a user's facial expressions.

[0922] "Customer service skills" refers to the abilities and techniques that sales staff in physical stores and other locations have to respond appropriately to customers.

[0923] "Means for supporting improvement of users' customer service skills" are technologies and systems that include feedback and advice provided to enable users to provide better customer service.

[0924] This invention provides a training system for sales staff in brick-and-mortar stores to improve their customer service skills. The system operates using three entities: a user, a server, and a terminal. The roles and specific operations of each of these entities are described below.

[0925] Server Roles

[0926] The server performs the following functions:

[0927] 1. Receiving and analyzing information:

[0928] The server receives the situation and the other party's attributes sent from the terminal. For example, the server receives the situation "handling a complaint" and the attributes "elderly woman, angry."

[0929] 2. Creating a conversation simulator:

[0930] The server uses a generative AI model (e.g., OpenAI GPT-3.5) to generate a customized conversation simulator based on the information it receives, including prompts entered by the user.

[0931] Example prompt sentence:

[0932] Situation: Complaint handling, Attributes: Elderly woman, angry

[0933] 3. Model Transfer:

[0934] The server transfers the generated conversation simulator model to the terminal so that the user can use it.

[0935] 4. Data Analysis and Emotion Recognition:

[0936] The server analyzes the conversational data and non-verbal data sent from the device and recognizes the user's emotions using an emotion recognition engine, which includes facial expression analysis and tone of voice analysis.

[0937] 5. Feedback Generation:

[0938] The server generates feedback to provide to the user based on the analysis results and emotion recognition results, including advice based on the content of the conversation and nonverbal expressions.

[0939] Device Role

[0940] The terminal is responsible for the following functions:

[0941] 1. Information input interface:

[0942] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or a multiple-choice input field.

[0943] 2. Submit your request:

[0944] The terminal starts generating a conversation simulator by transmitting the information input by the user to the server.

[0945] 3. Interfacing with the conversation simulator:

[0946] The terminal initiates an interactive session with the user using the generated conversation simulator received from the server, and provides an interface through which the user can conduct a conversation by text or voice.

[0947] 4. Nonverbal Data Collection:

[0948] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[0949] 5. Viewing Feedback:

[0950] The terminal displays the feedback received from the server on a user interface, allowing the user to receive specific advice on how to improve their performance.

[0951] User Roles

[0952] The user performs the following actions:

[0953] 1. Enter your information:

[0954] Through the terminal interface, specific situations and the attributes of the other party are input. For example, the situation of handling a complaint and the expected attributes of the customer at that time (elderly woman, angry) can be input.

[0955] 2. Execute the conversation:

[0956] The user interacts with the generated conversation simulator by speaking or texting. For example, the user can practice how to respond to a complaint.

[0957] 3. Review your feedback:

[0958] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[0959] Specific examples

[0960] As a concrete example, consider the following flow:

[0961] 1. The user starts up the terminal, selects the situation "Handling a complaint," and enters "elderly woman, angry" as the attribute.

[0962] 2. The server receives the information sent from the device and sends a prompt to the generative AI model with the following content: "Situation: Handling a complaint, Attributes: Elderly woman, Angry," and generates a customized conversation simulator.

[0963] 3. The user starts a conversation on the terminal using the conversation simulator returned from the server. The simulator plays the role of a customer and makes a complaint, and the user responds.

[0964] 4. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them to the server along with the content of the conversation.

[0965] 5. The server analyzes the collected data and recognizes the user's emotions using an emotion recognition engine.

[0966] 6. The device receives the feedback generated by the server and displays specific advice to the user, such as, "The content of your complaint response is good, but it would be better if you spoke with a calmer expression."

[0967] In this way, the system allows users to efficiently improve their customer service skills in physical stores. By utilizing an emotion recognition engine, the system can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[0968] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0969] Step 1:

[0970] The user starts up the terminal and inputs information about the situation and the other party's attributes. For example, the situation "handling a complaint" and the attributes "elderly woman, angry" are entered into the text boxes. This starts the information input interface.

[0971] Input: situation (e.g., handling a complaint), other party's attributes (e.g., elderly woman, angry)

[0972] Output: The information entered

[0973] Step 2:

[0974] The terminal transmits the information input by the user to the server, where it arrives as a customized prompt.

[0975] Input: Information entered by the user

[0976] Output: Request to server

[0977] Step 3:

[0978] The server analyzes the received information and generates prompt sentences. The prompt sentences are used to generate a customized conversation simulator for a generative AI model (e.g., OpenAI GPT-3.5).

[0979] Input: Prompt sentence (e.g., Situation: Handling a complaint, Attributes: Elderly woman, Angry)

[0980] Data processing: Creating a conversation simulator using a generative AI model

[0981] Output: A customized conversation simulator

[0982] Step 4:

[0983] The server transfers the generated conversation simulator to the terminal, which then starts a dialogue session with the user based on the received simulator.

[0984] Input: A customized conversation simulator

[0985] Output: Transfer of conversation simulator to terminal

[0986] Step 5:

[0987] The user initiates a dialogue with a customized conversation simulator displayed on the terminal, and the user converses with the simulator using voice or text.

[0988] Input: A customized conversation simulator and the user's conversation content

[0989] Output: Conversation progress

[0990] Step 6:

[0991] The device uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice, which is then sent to a server in real time along with the content of the conversation.

[0992] Input: User's facial expression data, voice data

[0993] Data Processing: Collection and Transmission of Nonverbal Data

[0994] Output: Nonverbal and conversational data

[0995] Step 7:

[0996] The server analyzes the received non-verbal data and conversation data, and recognizes the user's emotions using an emotion recognition engine. It also analyzes facial expressions and voice using a face recognition means.

[0997] Input: Nonverbal data, conversation data

[0998] Data Calculation: Analysis by Emotion Recognition Engine

[0999] Output: Analysis results (user emotion recognition results)

[1000] Step 8:

[1001] The server generates feedback based on the analysis results, including specific advice such as, "The content of your complaint handling is good, but it would be better if you spoke with a calmer expression."

[1002] Input: Analysis results (user emotion recognition results)

[1003] Data Calculation: Feedback Generation

[1004] Output: Feedback content

[1005] Step 9:

[1006] The terminal displays the generated feedback on a user interface, and the user can check the feedback and understand areas for improvement in their communication skills.

[1007] Input: Feedback

[1008] Output: Feedback display to the user

[1009] As a result, this system allows users to effectively practice conversation skills in specific situations and receive feedback, thereby improving their customer service skills.

[1010] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1011] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1012] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1013] [Third embodiment]

[1014] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1015] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1016] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1017] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1018] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1019] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1020] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1021] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1022] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1023] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1024] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1025] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1026] System Overview

[1027] This invention provides a conversation practice tool for users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and non-verbal data, and provides feedback. This invention uses three entities: a server, a terminal, and a user.

[1028] Server Roles

[1029] The server is primarily responsible for performing the following functions:

[1030] 1. Receiving and analyzing information:

[1031] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[1032] 2. Creating a conversation simulator:

[1033] Generative AI (e.g., a GPT model) is used to generate a customized conversation simulator based on the received information. The received information is used as a prompt to create an appropriate dialogue model.

[1034] 3. Model Transfer:

[1035] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[1036] 4. Data analysis and feedback generation:

[1037] The system analyzes conversational and nonverbal data sent from the device and generates feedback to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[1038] Device Role

[1039] The terminal is responsible for the interaction with the user and the communication with the server:

[1040] 1. Information input interface:

[1041] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[1042] 2. Submit your request:

[1043] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[1044] 3. Interfacing with the conversation simulator:

[1045] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[1046] 4. Nonverbal Data Collection:

[1047] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[1048] 5. Viewing Feedback:

[1049] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[1050] User Roles

[1051] The user typically performs the following actions:

[1052] 1. Enter your information:

[1053] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[1054] 2. Execute the conversation:

[1055] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[1056] 3. Review your feedback:

[1057] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[1058] Specific examples

[1059] For example:

[1060] 1. User Input:

[1061] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[1062] 2. Server Processing:

[1063] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1064] 3. Conversation session:

[1065] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[1066] 4. Data Collection and Analysis:

[1067] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to a server, which analyzes them to generate feedback and send it back to the device.

[1068] 5. Viewing Feedback:

[1069] The device displays the feedback received from the server to the user, such as advice like, "Your answers are accurate, but your facial expression is stiff. You'll make a better impression if you relax."

[1070] In this way, a system is realized that allows users to efficiently improve their communication skills.

[1071] The processing flow will be explained below.

[1072] Step 1:

[1073] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[1074] Step 2:

[1075] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[1076] Step 3:

[1077] The server receives the request sent from the terminal and analyzes the situation and attribute information.

[1078] Step 4:

[1079] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[1080] Step 5:

[1081] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[1082] Step 6:

[1083] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[1084] Step 7:

[1085] The user interacts with the generated conversation simulator via a terminal using text or voice.

[1086] Step 8:

[1087] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[1088] Step 9:

[1089] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[1090] Step 10:

[1091] The server analyzes the received speech and non-verbal data and generates feedback regarding the user's communication.

[1092] Step 11:

[1093] The server sends the generated feedback to the device, which displays it in the user interface, allowing the user to receive specific advice on how to improve their communication skills.

[1094] In this way, the system provides an environment in which users can actually attempt a dialogue, evaluate their performance from multiple angles, and provide feedback, allowing users to efficiently improve their communication skills.

[1095] Example 1

[1096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1097] Communication skills are extremely important in modern society, but there are limited tools available to effectively acquire and improve them. It's particularly difficult to objectively identify one's weaknesses and areas for improvement, making efficient training difficult. Furthermore, existing tools lack a system that comprehensively analyzes a variety of data, such as the content of conversations, facial expressions, and tone of voice, and provides feedback, limiting the ability to fully improve users' communication skills.

[1098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1099] In this invention, the server includes means for receiving information input by a user, means for generating prompts using a generative AI model and generating a customized conversation simulator, means for collecting conversation data and nonverbal data, means for analyzing the conversation data and nonverbal data and generating feedback, and means for displaying the feedback to the user. This allows the user to have their communication skills comprehensively evaluated and be given specific indications of areas for improvement, thereby enabling them to efficiently improve their skills.

[1100] "Means for receiving information entered by the user" refers to the function of the terminal providing an interface for the user to input information such as the situation and the attributes of the other party, and transmitting that information to the server.

[1101] "Means for generating prompts using a generative AI model and generating a customized conversation simulator" refers to a function that sends prompts to a generative AI model based on user information received by the server, thereby generating a conversation simulator that is suitable for a specific situation or the attributes of the other party.

[1102] "Means for collecting conversational data and non-verbal data" refers to the function of the device to record and save in real time using a camera or microphone voice data generated during a conversation with a user, as well as non-verbal data such as the user's facial expressions and tone of voice.

[1103] "Means for analyzing conversational data and non-verbal data and generating feedback" refers to a function in which the server uses analytical technology to evaluate collected conversational data and non-verbal data and generate specific advice and areas for improvement to improve the user's communication skills.

[1104] "Means for displaying feedback to the user" refers to a function that allows the terminal to display feedback information sent from the server on the user interface, allowing the user to check it and use it to improve their own communication skills.

[1105] A "situation" refers to a specific situation or scene that a user inputs to the conversation simulator, and includes specific situations such as an interview or a presentation.

[1106] "Partner attributes" refer to the characteristics and personality of the conversation partner that the user envisions in the conversation simulator, and include, for example, age, gender, and personality tendencies.

[1107] A "generative AI model" is an artificial intelligence model that provides generated text or dialogue based on input prompts, and includes, for example, a natural language processing model.

[1108] A "prompt" is a string of characters or data given to a generative AI model as an instruction or input, and is basic information used to determine the content of the generated conversation simulator.

[1109] A "conversation simulator" is a program or system that simulates dialogue based on specific situations and the attributes of the other party, generated by a generative AI model.

[1110] This invention is a conversation practice tool for users to improve their communication skills, and is a system that generates an AI conversation simulator based on information input by the user, collects and analyzes conversation data and non-verbal data, and provides feedback, thereby enabling users to improve their communication skills efficiently.

[1111] Server Roles

[1112] The server uses the following hardware and software:

[1113] Hardware: High-performance processors, memory, and storage devices

[1114] Software: Generative AI models (e.g., GPT-3)

[1115] The server first receives information about the situation and the other party's attributes sent from the device. For example, it receives information such as "Situation: Interview, Attributes: Middle-aged male, Strict personality." Next, it references the generative AI model based on the received information and generates an appropriate prompt. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality." This prompt is sent to the generative AI model, which generates a customized conversation simulator. The server then transfers the generated conversation simulator to the device.

[1116] The server also collects and analyzes speech and non-verbal data sent from the device, using voice and face recognition technology, and generates feedback to be provided to the user based on the analysis results and sends it to the device.

[1117] Device Role

[1118] The terminal is responsible for the interaction with the user and the communication with the server:

[1119] Hardware: Camera, microphone, display, input devices (keyboard and touchscreen)

[1120] Software: User Interface Application

[1121] The terminal provides an interface where the user can input information about the situation and the other person's attributes. For example, using text boxes and drop-down menus, the user can select "interview" and input "middle-aged male, strict personality." The information entered by the user is sent to the server, which then requests the generation of a conversation simulator.

[1122] The device initiates a dialogue session with the user using the conversation simulator received from the server. It provides an interface for the user to converse via voice or text, and uses a camera and microphone to collect the user's facial expressions and tone of voice in real time. The collected data is sent to the server for analysis and feedback generation.

[1123] Once the feedback is sent from the server, the device displays it to the user, and specific advice is provided in the user interface, allowing the user to analyze their performance and take action to improve.

[1124] User Roles

[1125] Users will use the system to improve their communication skills by:

[1126] First, a user inputs a specific situation and the attributes of the other person through the device interface. Then, the user engages in a dialogue with the generated conversation simulator. For example, in an interview simulator, the user responds to questions from the interviewer via voice or text. After the session ends, the user can review the feedback provided by the server and use it to improve their communication skills.

[1127] Specific examples

[1128] The user starts up the device, selects an interview situation, and enters "middle-aged male, strict personality" as the attributes. The device sends this information to the server, which uses a generative AI model to generate a prompt sentence: "Situation: Interview, Attributes: Middle-aged male, strict personality," and a conversation simulator is created.

[1129] The user begins a conversation with the simulator displayed on the device. The simulator acts as an interviewer, asking questions, and the user answers. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server. The server analyzes this, generates feedback, and sends it back to the device. The device displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your expression is stiff. You would make a better impression if you relaxed" may be displayed.

[1130] Through these steps, users can effectively improve their communication skills. The system also provides users with objective feedback that can be used for self-evaluation.

[1131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1132] System program processing flow

[1133] Step 1:

[1134] The user uses the device interface to input the situation and the attributes of the other person. For example, they might select "interview" and input "middle-aged man, strict personality" as the attributes. This input is done using text boxes and drop-down menus. The input is the user's selection and input data. The output is the input information being saved on the device.

[1135] Step 2:

[1136] The terminal sends the situation and attribute information entered by the user to the server. This transmission is performed via an HTTP request. Specifically, the terminal sends the information "Situation: Interview, Attributes: Middle-aged male, Strict personality" to the server in JSON format. The input is the situation and attribute data from the user. The output is the server receiving this data.

[1137] Step 3:

[1138] The server generates a prompt for the generative AI model based on the received information and generates a conversation simulator. Using a generative AI model (e.g., GPT-3), it generates a prompt sentence such as "Situation: Interview, Attributes: Middle-aged male, Strict personality" and sends it to the AI ​​model. The input is the situation and attribute information sent by the user. The output is the generated conversation simulator.

[1139] Step 4:

[1140] The server transfers the generated conversation simulator to the device. This transfer is also performed via an HTTP request. The input is the generated conversation simulator. The output is the device receiving the conversation simulator. Specifically, the server sends the conversation simulator to the device in JSON format.

[1141] Step 5:

[1142] The terminal provides the received conversation simulator to the user and starts a conversation session. The user converses using voice or text. The terminal interface displays the dialogue content to the user and provides an input field. The input is the dialogue input between the generated conversation simulator and the user. The output is the user's dialogue history and dialogue content generated in real time.

[1143] Step 6:

[1144] The device collects conversational and non-verbal data. It uses a camera and microphone to record the user's facial expressions and tone of voice, and saves them as a log. The input is the user's facial expression data and voice data. The output is the collected log data.

[1145] Step 7:

[1146] The device sends the collected data to the server. The data includes conversational content and nonverbal data and is sent to the server in JSON format. The input is the collected conversational data and nonverbal data. The output is that the server receives this data.

[1147] Step 8:

[1148] The server analyzes the received data, using natural language processing and facial recognition technologies to evaluate the content of the conversation and non-verbal expressions. The input is the collected conversation data and non-verbal data. The output is feedback as the analysis results.

[1149] Step 9:

[1150] The server generates feedback based on the analysis results and sends it to the device. The generated feedback is sent in JSON format. The input is the analysis results. The output is that the device receives the feedback.

[1151] Step 10:

[1152] The terminal displays the feedback received from the server on the user interface. The feedback includes an evaluation of the user's conversation content and advice on non-verbal expressions. Specific areas for improvement and advice are highlighted. The input is the feedback from the server. The output is the user being able to visually confirm the feedback.

[1153] This realizes a system that allows users to efficiently improve their communication skills.

[1154] (Application example 1)

[1155] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1156] In modern retail stores, sales associates are expected to consistently provide high-quality customer service. However, for new sales associates or those with uncertain skills, simulating actual customer interactions and improving their skills is a difficult task. In particular, because they are required to respond to diverse customer attributes and situations, real-time feedback is lacking. This can result in a decline in customer satisfaction, which can have a negative impact on the store's reputation and sales. There is a need for a system that can solve these issues and effectively improve sales associates' customer service skills.

[1157] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1158] In this invention, the server includes means for generating a customized conversation simulator based on a situation and attributes of the other party input by a user, interface means for collecting interaction data and nonverbal data with the conversation simulator, and means for analyzing the interaction data and nonverbal data and using the generative AI model to provide feedback, thereby enabling store clerks to simulate customer interaction situations in real time and improve their customer service skills based on the obtained feedback.

[1159] A "situation" is information that a user inputs to simulate a specific situation or scene.

[1160] "Partner attributes" is information that indicates the characteristics and personality of the person you are likely to be speaking with in a given situation.

[1161] A "customized conversation simulator" is a dialogue program that is generated based on the situation and attributes of the other party input by the user.

[1162] "Interaction data" refers to voice and text data exchanged between the user and the conversation simulator.

[1163] "Nonverbal data" refers to non-verbal information such as a user's facial expression, tone of voice, and posture.

[1164] A "generative AI model" is an artificial intelligence model that generates customized dialogue simulators based on user input information, or analyzes collected data and provides feedback.

[1165] "Customer service" refers to the customer service provided by retail store staff to actual customers.

[1166] "Feedback" is information for improvement or evaluation provided based on user interaction and non-verbal data.

[1167] This invention provides a system for improving the customer service skills of retail store staff. The system is composed of three main components: a server, a terminal, and a user.

[1168] Server Roles

[1169] The server is responsible for performing the following main functions:

[1170] 1. Receiving and analyzing information: The system receives information about the situation and the other party's attributes entered by the user on the terminal. For example, it receives the situation "serving an elderly person at a store" and the attributes "elderly woman, hard of hearing."

[1171] 2. Conversation simulator generation: A generative AI model (e.g., GPT-3 / 4 model) is used to generate a customized conversation simulator based on the received information. The AI ​​model uses the received information as a prompt to create an appropriate dialogue program.

[1172] 3. Analysis of interaction data and nonverbal data: Analyze the conversation data and nonverbal data sent from the device. Generate automatically generated feedback based on the analyzed data and send it to the device.

[1173] Device Role

[1174] The terminal is responsible for the interaction with the user and the communication with the server:

[1175] 1. Information input interface: Provide an interface for users to input information. Specifically, provide text boxes and multiple-choice input fields to allow users to enter the situation and the other person's attributes.

[1176] 2. Sending a request: The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[1177] 3. Interface with the conversation simulator: Start a dialogue session with the user using the generated conversation simulator received from the server. Provide an interface that allows the user to have a conversation via text or voice.

[1178] 4. Non-verbal data collection: A camera and microphone are used to collect the user's facial expressions and tone of voice in real time, which are then analyzed on the server.

[1179] 5. Displaying Feedback: The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[1180] User Roles

[1181] The user typically performs the following actions:

[1182] 1. Information input: A specific situation and the attributes of the other party are input through the terminal interface. For example, a situation involving dealing with an elderly person at a store, and the attributes of the elderly woman and her hearing impairment are input.

[1183] 2. Execution of conversation: Dialogue is carried out with the generated conversation simulator. At this time, the conversation is exchanged with the simulator via voice or text. For example, specific conversation content is confirmed as practice for dealing with elderly people.

[1184] 3. Check the feedback: Check the feedback provided by the server as an analysis result and improve your customer service skills based on it.

[1185] Specific examples

[1186] For example:

[1187] 1. User input: The user starts up the terminal and inputs the situation as "serving elderly people at a store" and the attributes as "elderly woman, hard of hearing."

[1188] 2. Server processing: Receives the information sent from the device, and generates a prompt for the generation AI based on the following: "Situation: Serving elderly customers at a store, Attributes: Elderly woman, hard of hearing," and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1189] 3. Conversation session: The user begins a conversation with the conversation simulator displayed on the terminal. The simulator plays the role of an elderly customer, prompting the user with questions and responses, and the user responds.

[1190] 4. Data collection and analysis: The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the conversation to the server. The server analyzes them and generates feedback, which is sent back to the device.

[1191] 5. Displaying feedback: The device displays the feedback received from the server to the user. For example, advice such as "You should speak more clearly. Also, it is important to treat the elderly with a kind expression" is displayed.

[1192] Example prompt sentence:

[1193] Situation: Assisting elderly customers in stores

[1194] Attributes: Elderly woman, hard of hearing

[1195] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1196] Step 1:

[1197] The user inputs the situation and the other person's attributes.

[1198] The user inputs a specific situation (e.g., "serving an elderly person at a store") and the other person's attributes (e.g., "elderly woman, hard of hearing") through the terminal interface. The terminal then receives the input data and prepares it to be sent to the server.

[1199] Input: Situation and partner's attribute information

[1200] Output: The request data sent to the server

[1201] Step 2:

[1202] Sending a request from the device to the server

[1203] The terminal sends the information entered by the user to the server, which then starts generating a customized conversation simulator. Specifically, the terminal sends the situation and the attributes of the other party as prompts to the server.

[1204] Input: Request data

[1205] Output: Prompt data received by the server

[1206] Step 3:

[1207] The server generates a conversation simulator.

[1208] The server uses the received prompt data to generate a customized conversation simulator using a generative AI model (e.g., GPT-3 / 4). The AI ​​model generates a dialogue program based on the prompt data and returns the generated conversation simulator to the device.

[1209] Input: prompt data

[1210] Output: Generated conversation simulator

[1211] Step 4:

[1212] The terminal receives the conversation simulator and starts the conversation session.

[1213] The terminal starts a dialogue session with the user using the generated conversation simulator received from the server. The user converses with the user using voice or text. Specifically, the terminal displays the conversation simulator and the user responds.

[1214] Input: Generated conversation simulator

[1215] Output: Interaction data

[1216] Step 5:

[1217] Nonverbal data collection

[1218] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and sends this non-verbal data to a server after the conversation ends.

[1219] Input: User's facial expression and tone of voice data

[1220] Output: Collected nonverbal data

[1221] Step 6:

[1222] Transmission of interaction data and non-verbal data

[1223] The terminal transmits the collected interaction data and nonverbal data to the server, which receives and analyzes the data.

[1224] Input: Interaction data, nonverbal data

[1225] Output: Data used by the server for analysis

[1226] Step 7:

[1227] The server analyzes the data and generates feedback

[1228] The server analyzes the received interaction data and nonverbal data using generative AI models and data analysis tools. Based on the analysis results, it generates feedback for the user and sends it back to the device.

[1229] Input: Interaction data, nonverbal data

[1230] Output: Generated feedback

[1231] Step 8:

[1232] The device displays feedback

[1233] The device displays the feedback received from the server to the user, who can review the feedback and receive specific advice on how to improve their performance.

[1234] Input: Generated feedback

[1235] Output: Feedback that is displayed to the user

[1236] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1237] System Overview

[1238] This invention provides a conversation practice tool that allows users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Furthermore, by combining it with an emotion engine, it has the ability to recognize the user's emotions and provide feedback based on the results. This invention uses three entities: a server, a terminal, and a user.

[1239] Server Roles

[1240] The server is primarily responsible for performing the following functions:

[1241] 1. Receiving and analyzing information:

[1242] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[1243] 2. Creating a conversation simulator:

[1244] It uses generative AI (e.g., a GPT model) to generate a customized conversation simulator based on the information it receives, including prompts entered by the user.

[1245] 3. Model Transfer:

[1246] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[1247] 4. Data Analysis and Emotion Recognition:

[1248] The system analyzes conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[1249] 5. Feedback Generation:

[1250] Based on the analysis results and emotion recognition results, feedback is generated to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[1251] Device Role

[1252] The terminal is responsible for the interaction with the user and the communication with the server:

[1253] 1. Information input interface:

[1254] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[1255] 2. Submit your request:

[1256] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[1257] 3. Interfacing with the conversation simulator:

[1258] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[1259] 4. Nonverbal Data Collection:

[1260] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[1261] 5. Viewing Feedback:

[1262] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[1263] User Roles

[1264] The user typically performs the following actions:

[1265] 1. Enter your information:

[1266] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[1267] 2. Execute the conversation:

[1268] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[1269] 3. Review your feedback:

[1270] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[1271] Specific examples

[1272] For example:

[1273] 1. User Input:

[1274] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[1275] 2. Server Processing:

[1276] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1277] 3. Conversation session:

[1278] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[1279] 4. Data Collection and Analysis:

[1280] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[1281] 5. Providing Feedback:

[1282] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[1283] In this way, we realize a system that allows users to efficiently improve their communication skills. By using an emotion engine, we can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[1284] The processing flow will be explained below.

[1285] Step 1:

[1286] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[1287] Step 2:

[1288] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[1289] Step 3:

[1290] The server receives the request from the device and analyzes the situation and attribute information. For example, it extracts the situation "interview" and the attributes "middle-aged male, strict personality."

[1291] Step 4:

[1292] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[1293] Step 5:

[1294] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[1295] Step 6:

[1296] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[1297] Step 7:

[1298] The user interacts with the generated conversation simulator via a terminal, either through text or voice. For example, the simulator playing the role of an interviewer asks, "Please introduce yourself," and the user responds.

[1299] Step 8:

[1300] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[1301] Step 9:

[1302] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[1303] Step 10:

[1304] The server analyzes the received speech and non-verbal data and uses an emotion engine to recognize the user's emotions, for example, by analyzing facial expressions to determine whether the user is smiling.

[1305] Step 11:

[1306] The server analyzes the collected data to generate feedback about the user's emotions, such as providing specific advice like, "Your answers are accurate, but your facial expression seems stiff, so you should try to relax."

[1307] Step 12:

[1308] The server sends the generated feedback to the terminal, which displays it on the user interface, allowing the user to receive specific feedback to improve their performance.

[1309] Example 2

[1310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1311] In modern society, improving communication skills is considered important, but there are limited opportunities to practice them effectively. It is particularly difficult to practice nonverbal communication (facial expressions, tone of voice, etc.). It is also difficult to objectively evaluate the effectiveness of practice and obtain specific feedback. To address these challenges, there is a need for a system that provides practical, individualized practice and feedback.

[1312] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1313] In this invention, the server includes means for receiving information input by the user, means for generating a customized conversation simulator using a generative model, means for transferring the simulator model to the user's terminal, means for collecting conversation data with the conversation simulator, and means for analyzing the conversation data and nonverbal data, performing emotion recognition, and providing feedback, thereby enabling the user to objectively evaluate the effects of their practice and identify specific areas for improvement.

[1314] "Information entered by the user" refers to data that the user enters to the system to specify a particular situation or the attributes of the other party.

[1315] A "generative model" is an algorithm or program that a pre-trained artificial intelligence uses to generate natural language based on specific input.

[1316] A "customized conversation simulator" is a conversation agent that is generated by a generative model based on information input by the user and that corresponds to a specific situation or the attributes of the other party.

[1317] The "means for transferring the simulator model to the user's terminal" is a mechanism for transmitting the generated conversation simulator from the server to the terminal used by the user.

[1318] "Conversation data with the conversation simulator" is text or voice data generated when the user converses with the conversation simulator.

[1319] "Nonverbal data" refers to data that includes non-verbal communication elements such as a user's facial expression and tone of voice.

[1320] "Emotion recognition" is a technology that analyzes speech and non-verbal data to identify a user's emotional state.

[1321] "Feedback" is information that is generated based on analyzed data and includes advice and suggestions for improving the user's communication skills.

[1322] This invention is a system that provides users with a conversation practice tool to improve their communication skills. The system generates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Three entities are involved: a server, a terminal, and a user.

[1323] System Overview

[1324] Server Roles

[1325] The server mainly performs the following functions:

[1326] 1. Receiving and analyzing information:

[1327] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[1328] 2. Creating a conversation simulator:

[1329] The server uses a generative AI model (e.g., GPT-3) to generate a customized conversation simulator based on the received information. The prompt includes the information entered by the user. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality."

[1330] 3. Model Transfer:

[1331] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[1332] 4. Data Analysis and Emotion Recognition:

[1333] The server analyzes the conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[1334] 5. Feedback Generation:

[1335] Based on the analysis results and emotion recognition results, feedback is generated to be provided to the user. The feedback is advice based on the content of the conversation and nonverbal expressions. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is provided.

[1336] Device Role

[1337] The terminal is responsible for the interaction with the user and the communication with the server.

[1338] 1. Information input interface:

[1339] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or an input field with multiple-choice options.

[1340] 2. Submit your request:

[1341] The terminal sends the information entered by the user to the server, which starts the generation of the conversation simulator.

[1342] 3. Interfacing with the conversation simulator:

[1343] The terminal uses the generated conversation simulator received from the server to initiate a dialogue session with the user, providing an interface through which the user can converse via text or voice.

[1344] 4. Nonverbal Data Collection:

[1345] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[1346] 5. Viewing Feedback:

[1347] The device displays the feedback received from the server in a user interface, allowing the user to receive specific advice on how to improve their performance.

[1348] User Roles

[1349] The user mainly performs the following actions:

[1350] 1. Enter your information:

[1351] The user inputs a specific situation and the attributes of the other person through the terminal interface. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[1352] 2. Execute the conversation:

[1353] The user interacts with the generated conversation simulator by speaking or texting, for example, answering hypothetical questions as practice for an interview.

[1354] 3. Review your feedback:

[1355] The user checks the feedback provided by the server as an analysis result and improves their own communication skills based on it.

[1356] Specific examples

[1357] Specific examples are shown below.

[1358] 1. User Input:

[1359] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[1360] 2. Server Processing:

[1361] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1362] 3. Conversation session:

[1363] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[1364] 4. Data Collection and Analysis:

[1365] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[1366] 5. Providing Feedback:

[1367] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[1368] Technology used and data processing

[1369] This system includes the following technologies and data processing:

[1370] Hardware: The device is equipped with a camera and microphone, which allows for real-time collection of the user's facial expressions and tone of voice.

[1371] Software: Generative AI models (e.g., GPT-3) are used to generate the conversation simulator, and an emotion engine is used to analyze nonverbal data and employ algorithms to recognize user emotions.

[1372] Data Processing: Techniques are used to collect and analyze speech and non-verbal data. The analyzed data provides important information for providing feedback to users.

[1373] Through this system, users can effectively improve their communication skills. By utilizing the emotion engine, users can receive more detailed and useful feedback.

[1374] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1375] Step 1:

[1376] The terminal provides an interface for users to access the system. Here, users select a situation (e.g., an interview) and enter the other party's attributes (e.g., middle-aged man, strict personality). The input data is saved in text format and sent from the terminal to the server in JSON format.

[1377] Input: Situations and attributes entered by the user

[1378] Output: JSON format data (situations and attributes)

[1379] Step 2:

[1380] The server receives and analyzes the JSON-formatted data sent from the device. As a result of the analysis, situation and attribute information is extracted. Based on this information, the server generates a prompt for the generative AI model.

[1381] Input: JSON format data (situations and attributes)

[1382] Output: Prompt sentence for the generative AI model (e.g., "Situation: Interview, Attributes: Middle-aged male, Strict personality")

[1383] Step 3:

[1384] The server sends the prompt sentence to a generative AI model (e.g., GPT-3) to generate a customized conversation simulator, which is returned to the server as text data.

[1385] Input: Prompt sentence for the generative AI model

[1386] Output: Text data of the generated conversation simulator

[1387] Step 4:

[1388] The server transfers the generated conversation simulator to the terminal, which displays the conversation simulator on its interface and starts a dialogue with the user.

[1389] Input: Text data of the generated conversation simulator

[1390] Output: Interface showing the conversation simulator

[1391] Step 5:

[1392] The user interacts with the conversation simulator and inputs responses. The device collects the conversation data in real time. It also uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice.

[1393] Input: User responses, facial expressions, and tone of voice

[1394] Output: Speech and non-verbal data

[1395] Step 6:

[1396] The device transmits the collected speech and non-verbal data in real time to a server, which receives the data and performs data analysis using NLP technology and an emotion engine.

[1397] Input: Speech and non-verbal data

[1398] Output: Analysis results and emotion recognition results

[1399] Step 7:

[1400] The server generates feedback to provide to the user based on the analysis and emotion recognition results, including specific advice based on the conversation content and nonverbal expressions.

[1401] Input: Analysis results and emotion recognition results

[1402] Output: Generated feedback

[1403] Step 8:

[1404] The server transmits the generated feedback to the terminal, which displays the feedback on a user interface for the user to confirm.

[1405] Input: Generated feedback

[1406] Output: Feedback displayed in the user interface

[1407] (Application example 2)

[1408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1409] To improve their communication skills, users need to be able to practice effectively and practically based on specific situations and the attributes of the people they are interacting with. However, conventional methods have faced challenges in providing practical training for specific situations and in obtaining effective feedback. This problem is particularly evident in training salespeople in brick-and-mortar stores to improve their customer service skills.

[1410] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information input by a user, means for generating a customized conversation simulator based on the information, means for collecting conversation data with the conversation simulator, means for analyzing the conversation data and nonverbal data and providing feedback, means for collecting the nonverbal data by face recognition means, and means for supporting the improvement of the user's customer interaction skills through the feedback. This enables the user to undergo practical training based on specific situations and the attributes of the other party and receive effective feedback.

[1411] A "user" is a person who utilizes the system to input information and receive feedback to improve their communication skills.

[1412] The "means for receiving information" is an interface or device for acquiring information input by the user about the situation and the attributes of the other party.

[1413] The "means for generating a conversation simulator" refers to a technique or device for creating a conversation simulator that corresponds to a specific situation or the attributes of the other person, based on information input by the user.

[1414] The "means for collecting conversation data" refers to a device or technology for recording the content of the conversation that takes place between the user and the generated conversation simulator.

[1415] "Nonverbal data" refers to data that includes information other than words, such as a user's facial expression or tone of voice.

[1416] A "means for providing feedback" is a device or technology that analyzes the collected speech and non-verbal data and indicates to the user areas for improvement in their communication skills.

[1417] "Facial recognition means" refers to technology or systems that use devices such as cameras to identify and analyze a user's facial expressions.

[1418] "Customer service skills" refers to the abilities and techniques that sales staff in physical stores and other locations have to respond appropriately to customers.

[1419] "Means for supporting improvement of users' customer service skills" are technologies and systems that include feedback and advice provided to enable users to provide better customer service.

[1420] This invention provides a training system for sales staff in brick-and-mortar stores to improve their customer service skills. The system operates using three entities: a user, a server, and a terminal. The roles and specific operations of each of these entities are described below.

[1421] Server Roles

[1422] The server performs the following functions:

[1423] 1. Receiving and analyzing information:

[1424] The server receives the situation and the other party's attributes sent from the terminal. For example, the server receives the situation "handling a complaint" and the attributes "elderly woman, angry."

[1425] 2. Creating a conversation simulator:

[1426] The server uses a generative AI model (e.g., OpenAI GPT-3.5) to generate a customized conversation simulator based on the information it receives, including prompts entered by the user.

[1427] Example prompt sentence:

[1428] Situation: Complaint handling, Attributes: Elderly woman, angry

[1429] 3. Model Transfer:

[1430] The server transfers the generated conversation simulator model to the terminal so that the user can use it.

[1431] 4. Data Analysis and Emotion Recognition:

[1432] The server analyzes the conversational data and non-verbal data sent from the device and recognizes the user's emotions using an emotion recognition engine, which includes facial expression analysis and tone of voice analysis.

[1433] 5. Feedback Generation:

[1434] The server generates feedback to provide to the user based on the analysis results and emotion recognition results, including advice based on the content of the conversation and nonverbal expressions.

[1435] Device Role

[1436] The terminal is responsible for the following functions:

[1437] 1. Information input interface:

[1438] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or a multiple-choice input field.

[1439] 2. Submit your request:

[1440] The terminal starts generating a conversation simulator by transmitting the information input by the user to the server.

[1441] 3. Interfacing with the conversation simulator:

[1442] The terminal initiates an interactive session with the user using the generated conversation simulator received from the server, and provides an interface through which the user can conduct a conversation by text or voice.

[1443] 4. Nonverbal Data Collection:

[1444] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[1445] 5. Viewing Feedback:

[1446] The terminal displays the feedback received from the server on a user interface, allowing the user to receive specific advice on how to improve their performance.

[1447] User Roles

[1448] The user performs the following actions:

[1449] 1. Enter your information:

[1450] Through the terminal interface, specific situations and the attributes of the other party are input. For example, the situation of handling a complaint and the expected attributes of the customer at that time (elderly woman, angry) can be input.

[1451] 2. Execute the conversation:

[1452] The user interacts with the generated conversation simulator by speaking or texting. For example, the user can practice how to respond to a complaint.

[1453] 3. Review your feedback:

[1454] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[1455] Specific examples

[1456] As a concrete example, consider the following flow:

[1457] 1. The user starts up the terminal, selects the situation "Handling a complaint," and enters "elderly woman, angry" as the attribute.

[1458] 2. The server receives the information sent from the device and sends a prompt to the generative AI model with the following content: "Situation: Handling a complaint, Attributes: Elderly woman, Angry," and generates a customized conversation simulator.

[1459] 3. The user starts a conversation on the terminal using the conversation simulator returned from the server. The simulator plays the role of a customer and makes a complaint, and the user responds.

[1460] 4. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them to the server along with the content of the conversation.

[1461] 5. The server analyzes the collected data and recognizes the user's emotions using an emotion recognition engine.

[1462] 6. The device receives the feedback generated by the server and displays specific advice to the user, such as, "The content of your complaint response is good, but it would be better if you spoke with a calmer expression."

[1463] In this way, the system allows users to efficiently improve their customer service skills in physical stores. By utilizing an emotion recognition engine, the system can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[1464] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1465] Step 1:

[1466] The user starts up the terminal and inputs information about the situation and the other party's attributes. For example, the situation "handling a complaint" and the attributes "elderly woman, angry" are entered into the text boxes. This starts the information input interface.

[1467] Input: situation (e.g., handling a complaint), other party's attributes (e.g., elderly woman, angry)

[1468] Output: The information entered

[1469] Step 2:

[1470] The terminal transmits the information input by the user to the server, where it arrives as a customized prompt.

[1471] Input: Information entered by the user

[1472] Output: Request to server

[1473] Step 3:

[1474] The server analyzes the received information and generates prompt sentences. The prompt sentences are used to generate a customized conversation simulator for a generative AI model (e.g., OpenAI GPT-3.5).

[1475] Input: Prompt sentence (e.g., Situation: Handling a complaint, Attributes: Elderly woman, Angry)

[1476] Data processing: Creating a conversation simulator using a generative AI model

[1477] Output: A customized conversation simulator

[1478] Step 4:

[1479] The server transfers the generated conversation simulator to the terminal, which then starts a dialogue session with the user based on the received simulator.

[1480] Input: A customized conversation simulator

[1481] Output: Transfer of conversation simulator to terminal

[1482] Step 5:

[1483] The user initiates a dialogue with a customized conversation simulator displayed on the terminal, and the user converses with the simulator using voice or text.

[1484] Input: A customized conversation simulator and the user's conversation content

[1485] Output: Conversation progress

[1486] Step 6:

[1487] The device uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice, which is then sent to a server in real time along with the content of the conversation.

[1488] Input: User's facial expression data, voice data

[1489] Data Processing: Collection and Transmission of Nonverbal Data

[1490] Output: Nonverbal and conversational data

[1491] Step 7:

[1492] The server analyzes the received non-verbal data and conversation data, and recognizes the user's emotions using an emotion recognition engine. It also analyzes facial expressions and voice using a face recognition means.

[1493] Input: Nonverbal data, conversation data

[1494] Data Calculation: Analysis by Emotion Recognition Engine

[1495] Output: Analysis results (user emotion recognition results)

[1496] Step 8:

[1497] The server generates feedback based on the analysis results, including specific advice such as, "The content of your complaint handling is good, but it would be better if you spoke with a calmer expression."

[1498] Input: Analysis results (user emotion recognition results)

[1499] Data Calculation: Feedback Generation

[1500] Output: Feedback content

[1501] Step 9:

[1502] The terminal displays the generated feedback on a user interface, and the user can check the feedback and understand areas for improvement in their communication skills.

[1503] Input: Feedback

[1504] Output: Feedback display to the user

[1505] As a result, this system allows users to effectively practice conversation skills in specific situations and receive feedback, thereby improving their customer service skills.

[1506] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1507] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1508] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1509] [Fourth embodiment]

[1510] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1511] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1512] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1513] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1514] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1515] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1516] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1517] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1518] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1519] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1520] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1521] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1522] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1523] System Overview

[1524] This invention provides a conversation practice tool for users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and non-verbal data, and provides feedback. This invention uses three entities: a server, a terminal, and a user.

[1525] Server Roles

[1526] The server is primarily responsible for performing the following functions:

[1527] 1. Receiving and analyzing information:

[1528] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[1529] 2. Creating a conversation simulator:

[1530] Generative AI (e.g., a GPT model) is used to generate a customized conversation simulator based on the received information. The received information is used as a prompt to create an appropriate dialogue model.

[1531] 3. Model Transfer:

[1532] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[1533] 4. Data analysis and feedback generation:

[1534] The system analyzes conversational and nonverbal data sent from the device and generates feedback to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[1535] Device Role

[1536] The terminal is responsible for the interaction with the user and the communication with the server:

[1537] 1. Information input interface:

[1538] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[1539] 2. Submit your request:

[1540] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[1541] 3. Interfacing with the conversation simulator:

[1542] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[1543] 4. Nonverbal Data Collection:

[1544] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[1545] 5. Viewing Feedback:

[1546] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[1547] User Roles

[1548] The user typically performs the following actions:

[1549] 1. Enter your information:

[1550] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[1551] 2. Execute the conversation:

[1552] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[1553] 3. Review your feedback:

[1554] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[1555] Specific examples

[1556] For example:

[1557] 1. User Input:

[1558] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[1559] 2. Server Processing:

[1560] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1561] 3. Conversation session:

[1562] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[1563] 4. Data Collection and Analysis:

[1564] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to a server, which analyzes them to generate feedback and send it back to the device.

[1565] 5. Viewing Feedback:

[1566] The device displays the feedback received from the server to the user, such as advice like, "Your answers are accurate, but your facial expression is stiff. You'll make a better impression if you relax."

[1567] In this way, a system is realized that allows users to efficiently improve their communication skills.

[1568] The processing flow will be explained below.

[1569] Step 1:

[1570] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[1571] Step 2:

[1572] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[1573] Step 3:

[1574] The server receives the request sent from the terminal and analyzes the situation and attribute information.

[1575] Step 4:

[1576] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[1577] Step 5:

[1578] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[1579] Step 6:

[1580] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[1581] Step 7:

[1582] The user interacts with the generated conversation simulator via a terminal using text or voice.

[1583] Step 8:

[1584] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[1585] Step 9:

[1586] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[1587] Step 10:

[1588] The server analyzes the received speech and non-verbal data and generates feedback regarding the user's communication.

[1589] Step 11:

[1590] The server sends the generated feedback to the device, which displays it in the user interface, allowing the user to receive specific advice on how to improve their communication skills.

[1591] In this way, the system provides an environment in which users can actually attempt a dialogue, evaluate their performance from multiple angles, and provide feedback, allowing users to efficiently improve their communication skills.

[1592] Example 1

[1593] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1594] Communication skills are extremely important in modern society, but there are limited tools available to effectively acquire and improve them. It's particularly difficult to objectively identify one's weaknesses and areas for improvement, making efficient training difficult. Furthermore, existing tools lack a system that comprehensively analyzes a variety of data, such as the content of conversations, facial expressions, and tone of voice, and provides feedback, limiting the ability to fully improve users' communication skills.

[1595] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1596] In this invention, the server includes means for receiving information input by a user, means for generating prompts using a generative AI model and generating a customized conversation simulator, means for collecting conversation data and nonverbal data, means for analyzing the conversation data and nonverbal data and generating feedback, and means for displaying the feedback to the user. This allows the user to have their communication skills comprehensively evaluated and be given specific indications of areas for improvement, thereby enabling them to efficiently improve their skills.

[1597] "Means for receiving information entered by the user" refers to the function of the terminal providing an interface for the user to input information such as the situation and the attributes of the other party, and transmitting that information to the server.

[1598] "Means for generating prompts using a generative AI model and generating a customized conversation simulator" refers to a function that sends prompts to a generative AI model based on user information received by the server, thereby generating a conversation simulator that is suitable for a specific situation or the attributes of the other party.

[1599] "Means for collecting conversational data and non-verbal data" refers to the function of the device to record and save in real time using a camera or microphone voice data generated during a conversation with a user, as well as non-verbal data such as the user's facial expressions and tone of voice.

[1600] "Means for analyzing conversational data and non-verbal data and generating feedback" refers to a function in which the server uses analytical technology to evaluate collected conversational data and non-verbal data and generate specific advice and areas for improvement to improve the user's communication skills.

[1601] "Means for displaying feedback to the user" refers to a function that allows the terminal to display feedback information sent from the server on the user interface, allowing the user to check it and use it to improve their own communication skills.

[1602] A "situation" refers to a specific situation or scene that a user inputs to the conversation simulator, and includes specific situations such as an interview or a presentation.

[1603] "Partner attributes" refer to the characteristics and personality of the conversation partner that the user envisions in the conversation simulator, and include, for example, age, gender, and personality tendencies.

[1604] A "generative AI model" is an artificial intelligence model that provides generated text or dialogue based on input prompts, and includes, for example, a natural language processing model.

[1605] A "prompt" is a string of characters or data given to a generative AI model as an instruction or input, and is basic information used to determine the content of the generated conversation simulator.

[1606] A "conversation simulator" is a program or system that simulates dialogue based on specific situations and the attributes of the other party, generated by a generative AI model.

[1607] This invention is a conversation practice tool for users to improve their communication skills, and is a system that generates an AI conversation simulator based on information input by the user, collects and analyzes conversation data and non-verbal data, and provides feedback, thereby enabling users to improve their communication skills efficiently.

[1608] Server Roles

[1609] The server uses the following hardware and software:

[1610] Hardware: High-performance processors, memory, and storage devices

[1611] Software: Generative AI models (e.g., GPT-3)

[1612] The server first receives information about the situation and the other party's attributes sent from the device. For example, it receives information such as "Situation: Interview, Attributes: Middle-aged male, Strict personality." Next, it references the generative AI model based on the received information and generates an appropriate prompt. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality." This prompt is sent to the generative AI model, which generates a customized conversation simulator. The server then transfers the generated conversation simulator to the device.

[1613] The server also collects and analyzes speech and non-verbal data sent from the device, using voice and face recognition technology, and generates feedback to be provided to the user based on the analysis results and sends it to the device.

[1614] Device Role

[1615] The terminal is responsible for the interaction with the user and the communication with the server:

[1616] Hardware: Camera, microphone, display, input devices (keyboard and touchscreen)

[1617] Software: User Interface Application

[1618] The terminal provides an interface where the user can input information about the situation and the other person's attributes. For example, using text boxes and drop-down menus, the user can select "interview" and input "middle-aged male, strict personality." The information entered by the user is sent to the server, which then requests the generation of a conversation simulator.

[1619] The device initiates a dialogue session with the user using the conversation simulator received from the server. It provides an interface for the user to converse via voice or text, and uses a camera and microphone to collect the user's facial expressions and tone of voice in real time. The collected data is sent to the server for analysis and feedback generation.

[1620] Once the feedback is sent from the server, the device displays it to the user, and specific advice is provided in the user interface, allowing the user to analyze their performance and take action to improve.

[1621] User Roles

[1622] Users will use the system to improve their communication skills by:

[1623] First, a user inputs a specific situation and the attributes of the other person through the device interface. Then, the user engages in a dialogue with the generated conversation simulator. For example, in an interview simulator, the user responds to questions from the interviewer via voice or text. After the session ends, the user can review the feedback provided by the server and use it to improve their communication skills.

[1624] Specific examples

[1625] The user starts up the device, selects an interview situation, and enters "middle-aged male, strict personality" as the attributes. The device sends this information to the server, which uses a generative AI model to generate a prompt sentence: "Situation: Interview, Attributes: Middle-aged male, strict personality," and a conversation simulator is created.

[1626] The user begins a conversation with the simulator displayed on the device. The simulator acts as an interviewer, asking questions, and the user answers. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server. The server analyzes this, generates feedback, and sends it back to the device. The device displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your expression is stiff. You would make a better impression if you relaxed" may be displayed.

[1627] Through these steps, users can effectively improve their communication skills. The system also provides users with objective feedback that can be used for self-evaluation.

[1628] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1629] System program processing flow

[1630] Step 1:

[1631] The user uses the device interface to input the situation and the attributes of the other person. For example, they might select "interview" and input "middle-aged man, strict personality" as the attributes. This input is done using text boxes and drop-down menus. The input is the user's selection and input data. The output is the input information being saved on the device.

[1632] Step 2:

[1633] The terminal sends the situation and attribute information entered by the user to the server. This transmission is performed via an HTTP request. Specifically, the terminal sends the information "Situation: Interview, Attributes: Middle-aged male, Strict personality" to the server in JSON format. The input is the situation and attribute data from the user. The output is the server receiving this data.

[1634] Step 3:

[1635] The server generates a prompt for the generative AI model based on the received information and generates a conversation simulator. Using a generative AI model (e.g., GPT-3), it generates a prompt sentence such as "Situation: Interview, Attributes: Middle-aged male, Strict personality" and sends it to the AI ​​model. The input is the situation and attribute information sent by the user. The output is the generated conversation simulator.

[1636] Step 4:

[1637] The server transfers the generated conversation simulator to the device. This transfer is also performed via an HTTP request. The input is the generated conversation simulator. The output is the device receiving the conversation simulator. Specifically, the server sends the conversation simulator to the device in JSON format.

[1638] Step 5:

[1639] The terminal provides the received conversation simulator to the user and starts a conversation session. The user converses using voice or text. The terminal interface displays the dialogue content to the user and provides an input field. The input is the dialogue input between the generated conversation simulator and the user. The output is the user's dialogue history and dialogue content generated in real time.

[1640] Step 6:

[1641] The device collects conversational and non-verbal data. It uses a camera and microphone to record the user's facial expressions and tone of voice, and saves them as a log. The input is the user's facial expression data and voice data. The output is the collected log data.

[1642] Step 7:

[1643] The device sends the collected data to the server. The data includes conversational content and nonverbal data and is sent to the server in JSON format. The input is the collected conversational data and nonverbal data. The output is that the server receives this data.

[1644] Step 8:

[1645] The server analyzes the received data, using natural language processing and facial recognition technologies to evaluate the content of the conversation and non-verbal expressions. The input is the collected conversation data and non-verbal data. The output is feedback as the analysis results.

[1646] Step 9:

[1647] The server generates feedback based on the analysis results and sends it to the device. The generated feedback is sent in JSON format. The input is the analysis results. The output is that the device receives the feedback.

[1648] Step 10:

[1649] The terminal displays the feedback received from the server on the user interface. The feedback includes an evaluation of the user's conversation content and advice on non-verbal expressions. Specific areas for improvement and advice are highlighted. The input is the feedback from the server. The output is the user being able to visually confirm the feedback.

[1650] This realizes a system that allows users to efficiently improve their communication skills.

[1651] (Application example 1)

[1652] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1653] In modern retail stores, sales associates are expected to consistently provide high-quality customer service. However, for new sales associates or those with uncertain skills, simulating actual customer interactions and improving their skills is a difficult task. In particular, because they are required to respond to diverse customer attributes and situations, real-time feedback is lacking. This can result in a decline in customer satisfaction, which can have a negative impact on the store's reputation and sales. There is a need for a system that can solve these issues and effectively improve sales associates' customer service skills.

[1654] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1655] In this invention, the server includes means for generating a customized conversation simulator based on a situation and attributes of the other party input by a user, interface means for collecting interaction data and nonverbal data with the conversation simulator, and means for analyzing the interaction data and nonverbal data and using the generative AI model to provide feedback, thereby enabling store clerks to simulate customer interaction situations in real time and improve their customer service skills based on the obtained feedback.

[1656] A "situation" is information that a user inputs to simulate a specific situation or scene.

[1657] "Partner attributes" is information that indicates the characteristics and personality of the person you are likely to be speaking with in a given situation.

[1658] A "customized conversation simulator" is a dialogue program that is generated based on the situation and attributes of the other party input by the user.

[1659] "Interaction data" refers to voice and text data exchanged between the user and the conversation simulator.

[1660] "Nonverbal data" refers to non-verbal information such as a user's facial expression, tone of voice, and posture.

[1661] A "generative AI model" is an artificial intelligence model that generates customized dialogue simulators based on user input information, or analyzes collected data and provides feedback.

[1662] "Customer service" refers to the customer service provided by retail store staff to actual customers.

[1663] "Feedback" is information for improvement or evaluation provided based on user interaction and non-verbal data.

[1664] This invention provides a system for improving the customer service skills of retail store staff. The system is composed of three main components: a server, a terminal, and a user.

[1665] Server Roles

[1666] The server is responsible for performing the following main functions:

[1667] 1. Receiving and analyzing information: The system receives information about the situation and the other party's attributes entered by the user on the terminal. For example, it receives the situation "serving an elderly person at a store" and the attributes "elderly woman, hard of hearing."

[1668] 2. Conversation simulator generation: A generative AI model (e.g., GPT-3 / 4 model) is used to generate a customized conversation simulator based on the received information. The AI ​​model uses the received information as a prompt to create an appropriate dialogue program.

[1669] 3. Analysis of interaction data and nonverbal data: Analyze the conversation data and nonverbal data sent from the device. Generate automatically generated feedback based on the analyzed data and send it to the device.

[1670] Device Role

[1671] The terminal is responsible for the interaction with the user and the communication with the server:

[1672] 1. Information input interface: Provide an interface for users to input information. Specifically, provide text boxes and multiple-choice input fields to allow users to enter the situation and the other person's attributes.

[1673] 2. Sending a request: The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[1674] 3. Interface with the conversation simulator: Start a dialogue session with the user using the generated conversation simulator received from the server. Provide an interface that allows the user to have a conversation via text or voice.

[1675] 4. Non-verbal data collection: A camera and microphone are used to collect the user's facial expressions and tone of voice in real time, which are then analyzed on the server.

[1676] 5. Displaying Feedback: The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[1677] User Roles

[1678] The user typically performs the following actions:

[1679] 1. Information input: A specific situation and the attributes of the other party are input through the terminal interface. For example, a situation involving dealing with an elderly person at a store, and the attributes of the elderly woman and her hearing impairment are input.

[1680] 2. Execution of conversation: Dialogue is carried out with the generated conversation simulator. At this time, the conversation is exchanged with the simulator via voice or text. For example, specific conversation content is confirmed as practice for dealing with elderly people.

[1681] 3. Check the feedback: Check the feedback provided by the server as an analysis result and improve your customer service skills based on it.

[1682] Specific examples

[1683] For example:

[1684] 1. User input: The user starts up the terminal and inputs the situation as "serving elderly people at a store" and the attributes as "elderly woman, hard of hearing."

[1685] 2. Server processing: Receives the information sent from the device, and generates a prompt for the generation AI based on the following: "Situation: Serving elderly customers at a store, Attributes: Elderly woman, hard of hearing," and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1686] 3. Conversation session: The user begins a conversation with the conversation simulator displayed on the terminal. The simulator plays the role of an elderly customer, prompting the user with questions and responses, and the user responds.

[1687] 4. Data collection and analysis: The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the conversation to the server. The server analyzes them and generates feedback, which is sent back to the device.

[1688] 5. Displaying feedback: The device displays the feedback received from the server to the user. For example, advice such as "You should speak more clearly. Also, it is important to treat the elderly with a kind expression" is displayed.

[1689] Example prompt sentence:

[1690] Situation: Assisting elderly customers in stores

[1691] Attributes: Elderly woman, hard of hearing

[1692] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1693] Step 1:

[1694] The user inputs the situation and the other person's attributes.

[1695] The user inputs a specific situation (e.g., "serving an elderly person at a store") and the other person's attributes (e.g., "elderly woman, hard of hearing") through the terminal interface. The terminal then receives the input data and prepares it to be sent to the server.

[1696] Input: Situation and partner's attribute information

[1697] Output: The request data sent to the server

[1698] Step 2:

[1699] Sending a request from the device to the server

[1700] The terminal sends the information entered by the user to the server, which then starts generating a customized conversation simulator. Specifically, the terminal sends the situation and the attributes of the other party as prompts to the server.

[1701] Input: Request data

[1702] Output: Prompt data received by the server

[1703] Step 3:

[1704] The server generates a conversation simulator.

[1705] The server uses the received prompt data to generate a customized conversation simulator using a generative AI model (e.g., GPT-3 / 4). The AI ​​model generates a dialogue program based on the prompt data and returns the generated conversation simulator to the device.

[1706] Input: prompt data

[1707] Output: Generated conversation simulator

[1708] Step 4:

[1709] The terminal receives the conversation simulator and starts the conversation session.

[1710] The terminal starts a dialogue session with the user using the generated conversation simulator received from the server. The user converses with the user using voice or text. Specifically, the terminal displays the conversation simulator and the user responds.

[1711] Input: Generated conversation simulator

[1712] Output: Interaction data

[1713] Step 5:

[1714] Nonverbal data collection

[1715] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and sends this non-verbal data to a server after the conversation ends.

[1716] Input: User's facial expression and tone of voice data

[1717] Output: Collected nonverbal data

[1718] Step 6:

[1719] Transmission of interaction data and non-verbal data

[1720] The terminal transmits the collected interaction data and nonverbal data to the server, which receives and analyzes the data.

[1721] Input: Interaction data, nonverbal data

[1722] Output: Data used by the server for analysis

[1723] Step 7:

[1724] The server analyzes the data and generates feedback

[1725] The server analyzes the received interaction data and nonverbal data using generative AI models and data analysis tools. Based on the analysis results, it generates feedback for the user and sends it back to the device.

[1726] Input: Interaction data, nonverbal data

[1727] Output: Generated feedback

[1728] Step 8:

[1729] The device displays feedback

[1730] The device displays the feedback received from the server to the user, who can review the feedback and receive specific advice on how to improve their performance.

[1731] Input: Generated feedback

[1732] Output: Feedback that is displayed to the user

[1733] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1734] System Overview

[1735] This invention provides a conversation practice tool that allows users to improve their communication skills. The system creates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Furthermore, by combining it with an emotion engine, it has the ability to recognize the user's emotions and provide feedback based on the results. This invention uses three entities: a server, a terminal, and a user.

[1736] Server Roles

[1737] The server is primarily responsible for performing the following functions:

[1738] 1. Receiving and analyzing information:

[1739] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[1740] 2. Creating a conversation simulator:

[1741] It uses generative AI (e.g., a GPT model) to generate a customized conversation simulator based on the information it receives, including prompts entered by the user.

[1742] 3. Model Transfer:

[1743] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[1744] 4. Data Analysis and Emotion Recognition:

[1745] The system analyzes conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[1746] 5. Feedback Generation:

[1747] Based on the analysis results and emotion recognition results, feedback is generated to provide to the user, which is advice based on the content of the conversation and nonverbal expressions.

[1748] Device Role

[1749] The terminal is responsible for the interaction with the user and the communication with the server:

[1750] 1. Information input interface:

[1751] An interface is provided for the user to input information about the situation and the other party's attributes. For example, a text box or multiple-choice input field is provided.

[1752] 2. Submit your request:

[1753] The information entered by the user is sent to the server, which starts the generation of the conversation simulator.

[1754] 3. Interfacing with the conversation simulator:

[1755] Initiate an interactive session with the user using the generated conversation simulator received from the server, and provide an interface through which the user can conduct a conversation via text or voice.

[1756] 4. Nonverbal Data Collection:

[1757] It uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is then analyzed on a server.

[1758] 5. Viewing Feedback:

[1759] The feedback received from the server is displayed in the user interface, allowing the user to receive specific advice on how to improve their performance.

[1760] User Roles

[1761] The user typically performs the following actions:

[1762] 1. Enter your information:

[1763] Through the terminal interface, users input specific situations and the attributes of the other person. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[1764] 2. Execute the conversation:

[1765] Dialogue with the generated conversation simulator is carried out by exchanging voice or text with the simulator, for example, answering hypothetical questions as practice for an interview.

[1766] 3. Review your feedback:

[1767] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[1768] Specific examples

[1769] For example:

[1770] 1. User Input:

[1771] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[1772] 2. Server Processing:

[1773] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1774] 3. Conversation session:

[1775] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[1776] 4. Data Collection and Analysis:

[1777] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[1778] 5. Providing Feedback:

[1779] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[1780] In this way, we realize a system that allows users to efficiently improve their communication skills. By using an emotion engine, we can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[1781] The processing flow will be explained below.

[1782] Step 1:

[1783] Through the terminal interface, the user inputs a specific situation and the other person's attributes, such as "interview" and "middle-aged man, strict personality."

[1784] Step 2:

[1785] The device formats the situation and attribute information entered by the user in JSON format and sends a request to the server. For example, the request might look like this: {"situation": "Interview", "attributes": "Middle-aged man, strict personality"}.

[1786] Step 3:

[1787] The server receives the request from the device and analyzes the situation and attribute information. For example, it extracts the situation "interview" and the attributes "middle-aged male, strict personality."

[1788] Step 4:

[1789] The server uses a generative AI (e.g., a GPT model) based on the analyzed information to generate an appropriate conversation simulator, whose prompts include the information entered by the user.

[1790] Step 5:

[1791] The server transfers the customized conversation simulator model created by the generation AI to the terminal.

[1792] Step 6:

[1793] The terminal analyzes the conversation simulator model received from the server and displays it on a user interface, through which the user can start a dialogue.

[1794] Step 7:

[1795] The user interacts with the generated conversation simulator via a terminal, either through text or voice. For example, the simulator playing the role of an interviewer asks, "Please introduce yourself," and the user responds.

[1796] Step 8:

[1797] The device uses a camera and microphone to capture the user's facial expressions and tone of voice during a conversation, which are recorded as nonverbal data.

[1798] Step 9:

[1799] The collected conversational data and non-verbal data are transmitted from the terminal to a server.

[1800] Step 10:

[1801] The server analyzes the received speech and non-verbal data and uses an emotion engine to recognize the user's emotions, for example, by analyzing facial expressions to determine whether the user is smiling.

[1802] Step 11:

[1803] The server analyzes the collected data to generate feedback about the user's emotions, such as providing specific advice like, "Your answers are accurate, but your facial expression seems stiff, so you should try to relax."

[1804] Step 12:

[1805] The server sends the generated feedback to the terminal, which displays it on the user interface, allowing the user to receive specific feedback to improve their performance.

[1806] Example 2

[1807] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1808] In modern society, improving communication skills is considered important, but there are limited opportunities to practice them effectively. It is particularly difficult to practice nonverbal communication (facial expressions, tone of voice, etc.). It is also difficult to objectively evaluate the effectiveness of practice and obtain specific feedback. To address these challenges, there is a need for a system that provides practical, individualized practice and feedback.

[1809] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1810] In this invention, the server includes means for receiving information input by the user, means for generating a customized conversation simulator using a generative model, means for transferring the simulator model to the user's terminal, means for collecting conversation data with the conversation simulator, and means for analyzing the conversation data and nonverbal data, performing emotion recognition, and providing feedback, thereby enabling the user to objectively evaluate the effects of their practice and identify specific areas for improvement.

[1811] "Information entered by the user" refers to data that the user enters to the system to specify a particular situation or the attributes of the other party.

[1812] A "generative model" is an algorithm or program that a pre-trained artificial intelligence uses to generate natural language based on specific input.

[1813] A "customized conversation simulator" is a conversation agent that is generated by a generative model based on information input by the user and that corresponds to a specific situation or the attributes of the other party.

[1814] The "means for transferring the simulator model to the user's terminal" is a mechanism for transmitting the generated conversation simulator from the server to the terminal used by the user.

[1815] "Conversation data with the conversation simulator" is text or voice data generated when the user converses with the conversation simulator.

[1816] "Nonverbal data" refers to data that includes non-verbal communication elements such as a user's facial expression and tone of voice.

[1817] "Emotion recognition" is a technology that analyzes speech and non-verbal data to identify a user's emotional state.

[1818] "Feedback" is information that is generated based on analyzed data and includes advice and suggestions for improving the user's communication skills.

[1819] This invention is a system that provides users with a conversation practice tool to improve their communication skills. The system generates an AI conversation simulator based on information entered by the user, collects and analyzes conversational and nonverbal data, and provides feedback. Three entities are involved: a server, a terminal, and a user.

[1820] System Overview

[1821] Server Roles

[1822] The server mainly performs the following functions:

[1823] 1. Receiving and analyzing information:

[1824] The server receives the situation and the other party's attributes sent from the terminal. For example, it receives the situation "interview" and the attributes "middle-aged man, strict personality."

[1825] 2. Creating a conversation simulator:

[1826] The server uses a generative AI model (e.g., GPT-3) to generate a customized conversation simulator based on the received information. The prompt includes the information entered by the user. An example of a prompt sentence is "Situation: Interview, Attributes: Middle-aged male, Strict personality."

[1827] 3. Model Transfer:

[1828] The generated conversation simulator model is transferred to a terminal so that it can be used by the user.

[1829] 4. Data Analysis and Emotion Recognition:

[1830] The server analyzes the conversational and non-verbal data sent from the device and uses an emotion engine to recognize the user's emotions, including facial expression analysis and tone of voice analysis.

[1831] 5. Feedback Generation:

[1832] Based on the analysis results and emotion recognition results, feedback is generated to be provided to the user. The feedback is advice based on the content of the conversation and nonverbal expressions. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is provided.

[1833] Device Role

[1834] The terminal is responsible for the interaction with the user and the communication with the server.

[1835] 1. Information input interface:

[1836] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or an input field with multiple-choice options.

[1837] 2. Submit your request:

[1838] The terminal sends the information entered by the user to the server, which starts the generation of the conversation simulator.

[1839] 3. Interfacing with the conversation simulator:

[1840] The terminal uses the generated conversation simulator received from the server to initiate a dialogue session with the user, providing an interface through which the user can converse via text or voice.

[1841] 4. Nonverbal Data Collection:

[1842] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[1843] 5. Viewing Feedback:

[1844] The device displays the feedback received from the server in a user interface, allowing the user to receive specific advice on how to improve their performance.

[1845] User Roles

[1846] The user mainly performs the following actions:

[1847] 1. Enter your information:

[1848] The user inputs a specific situation and the attributes of the other person through the terminal interface. For example, they input a job interview situation and the expected attributes of the interviewer (middle-aged man, strict personality).

[1849] 2. Execute the conversation:

[1850] The user interacts with the generated conversation simulator by speaking or texting, for example, answering hypothetical questions as practice for an interview.

[1851] 3. Review your feedback:

[1852] The user checks the feedback provided by the server as an analysis result and improves their own communication skills based on it.

[1853] Specific examples

[1854] Specific examples are shown below.

[1855] 1. User Input:

[1856] The user starts up the terminal, selects an interview situation, and inputs "middle-aged male, strict personality" as attributes.

[1857] 2. Server Processing:

[1858] The system receives the information sent from the device, generates a prompt in the generation AI with the content "Situation: Interview, Attributes: Middle-aged male, Strict personality", and sends it to the AI ​​model. The AI ​​model then returns the generated customized conversation simulator to the device.

[1859] 3. Conversation session:

[1860] The user begins a conversation with the conversation simulator displayed on the terminal. The simulator acts as an interviewer and asks questions, and the user answers.

[1861] 4. Data Collection and Analysis:

[1862] The device uses a camera and microphone to capture the user's facial expressions and tone of voice, and sends them along with the content of the conversation to the server, which then analyzes them and uses an emotion engine to recognize the user's emotions.

[1863] 5. Providing Feedback:

[1864] The server generates feedback and sends it to the device, which then displays the feedback received from the server to the user. For example, advice such as "Your answers are accurate, but your facial expression is stiff. You would make a better impression if you relaxed" is displayed.

[1865] Technology used and data processing

[1866] This system includes the following technologies and data processing:

[1867] Hardware: The device is equipped with a camera and microphone, which allows for real-time collection of the user's facial expressions and tone of voice.

[1868] Software: Generative AI models (e.g., GPT-3) are used to generate the conversation simulator, and an emotion engine is used to analyze nonverbal data and employ algorithms to recognize user emotions.

[1869] Data Processing: Techniques are used to collect and analyze speech and non-verbal data. The analyzed data provides important information for providing feedback to users.

[1870] Through this system, users can effectively improve their communication skills. By utilizing the emotion engine, users can receive more detailed and useful feedback.

[1871] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1872] Step 1:

[1873] The terminal provides an interface for users to access the system. Here, users select a situation (e.g., an interview) and enter the other party's attributes (e.g., middle-aged man, strict personality). The input data is saved in text format and sent from the terminal to the server in JSON format.

[1874] Input: Situations and attributes entered by the user

[1875] Output: JSON format data (situations and attributes)

[1876] Step 2:

[1877] The server receives and analyzes the JSON-formatted data sent from the device. As a result of the analysis, situation and attribute information is extracted. Based on this information, the server generates a prompt for the generative AI model.

[1878] Input: JSON format data (situations and attributes)

[1879] Output: Prompt sentence for the generative AI model (e.g., "Situation: Interview, Attributes: Middle-aged male, Strict personality")

[1880] Step 3:

[1881] The server sends the prompt sentence to a generative AI model (e.g., GPT-3) to generate a customized conversation simulator, which is returned to the server as text data.

[1882] Input: Prompt sentence for the generative AI model

[1883] Output: Text data of the generated conversation simulator

[1884] Step 4:

[1885] The server transfers the generated conversation simulator to the terminal, which displays the conversation simulator on its interface and starts a dialogue with the user.

[1886] Input: Text data of the generated conversation simulator

[1887] Output: Interface showing the conversation simulator

[1888] Step 5:

[1889] The user interacts with the conversation simulator and inputs responses. The device collects the conversation data in real time. It also uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice.

[1890] Input: User responses, facial expressions, and tone of voice

[1891] Output: Speech and non-verbal data

[1892] Step 6:

[1893] The device transmits the collected speech and non-verbal data in real time to a server, which receives the data and performs data analysis using NLP technology and an emotion engine.

[1894] Input: Speech and non-verbal data

[1895] Output: Analysis results and emotion recognition results

[1896] Step 7:

[1897] The server generates feedback to provide to the user based on the analysis and emotion recognition results, including specific advice based on the conversation content and nonverbal expressions.

[1898] Input: Analysis results and emotion recognition results

[1899] Output: Generated feedback

[1900] Step 8:

[1901] The server transmits the generated feedback to the terminal, which displays the feedback on a user interface for the user to confirm.

[1902] Input: Generated feedback

[1903] Output: Feedback displayed in the user interface

[1904] (Application example 2)

[1905] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1906] To improve their communication skills, users need to be able to practice effectively and practically based on specific situations and the attributes of the people they are interacting with. However, conventional methods have faced challenges in providing practical training for specific situations and in obtaining effective feedback. This problem is particularly evident in training salespeople in brick-and-mortar stores to improve their customer service skills.

[1907] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving information input by a user, means for generating a customized conversation simulator based on the information, means for collecting conversation data with the conversation simulator, means for analyzing the conversation data and nonverbal data and providing feedback, means for collecting the nonverbal data by face recognition means, and means for supporting the improvement of the user's customer interaction skills through the feedback. This enables the user to undergo practical training based on specific situations and the attributes of the other party and receive effective feedback.

[1908] A "user" is a person who utilizes the system to input information and receive feedback to improve their communication skills.

[1909] The "means for receiving information" is an interface or device for acquiring information input by the user about the situation and the attributes of the other party.

[1910] The "means for generating a conversation simulator" refers to a technique or device for creating a conversation simulator that corresponds to a specific situation or the attributes of the other person, based on information input by the user.

[1911] The "means for collecting conversation data" refers to a device or technology for recording the content of the conversation that takes place between the user and the generated conversation simulator.

[1912] "Nonverbal data" refers to data that includes information other than words, such as a user's facial expression or tone of voice.

[1913] A "means for providing feedback" is a device or technology that analyzes the collected speech and non-verbal data and indicates to the user areas for improvement in their communication skills.

[1914] "Facial recognition means" refers to technology or systems that use devices such as cameras to identify and analyze a user's facial expressions.

[1915] "Customer service skills" refers to the abilities and techniques that sales staff in physical stores and other locations have to respond appropriately to customers.

[1916] "Means for supporting improvement of users' customer service skills" are technologies and systems that include feedback and advice provided to enable users to provide better customer service.

[1917] This invention provides a training system for sales staff in brick-and-mortar stores to improve their customer service skills. The system operates using three entities: a user, a server, and a terminal. The roles and specific operations of each of these entities are described below.

[1918] Server Roles

[1919] The server performs the following functions:

[1920] 1. Receiving and analyzing information:

[1921] The server receives the situation and the other party's attributes sent from the terminal. For example, the server receives the situation "handling a complaint" and the attributes "elderly woman, angry."

[1922] 2. Creating a conversation simulator:

[1923] The server uses a generative AI model (e.g., OpenAI GPT-3.5) to generate a customized conversation simulator based on the information it receives, including prompts entered by the user.

[1924] Example prompt sentence:

[1925] Situation: Complaint handling, Attributes: Elderly woman, angry

[1926] 3. Model Transfer:

[1927] The server transfers the generated conversation simulator model to the terminal so that the user can use it.

[1928] 4. Data Analysis and Emotion Recognition:

[1929] The server analyzes the conversational data and non-verbal data sent from the device and recognizes the user's emotions using an emotion recognition engine, which includes facial expression analysis and tone of voice analysis.

[1930] 5. Feedback Generation:

[1931] The server generates feedback to provide to the user based on the analysis results and emotion recognition results, including advice based on the content of the conversation and nonverbal expressions.

[1932] Device Role

[1933] The terminal is responsible for the following functions:

[1934] 1. Information input interface:

[1935] The terminal provides an interface for the user to input information about the situation and the attributes of the other party, such as a text box or a multiple-choice input field.

[1936] 2. Submit your request:

[1937] The terminal starts generating a conversation simulator by transmitting the information input by the user to the server.

[1938] 3. Interfacing with the conversation simulator:

[1939] The terminal initiates an interactive session with the user using the generated conversation simulator received from the server, and provides an interface through which the user can conduct a conversation by text or voice.

[1940] 4. Nonverbal Data Collection:

[1941] The device uses a camera and microphone to collect the user's facial expressions and tone of voice in real time, and this data is later analyzed by a server.

[1942] 5. Viewing Feedback:

[1943] The terminal displays the feedback received from the server on a user interface, allowing the user to receive specific advice on how to improve their performance.

[1944] User Roles

[1945] The user performs the following actions:

[1946] 1. Enter your information:

[1947] Through the terminal interface, specific situations and the attributes of the other party are input. For example, the situation of handling a complaint and the expected attributes of the customer at that time (elderly woman, angry) can be input.

[1948] 2. Execute the conversation:

[1949] The user interacts with the generated conversation simulator by speaking or texting. For example, the user can practice how to respond to a complaint.

[1950] 3. Review your feedback:

[1951] Check the feedback provided by the server as a result of the analysis and improve your communication skills based on it.

[1952] Specific examples

[1953] As a concrete example, consider the following flow:

[1954] 1. The user starts up the terminal, selects the situation "Handling a complaint," and enters "elderly woman, angry" as the attribute.

[1955] 2. The server receives the information sent from the device and sends a prompt to the generative AI model with the following content: "Situation: Handling a complaint, Attributes: Elderly woman, Angry," and generates a customized conversation simulator.

[1956] 3. The user starts a conversation on the terminal using the conversation simulator returned from the server. The simulator plays the role of a customer and makes a complaint, and the user responds.

[1957] 4. The device uses a camera and microphone to collect the user's facial expressions and tone of voice, and sends them to the server along with the content of the conversation.

[1958] 5. The server analyzes the collected data and recognizes the user's emotions using an emotion recognition engine.

[1959] 6. The device receives the feedback generated by the server and displays specific advice to the user, such as, "The content of your complaint response is good, but it would be better if you spoke with a calmer expression."

[1960] In this way, the system allows users to efficiently improve their customer service skills in physical stores. By utilizing an emotion recognition engine, the system can provide more detailed feedback and suggest specific areas for improvement based on the user's emotions.

[1961] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1962] Step 1:

[1963] The user starts up the terminal and inputs information about the situation and the other party's attributes. For example, the situation "handling a complaint" and the attributes "elderly woman, angry" are entered into the text boxes. This starts the information input interface.

[1964] Input: situation (e.g., handling a complaint), other party's attributes (e.g., elderly woman, angry)

[1965] Output: The information entered

[1966] Step 2:

[1967] The terminal transmits the information input by the user to the server, where it arrives as a customized prompt.

[1968] Input: Information entered by the user

[1969] Output: Request to server

[1970] Step 3:

[1971] The server analyzes the received information and generates prompt sentences. The prompt sentences are used to generate a customized conversation simulator for a generative AI model (e.g., OpenAI GPT-3.5).

[1972] Input: Prompt sentence (e.g., Situation: Handling a complaint, Attributes: Elderly woman, Angry)

[1973] Data processing: Creating a conversation simulator using a generative AI model

[1974] Output: A customized conversation simulator

[1975] Step 4:

[1976] The server transfers the generated conversation simulator to the terminal, which then starts a dialogue session with the user based on the received simulator.

[1977] Input: A customized conversation simulator

[1978] Output: Transfer of conversation simulator to terminal

[1979] Step 5:

[1980] The user initiates a dialogue with a customized conversation simulator displayed on the terminal, and the user converses with the simulator using voice or text.

[1981] Input: A customized conversation simulator and the user's conversation content

[1982] Output: Conversation progress

[1983] Step 6:

[1984] The device uses a camera and microphone to collect non-verbal data such as the user's facial expressions and tone of voice, which is then sent to a server in real time along with the content of the conversation.

[1985] Input: User's facial expression data, voice data

[1986] Data Processing: Collection and Transmission of Nonverbal Data

[1987] Output: Nonverbal and conversational data

[1988] Step 7:

[1989] The server analyzes the received non-verbal data and conversation data, and recognizes the user's emotions using an emotion recognition engine. It also analyzes facial expressions and voice using a face recognition means.

[1990] Input: Nonverbal data, conversation data

[1991] Data Calculation: Analysis by Emotion Recognition Engine

[1992] Output: Analysis results (user emotion recognition results)

[1993] Step 8:

[1994] The server generates feedback based on the analysis results, including specific advice such as, "The content of your complaint handling is good, but it would be better if you spoke with a calmer expression."

[1995] Input: Analysis results (user emotion recognition results)

[1996] Data Calculation: Feedback Generation

[1997] Output: Feedback content

[1998] Step 9:

[1999] The terminal displays the generated feedback on a user interface, and the user can check the feedback and understand areas for improvement in their communication skills.

[2000] Input: Feedback

[2001] Output: Feedback display to the user

[2002] As a result, this system allows users to effectively practice conversation skills in specific situations and receive feedback, thereby improving their customer service skills.

[2003] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2004] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2005] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2006] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2007] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2008] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2009] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2010] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2011] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2012] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2013] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2014] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2015] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2016] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2017] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2018] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2019] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2020] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2021] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2022] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without sayi...

Claims

1. means for receiving user input information; means for generating a customized conversation simulator based on said information; means for collecting conversation data with the conversation simulator; means for analyzing the speech data and non-verbal data and providing feedback; A system including:

2. 2. The system according to claim 1, wherein said conversation simulator is generated based on a situation input by a user and attributes of the other party.

3. 2. The system of claim 1, wherein the nonverbal data includes the user's facial expression and tone of voice.

Images