System

Abstract

To provide a system for customer service training.SOLUTION: A virtual reality system for field service training includes means for providing an interface for a training subject to input persona information of a customer, means for using generative artificial intelligence to generate a customer avatar based on the persona information, means for using a virtual reality device for the training subject to interact with the customer avatar, means for using a server to generate and provide a response of the customer avatar to the virtual reality device in real-time, and means for logging the interaction and providing feedback to the training subject.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional customer service training required many resources, including physical locations, and could only be conducted at limited times and locations. Training content was also inconsistent, making it difficult to provide unique experiences for each participant, resulting in ineffective training. Furthermore, it was difficult to realistically simulate diverse situations and customer personas, resulting in a lack of realism in the training. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing the following means in a virtual reality system for customer service training.

[0006] The training includes a means for providing an interface for a trainee to input customer persona information, and a means for using generative artificial intelligence to generate a customer avatar based on the persona information.

[0007] The method further includes means for using a virtual reality device for the training subject to interact with the customer avatar, and means for using a server for generating responses of the customer avatar in real time and providing the responses to the virtual reality device.

[0008] The system also includes a means for logging the interactions and providing feedback to trainees, allowing for consistent training anytime, anywhere and for realistically simulating a variety of situations and customer personas.

[0009] "Customer service training" refers to an education and training program that companies and organizations offer to their employees to improve their customer service skills and customer service delivery skills.

[0010] A "virtual reality system" is a system that allows users to have various experiences in a virtual space, and is a technology that includes devices such as headsets and controllers.

[0011] "Persona information" is information used to create a virtual profile of a customer, and is data that includes attributes such as age, gender, hobbies, and characteristics.

[0012] "Interface" is a technology that refers to the screen and input means that users use to provide input information to a system and operate it.

[0013] "Generative artificial intelligence" is a general term for algorithms and technologies that automatically generate new data or responses based on input data.

[0014] A "customer avatar" is a virtual customer character generated based on persona information, and is a digital presence used to interact with users.

[0015] A "virtual reality device" is a physical piece of equipment, such as a headset or hand controllers, that a user uses to engage in a virtual reality experience.

[0016] A "server" is a computer system that stores, processes, and distributes data over a network, and in the present invention, it plays a role in managing various data of the virtual reality system.

[0017] A "log" is data that records the history of system operations and interactions, and is used later for analysis and to provide feedback.

[0018] "Feedback" is information that clarifies areas for improvement and strengths by providing evaluations and advice on a user's behavior and performance. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] MODE FOR CARRYING OUT THE INVENTION

[0041] The virtual reality system for customer service training of the present invention is implemented as follows.

[0042] Customer persona creation

[0043] 1. The user logs in to the system and accesses the persona setting screen.

[0044] 2. The user enters the following persona information into the form:

[0045] age

[0046] sex

[0047] hobby

[0048] Characteristics (e.g., many complaints, high purchasing intent, etc.)

[0049] 3. After entering the information, the user clicks the "Submit" button.

[0050] 4. The server receives the persona information and stores it in a database.

[0051] Customer avatar generation

[0052] 1. The server calls the generative AI model using the saved persona information.

[0053] 2. Generative AI generates the appearance, clothing, and behavior of a customer avatar based on the input persona information.

[0054] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[0055] Setting the situation

[0056] 1. The user accesses the situation setting screen.

[0057] 2. The user selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[0058] 3. The user presses the "Set Situation" button.

[0059] 4. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[0060] Simulation and Real-Time Response

[0061] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[0062] 2. When the user speaks to the customer avatar,

[0063] The device collects the audio and transmits it to a server in real time.

[0064] 3. The server receives the voice data and converts it into text using a speech analysis engine.

[0065] 4. The server uses generative AI to generate a response for the customer avatar based on the analyzed text.

[0066] 5. The server sends the generated response data to the terminal.

[0067] 6. The terminal displays the response as a voice and animation of the customer avatar.

[0068] Providing an interactive training experience

[0069] 1. Users continuously interact with the customer avatar and experience various situations.

[0070] 2. The server logs the user's interactions with the customer avatar for later analysis.

[0071] 3. After the training is completed, the server generates feedback from the analyzed logs and provides it to the user.

[0072] Specific examples

[0073] For example, let's take the situation of handling a complaint at a fashion retail store.

[0074] 1. The user persona is set as "a woman in her 30s who makes many complaints and is knowledgeable about brands."

[0075] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[0076] 3. The user starts the simulation in the situation of "handling a complaint."

[0077] 4. When the user says, "There is a problem with this product," the device sends the speech to the server.

[0078] 5. The server analyzes the voice and uses generative AI to generate a real-time response such as "Sorry, can you tell me more?", which is displayed on the device.

[0079] 6. Users can then ask further questions about the responses to practice realistic customer interactions.

[0080] In this way, users can receive training for a variety of situations through the virtual reality system, improving their actual customer service skills.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] A user logs in to the system and accesses the persona configuration screen.

[0084] Step 2:

[0085] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[0086] Step 3:

[0087] The server receives the persona information and stores it in a database.

[0088] Step 4:

[0089] The server calls the generative AI model based on the stored persona information.

[0090] Step 5:

[0091] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[0092] Step 6:

[0093] The server prepares the generated customer avatar data for the virtual reality environment.

[0094] Step 7:

[0095] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[0096] Step 8:

[0097] The user presses the "Set Situation" button.

[0098] Step 9:

[0099] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[0100] Step 10:

[0101] The user puts on the VR device (terminal) and enters the simulation environment.

[0102] Step 11:

[0103] The user speaks to the customer avatar.

[0104] Step 12:

[0105] The device collects the user's voice and transmits it to the server in real time.

[0106] Step 13:

[0107] The server analyzes the received voice data and converts it into text.

[0108] Step 14:

[0109] The server uses a generative AI to generate a response for the customer avatar based on the analyzed text.

[0110] Step 15:

[0111] The server transmits the generated response data to the terminal.

[0112] Step 16:

[0113] The terminal displays the response as audio and animation of a customer avatar.

[0114] Step 17:

[0115] The user and customer avatar continuously interact and experience various situations.

[0116] Step 18:

[0117] The server logs the interactions between the user and the customer avatar.

[0118] Step 19:

[0119] The server analyzes the log after the training is completed and provides feedback to the user.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Traditional customer service training has limited resources and flexibility to effectively improve real-world customer interaction skills. For example, it is difficult to perfectly recreate different customer personas and diverse situations in a real-world training environment, resulting in a lack of practical training. Additionally, feedback on customer interactions is generally not immediate. This creates the problem of taking a long time to acquire appropriate customer interaction skills.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes: means for providing an interface for a trainee to input customer persona information; means for using a generative AI model to generate a customer avatar based on the persona information; means for the trainee to use a virtual reality device to interact with the customer avatar; means for using a server to generate responses using a voice analysis engine and the generative AI model to generate responses for the customer avatar in real time and provide the responses to the virtual reality device; and means for recording a log of the interactions and providing feedback to the trainee. This effectively recreates different customer personas and a variety of situations, allowing the trainee to acquire practical customer service skills in real time and receive immediate feedback.

[0125] The "Virtual Reality System for Customer Service Training" is a system that allows trainees to train customer service skills using a virtual reality environment.

[0126] "Training Recipient" means an individual receiving training to improve their customer service skills.

[0127] "Persona information" refers to attribute information such as the age, gender, hobbies, and characteristics of a virtual customer.

[0128] "Interface" refers to a screen or input device that allows a user to input and operate persona information.

[0129] "Generative AI model" refers to a model that uses artificial intelligence technology to generate customer avatars based on input persona information.

[0130] "Virtual Reality Device" means a device (e.g., a VR headset) used by a trainee to interact with a customer avatar in a virtual reality environment.

[0131] "Server" refers to a computer system used to manage the entire system, process data, call generative AI models, record logs, and provide feedback.

[0132] "Speech analysis engine" refers to technology or software for converting voice data into text.

[0133] "Customer avatar" refers to a 3D model of a virtual customer generated based on persona information.

[0134] "Situation" refers to a specific scene or situation in customer service training.

[0135] "Response" refers to the reply or reaction that the customer avatar gives to the training recipient's input.

[0136] "Log" refers to a record of interactions between a user and a customer avatar during training.

[0137] "Feedback" refers to evaluation and advice regarding the user's performance provided after the training is completed.

[0138] The virtual reality system for customer service training of this invention is implemented as follows: A user first logs in to the system and accesses an interface for inputting persona information. Here, the user inputs and submits persona information such as age, gender, hobbies, and characteristics. The server receives this information and stores it in a database.

[0139] The server then invokes a generative AI model based on the stored persona information to generate a customer avatar. This generative AI model uses OpenAI's GPT-3, for example. The generated avatar's appearance, clothing, and behavior are customized based on the persona information. The server then prepares this avatar data for the virtual reality environment.

[0140] The user then accesses the situation setting screen and selects a specific situation (e.g., handling a complaint, explaining a new product, etc.). The server loads the script or scenario corresponding to the selected situation and completes preparations.

[0141] The user then puts on a VR device (e.g., Oculus Rift) and enters the simulation environment. When the user speaks to the customer avatar, the device (VR device) collects the speech and transmits it to the server in real time. The server receives the speech and converts it into text using a speech analysis engine (e.g., Google Speech-to-Text).

[0142] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The generated response is sent to the device in real time and displayed as the voice and animation of the customer avatar. The user can continue to interact with this response and experience various situations.

[0143] The server records the interactions between the user and the customer avatar as logs, and after the training is completed, the server generates and provides feedback to the user from the analyzed logs, which allows the user to effectively improve their customer service skills.

[0144] Specific examples

[0145] For example, consider a situation where a customer is handling a complaint at a fashion retailer. The user sets the persona as "a woman in her 30s who makes many complaints and is knowledgeable about brands." The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior. The user then starts the simulation in the "complaint handling" situation.

[0146] When a user says, "There's a problem with this product," the device sends the speech to a server. The server analyzes the speech and uses a generative AI model to generate a real-time response, such as "I'm sorry. Could you please tell me more?", which is displayed on the device. The user can then ask further questions about the response, practicing realistic customer interactions.

[0147] Prompt Sentence Examples

[0148] "Generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand."

[0149] "Please load a script for a complaint handling situation."

[0150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0151] Step 1:

[0152] A user logs into the system:

[0153] Input: Username and Password

[0154] What happens: A user enters their username and password on the login screen and clicks the "Login" button.

[0155] Data processing: The server receives the entered authentication information and checks it against a database.

[0156] Output: If authentication is successful, display the user's dashboard.

[0157] Step 2:

[0158] User accesses the Persona settings screen:

[0159] Input: Persona information (age, gender, hobbies, characteristics)

[0160] Action: The user selects the "Persona Settings" menu from the dashboard and proceeds to the settings screen.

[0161] Output: Displays an interface for entering persona information.

[0162] Step 3:

[0163] User enters persona information and submits:

[0164] Input: Values ​​corresponding to age, gender, hobbies, and characteristics

[0165] How it works: The user enters their persona information and clicks the "Submit" button.

[0166] Data processing: The server receives the entered data and performs validation (checking the accuracy and completeness of the data).

[0167] Output: Save the persona information that passed validation to the database.

[0168] Step 4:

[0169] The server invokes the generative AI model to generate the customer avatar:

[0170] Input: Saved persona information

[0171] How it works: The server calls the generative AI model based on the persona information, generates a prompt (e.g., "Please generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand"), and sends it.

[0172] Data Computation: A generative AI model generates the appearance, clothing, and behavior of a customer avatar based on the prompt text.

[0173] Output: The generated customer avatar data is sent back to the server.

[0174] Step 5:

[0175] The server prepares the customer avatar data for the VR environment:

[0176] Input: Generated avatar data

[0177] How it works: The server converts the received avatar data into a format suitable for the virtual reality environment.

[0178] Output: Avatar data that can be used in a virtual reality environment

[0179] Step 6:

[0180] User accesses the situation settings screen and selects a situation:

[0181] Input: Specific situation (e.g., handling a complaint, explaining a new product, etc.)

[0182] How it works: The user accesses the situation setup screen, selects a situation from the drop-down list, and presses the "Set Situation" button.

[0183] Output: Loads the scripts and scenarios corresponding to the selected situations into the server.

[0184] Step 7:

[0185] The user puts on the VR device and enters the simulated environment:

[0186] Input: Set situation

[0187] How it works: The user puts on a compatible VR device (e.g., Oculus Rift) and enters the simulated environment.

[0188] Output: The state in which the user begins interacting with the customer avatar in the simulated environment.

[0189] Step 8:

[0190] The user speaks to the customer avatar and audio is collected:

[0191] Input: User's voice

[0192] How it works: When a user speaks to a customer avatar, the terminal (VR device) collects the voice and sends it to the server in real time.

[0193] Output: The collected audio data is sent to the server.

[0194] Step 9:

[0195] The server analyzes the audio data and converts it to text:

[0196] Input: Collected audio data

[0197] How it works: The server receives the audio data and converts it to text using a speech analysis engine (e.g., Google Speech-to-Text).

[0198] Data calculation: Converting voice data into text

[0199] Output: Parsed text data

[0200] Step 10:

[0201] The server uses the generative AI model to generate a response for the customer avatar:

[0202] Input: Parsed text data

[0203] How it works: The server uses a generative AI model to generate a response for the customer avatar based on the parsed text.

[0204] Data Calculation: Generating Response Text

[0205] Output: Generated response data

[0206] Step 11:

[0207] The server sends the response data to the device:

[0208] Input: Generated response data

[0209] Operation: The server sends the generated response data to the terminal.

[0210] Output: Response data sent to the terminal

[0211] Step 12:

[0212] The device displays the response as audio and animation of a customer avatar:

[0213] Input: Response data sent

[0214] Action: The device displays the response as audio and animation of the customer avatar, providing feedback to the user.

[0215] Output: Customer avatar response displayed to the user

[0216] Step 13:

[0217] The server logs the interaction and generates feedback:

[0218] Input: User and customer avatar interaction data

[0219] How it works: The server logs interactions and generates feedback from the logs that are analyzed after the training is completed.

[0220] Output: Feedback provided to the user

[0221] (Application example 1)

[0222] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0223] Modern customer service training lacks practical training tools to improve customer service skills. It's also difficult to recreate realistic situations and allow trainees to gain diverse experience in a manner that closely resembles the real world. Traditional training methods also make it difficult to provide real-time feedback or use scripts tailored to specific situations. As a result, the results of training are often not fully utilized in the workplace.

[0224] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0225] In this invention, the server includes means for providing an interface for the trainee to input customer persona information, means for using a generative model to generate a customer avatar based on the persona information, means for using a virtual reality device for the trainee to interact with the customer avatar, means for using a data processing device to generate responses of the customer avatar in real time and provide them to the virtual reality device, means for recording a log of the interaction and providing feedback to the trainee, and means for loading scripts and events corresponding to specific situations such as product explanations and order taking in a physical store environment, thereby allowing the trainee to experience realistic customer interaction simulations and improve their response skills and ability to respond.

[0226] An "interface" is a point of contact between a device or software and a user for inputting or outputting information.

[0227] A "generative model" is an algorithm or program that automatically generates customer avatars based on persona information.

[0228] A "virtual reality device" is a device used to interact with a virtual reality environment, including head-mounted displays and VR goggles.

[0229] A "data processing device" is a hardware or software system that analyzes and processes input voice or text data and generates and transmits an appropriate response.

[0230] A "script" is a scenario or script for responding to a specific situation or event.

[0231] "Interaction" refers to the interaction between the trainee and the customer avatar, including conversation and behavioral exchanges.

[0232] "Persona information" refers to information such as age, gender, hobbies, and characteristics that the trainee enters to define the characteristics of a virtual customer.

[0233] A "customer avatar" is a virtual customer character generated based on persona information.

[0234] A "log" is a record of the interaction between the trainee and the customer avatar, and is data that is later used for analysis and feedback.

[0235] "Feedback" refers to the results, evaluations, and advice analyzed based on interaction logs, and is provided to trainees.

[0236] The present invention relates to a customer service training system that utilizes virtual reality. Hereinafter, an embodiment of this system will be described in detail.

[0237] System Configuration

[0238] User Login and Persona Configuration

[0239] The server provides an interface for users to log in and enter persona information (age, gender, hobbies, characteristics, etc.). This interface can be implemented as a web application or a mobile application, for example. The persona information is sent to the server and stored in a database (for example, MySQL or PostgreSQL).

[0240] Customer avatar generation

[0241] The server uses the stored persona information to invoke a generation algorithm that generates a customer avatar based on this information, using a generative AI model such as OpenAI's GPT-4, and then converts the generated customer avatar into a format that can be used in a virtual reality environment.

[0242] Using a Virtual Reality Device

[0243] The user enters the simulated environment wearing a virtual reality device (e.g., a head-mounted display), such as the Oculus Rift or HTC Vive.

[0244] Setting up situations and generating responses

[0245] The user selects a specific situation (e.g., handling a complaint, explaining a new product, accepting an order, etc.) through the situation setting interface, and the server loads the scripts and events corresponding to this situation.

[0246] When a user speaks to the customer avatar, the virtual reality device collects the speech data and sends it to a server. The server converts the speech into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The converted text is analyzed by a generative AI model (e.g., GPT-4) to generate an appropriate response. This response is sent to the user's virtual reality device in real time.

[0247] Interaction recording and feedback

[0248] The server records the user's interactions with the customer avatar as a log, which allows for later analysis and feedback. After the training is completed, the server analyzes the log and provides feedback to the user, including specific advice and evaluations to improve the user's response skills.

[0249] Specific examples

[0250] For example, consider a situation where a new product is being introduced in a physical store environment.

[0251] 1. The user persona is set as "a woman in her 20s, with a high desire to purchase and a keen sense of fashion."

[0252] 2. The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior.

[0253] 3. The user starts the simulation in the situation of "explaining a new product."

[0254] 4. When the user says, "Tell me about this new dress," the server analyzes the speech and generates a real-time response, such as, "This dress has a design that incorporates the latest trends..."

[0255] Prompt Sentence Examples

[0256] User: "Tell me about this new dress."

[0257] Example prompt for a generative AI model (GPT-4): "Describe a new dress for a fashion-conscious female customer in her 20s with a high purchasing intent."

[0258] Generated AI response: "This dress is designed with the latest trends in mind, making it perfect for spring and summer. It's made from premium materials and is incredibly comfortable. Plus, this dress..."

[0259] In this way, users can receive training in a variety of situations through the virtual reality system, improving their actual customer service skills.

[0260] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0261] Step 1:

[0262] A user logs into a customer service training application. The input is the user's login information (e.g., username and password), which the server verifies and authenticates the user. The output is the main interface screen after the user has logged in.

[0263] Step 2:

[0264] The user accesses an interface to input persona information. The input is persona information such as age, gender, hobbies, and characteristics, which the server receives and stores in a database. The output is a confirmation message that the persona information has been saved.

[0265] Step 3:

[0266] The server calls the generative model based on the stored persona information. The input is the persona information (e.g., "female in her 30s, frequent complainer, knowledgeable about brands"), and generates a customer avatar using a generative AI model (e.g., OpenAI GPT-4). The output is the data of the generated customer avatar.

[0267] Step 4:

[0268] The user puts on the virtual reality device and enters the training simulation environment. The input is a start-up signal for the virtual reality device, and the terminal loads the simulation environment. The output is a confirmation screen that the user has entered the virtual reality environment.

[0269] Step 5:

[0270] The user selects a specific situation through the situation setting interface. The input is the situation information (e.g., "Handling a complaint," "Explaining a new product," "Taking an order"), and the server loads the associated scripts and events. The output is a message indicating that the selected situation is ready.

[0271] Step 6:

[0272] When a user speaks to the customer avatar, the virtual reality device collects the voice data and sends it to the server in real time. The input is voice data, and the server converts it into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The output is the voice-to-text data.

[0273] Step 7:

[0274] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The input is the user's textual utterance, and a natural-sounding response is generated through the generative AI model (e.g., GPT-4). The output is the generated response text.

[0275] Step 8:

[0276] The server sends the generated response data to the virtual reality device, where it is displayed as voice and animation of the customer avatar. The input is the response text, which the device converts into voice data using a speech synthesis engine and synchronizes with the avatar's movements. The output is the avatar's response animation.

[0277] Step 9:

[0278] The server logs the interactions between the user and the customer avatar and stores the data for later analysis. The input is the conversation log between the user and the avatar, and the output is the log data stored in the database.

[0279] Step 10:

[0280] After the training is completed, the server analyzes the logs and generates feedback. The input is the saved log data, and the data analysis algorithm evaluates the user's response skills. The output is a feedback report provided to the user.

[0281] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0282] MODE FOR CARRYING OUT THE INVENTION

[0283] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The system is implemented as follows.

[0284] Persona Settings

[0285] 1. A user logs in to the system and accesses the persona configuration screen.

[0286] 2. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[0287] 3. The server receives the persona information and stores it in a database.

[0288] Customer avatar generation

[0289] 1. The server calls the generative AI model using the saved persona information.

[0290] 2. The generation AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[0291] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[0292] Setting the situation

[0293] 1. The user accesses the situation setting screen and selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[0294] 2. The user presses the "Set Situation" button.

[0295] 3. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[0296] Simulation and Real-Time Response

[0297] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[0298] 2. The user speaks to the customer avatar.

[0299] 3. The device collects the user's voice, facial expressions, and gestures using the camera and microphone, and transmits them to the server in real time.

[0300] Emotion Recognition and Response Generation

[0301] 1. The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine.

[0302] 2. Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[0303] 3. The server sends the generated response data to the terminal.

[0304] 4. The device displays the response as audio and animation of the customer avatar.

[0305] Interactive training experience and feedback

[0306] 1. Users continuously interact with the customer avatar and experience various situations.

[0307] 2. The server records the user's interactions with the customer avatar and the emotion recognition logs.

[0308] 3. After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data.

[0309] Specific examples

[0310] For example, let's take the situation of handling a complaint at a restaurant.

[0311] 1. The user persona is set as "a man in his 40s with extensive experience in the restaurant industry who is very prone to dissatisfaction."

[0312] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[0313] 3. The user starts the simulation in the situation of "handling a complaint."

[0314] 4. When the user says, "This dish is cold," and their face turns grim, the device sends the voice and facial expression to the server.

[0315] 5. The server analyzes the voice and facial expression data and recognizes "dissatisfaction" using an emotion engine.

[0316] 6. The server uses a generation AI to generate a response such as, "Sorry. We will bring you a new dish right away," and sends it to the device.

[0317] 7. The terminal displays a customer avatar responding with voice and animation.

[0318] 8. The user continues to ask questions about the response, practicing realistic customer interactions using emotion recognition.

[0319] In this way, this system, which combines an emotion engine, can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[0320] The processing flow will be explained below.

[0321] MODE FOR CARRYING OUT THE INVENTION

[0322] This invention is a virtual reality system for customer service training that combines an emotion engine that recognizes the user's emotions. The specific processing flow is shown below.

[0323] Processing Steps

[0324] Step 1:

[0325] A user logs in to the system and accesses the persona configuration screen.

[0326] Step 2:

[0327] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[0328] Step 3:

[0329] The server receives the persona information and stores it in a database.

[0330] Step 4:

[0331] The server calls the generative AI model based on the stored persona information.

[0332] Step 5:

[0333] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[0334] Step 6:

[0335] The server prepares the generated customer avatar data for the virtual reality environment.

[0336] Step 7:

[0337] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[0338] Step 8:

[0339] The user presses the "Set Situation" button.

[0340] Step 9:

[0341] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[0342] Step 10:

[0343] The user puts on the VR device (terminal) and enters the simulation environment.

[0344] Step 11:

[0345] The user speaks to the customer avatar.

[0346] Step 12:

[0347] The device collects the user's voice, facial expressions, and gestures using a camera and microphone, and transmits the data to a server in real time.

[0348] Step 13:

[0349] The server analyzes the received voice, facial expressions, and gesture data, and recognizes the user's emotions using an emotion engine.

[0350] Step 14:

[0351] Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[0352] Step 15:

[0353] The server transmits the generated response data to the terminal.

[0354] Step 16:

[0355] The terminal displays the response as audio and animation of a customer avatar.

[0356] Step 17:

[0357] Users continuously interact with the customer avatar and experience various situations.

[0358] Step 18:

[0359] The server logs the user's interactions with the customer avatar and the emotion recognition.

[0360] Step 19:

[0361] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[0362] Specific examples

[0363] For example, let's take the situation of handling a complaint at a restaurant.

[0364] Step 1:

[0365] A user logs in to the system and accesses the persona configuration screen.

[0366] Step 2:

[0367] The user enters the persona information as "male in his 40s, has extensive experience in the restaurant industry, and is very prone to dissatisfaction" and clicks the "Submit" button.

[0368] Step 3:

[0369] The server receives the persona information and stores it in a database.

[0370] Step 4:

[0371] The server calls the generative AI model based on the stored persona information.

[0372] Step 5:

[0373] The generation AI generates a customer avatar with specific clothing and behavior based on the persona information entered.

[0374] Step 6:

[0375] The server prepares the generated customer avatar data for the virtual reality environment.

[0376] Step 7:

[0377] The user accesses the situation setting screen and selects the situation "Handling a complaint."

[0378] Step 8:

[0379] The user presses the "Set Situation" button.

[0380] Step 9:

[0381] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[0382] Step 10:

[0383] The user puts on the VR device (terminal) and enters the simulation environment.

[0384] Step 11:

[0385] The user complains to the customer avatar, saying, "This food is cold."

[0386] Step 12:

[0387] The device collects the user's voice and facial expressions and transmits them to a server in real time.

[0388] Step 13:

[0389] The server analyzes the voice, facial expressions, and gesture data it receives, and uses an emotion engine to recognize "dissatisfaction."

[0390] Step 14:

[0391] Based on the results of emotion recognition, the server uses generative AI to generate a response such as, "We're sorry. We'll bring you a new dish right away."

[0392] Step 15:

[0393] The server transmits the generated response data to the terminal.

[0394] Step 16:

[0395] The device displays a customer avatar responding with voice and animation.

[0396] Step 17:

[0397] The user continues to ask further questions based on the responses, practicing realistic customer interactions using emotion recognition.

[0398] Step 18:

[0399] The server logs the user's interactions with the customer avatar and the emotion recognition.

[0400] Step 19:

[0401] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[0402] Example 2

[0403] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] Modern customer service training requires practical and interactive training methods. However, traditional training methods have difficulty recreating real-life customer interactions, limiting the opportunities for trainees to demonstrate their actual skills. Furthermore, the lack of a system that can recognize trainees' emotions and reactions in real time and provide adaptive feedback based on that information makes it difficult to achieve deeper learning.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0406] In this invention, the server includes means for collecting and transmitting the voice, facial expressions, and gestures of the trainee to the server, means for analyzing the received voice, facial expressions, and gesture data to recognize emotions, and means for using a generative artificial intelligence to adjust and generate responses of a customer avatar based on the results of the emotion recognition. This makes it possible to recognize the emotions and reactions of the trainee in real time and provide adaptive feedback based on them.

[0407] "Training Recipient" means a person receiving customer service training.

[0408] "Persona information" refers to data such as age, gender, hobbies, and characteristics that are used as basic information when generating a customer avatar.

[0409] "Interface" refers to the screen or operating means used by trainees to enter persona information.

[0410] "Generative AI" is an AI technology for generating the appearance, clothing, and behavior of a customer avatar based on persona information.

[0411] "Virtual reality device" means equipment used by a trainee to interact with a customer avatar in a virtual environment.

[0412] "Server" means a computer system for storing persona information and providing responses of generated customer avatars.

[0413] "Voice, facial expression, and gesture data" refers to information about the voice, facial expression, and body movements of the training subject, and is data used for emotion recognition.

[0414] "Emotion recognition" is a technology that grasps the emotions of trainees from their voices, facial expressions, and gestures.

[0415] A "customer avatar" is a virtual character that is generated based on persona information and interacts with the training subject in a virtual environment.

[0416] "Response" refers to the reply or action that the customer avatar gives in response to an interaction from the training subject.

[0417] "Log" refers to data that records the interactions between the training subject and the customer avatar and the results of emotion recognition.

[0418] "Feedback" refers to information, including evaluations and areas for improvement, that is provided to trainees after the training has been completed.

[0419] "Specific situations" are specific situations or scenes that trainees experience in a virtual environment, such as handling complaints, explaining new products, or accepting orders.

[0420] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The main components of this system include a user terminal, a server, a generative AI model, and a virtual reality device. Specific implementation methods are described below.

[0421] First, the user logs in to the system and accesses the persona setting screen. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. The server receives the submitted persona information and stores it in a database. The server then invokes a generative AI model based on the stored persona information to generate the appearance, clothing, and behavior of the customer avatar. An example of a prompt message for this is as follows:

[0422] "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[0423] The generated customer avatar data is prepared for the virtual reality environment by the server. The user accesses the situation setting screen and selects a specific situation. Examples include "handling a complaint," "explaining a new product," and "accepting an order." When the user presses the "Set Situation" button, the server loads the script or scenario corresponding to the selected situation and completes the preparation.

[0424] Next, the user puts on the VR device (terminal) and enters the simulation environment. When the user speaks to the customer avatar, the terminal collects the user's voice, as well as facial expressions and gestures using a camera and microphone, and sends the collected data to the server in real time. The server analyzes the received voice, facial expressions, and gesture data and recognizes the user's emotions using an emotion engine. For example, if a user says, "This dish is cold," and their face turns grim, the server analyzes the voice and facial expression data and recognizes the emotion as "dissatisfied."

[0425] Based on the emotion recognition results, the server uses a generative AI model to adjust and generate a response for the customer avatar. For example, a response such as "Sorry, we'll bring you a new dish right away" may be generated. The server then sends the generated response data to the device, which then displays the customer avatar's response with voice and animation.

[0426] In this way, the user continuously interacts with the customer avatar and experiences various situations. The server records the interaction between the user and the customer avatar and the emotion recognition log, analyzes the log after the training is completed, and provides the user with feedback including the emotion data.

[0427] For example, in a situation where a customer is handling a complaint at a restaurant, the user sets their persona as "a man in his 40s with extensive restaurant experience who is very prone to becoming dissatisfied." The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior, and the user begins the simulation in the "handling a complaint" situation. When the user says, "This dish is cold," the device sends their voice and facial expression to the server, which analyzes it and generates an appropriate response, which is sent back to the device.

[0428] In this way, the system can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[0429] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0430] Step 1:

[0431] A user logs in to the system and accesses the persona configuration screen. They provide their username and password as input, and the server performs the authentication process. After successful authentication, they are redirected to the persona configuration screen.

[0432] Step 2:

[0433] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. Input data includes age "40", gender "male", hobbies "reading", and characteristics "kind". The server receives the persona information and saves it in the database. Specifically, the server analyzes the persona information sent, generates an SQL statement to insert into the database, connects to the database, and executes the SQL statement.

[0434] Step 3:

[0435] The server calls the generative AI model using the saved persona information. It uses the persona information as input to generate a prompt. This prompt is passed to the generative AI model to generate a customer avatar. As output, appearance information (hair color, clothing style, etc.) and behavior information are obtained. An example prompt is "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[0436] Step 4:

[0437] The server prepares the generated customer avatar data for the virtual reality environment. It uses the generated appearance and behavior information as input data, converts it into a format suitable for the VR environment, and organizes the necessary files. As an output, it obtains customer avatar data for the VR environment.

[0438] Step 5:

[0439] The user accesses the situation setting screen and selects a specific situation from a drop-down list. The user selects a situation (e.g., "Complaint handling") as input data and presses the "Set Situation" button. The server retrieves the script corresponding to the selected scenario from the database and loads it into memory. The scenario data is prepared as output.

[0440] Step 6:

[0441] The user puts on the VR device (terminal) and enters the simulation environment. The input is to start up the VR device and connect it to the system. The terminal starts up the VR environment and places the user in the simulation. This allows the user to start training in the virtual environment.

[0442] Step 7:

[0443] The user speaks to the customer avatar. For example, they may say, "This dish is cold." The device collects the user's voice and simultaneously records their facial expressions and gestures using a camera and microphone. This data is sent to the server in real time. The collected voice, facial expressions, and gesture data is sent to the server as output.

[0444] Step 8:

[0445] The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine. Voice data, facial expression data, and gesture data are used as input, and an emotion recognition algorithm is applied. The output is an emotion result such as "dissatisfied."

[0446] Step 9:

[0447] The server uses generative AI to adjust and generate the customer avatar's response based on the emotion recognition results. The emotion recognition results and generative AI model are used as input. When the generative AI model is sent a prompt such as "How should you respond to an angry customer?", the output generated is a response such as "We're sorry. We'll bring you a new dish right away."

[0448] Step 10:

[0449] The server sends the generated response data to the terminal, uses the generated response data as input and sends it to the terminal, and when the terminal receives the response data, displays the response as voice and animation of the customer avatar.

[0450] Step 11:

[0451] Users continually interact with customer avatars and experience various situations, allowing them to simulate real-life customer service scenarios and improve their response skills.

[0452] Step 12:

[0453] The server records the interaction between the user and the customer avatar and the emotion recognition log. Each interaction and the emotion recognition results are used as input and stored in a database. After the training is completed, the log is analyzed and feedback, including the emotion data, is provided to the user.

[0454] Step 13:

[0455] After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data. Using the recorded log data as input, the server generates a report that includes the user's behavioral trends and areas for improvement. Detailed feedback is provided to the user as output.

[0456] (Application example 2)

[0457] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0458] Traditional customer service training systems are limited to standard scenarios and situations, and are unable to respond to trainees' emotions and reactions in real time. This limits the effectiveness of training and causes problems such as a lack of flexibility and responsiveness required for actual customer interactions. It is also difficult to track trainees' emotions and provide appropriate feedback based on them.

[0459] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0460] In this invention, the server includes means for collecting voice data and facial expression and gesture data captured by a camera of the trainee and transmitting them to the server in real time, means for analyzing the collected data and recognizing the trainee's emotions with an emotion engine, and means for adjusting and generating responses of a customer avatar using generative artificial intelligence based on the results of emotion recognition. This makes it possible to generate responses in real time based on the trainee's emotions and improve customer service skills in brick-and-mortar stores in a manner closer to reality.

[0461] "Training Participant" means a person who participates in a customer service training program and seeks to improve their skills using a virtual reality system.

[0462] "Persona information" refers to personal information such as age, gender, hobbies, and characteristics entered by the training participant, and a customer avatar is generated based on this information.

[0463] "Generative AI" refers to AI that has the ability to generate new data based on input data, and is used to generate the appearance and behavior of customer avatars.

[0464] A "virtual reality device" is a device that displays a virtual reality space and allows users to interact with that space. Examples include head-mounted displays and VR goggles.

[0465] A "server" is a computer that processes and stores data over a network, and is used here for generating responses for customer avatars, recording logs, and analyzing emotions.

[0466] A "customer avatar" is a virtual person generated within a virtual reality system based on persona information, and is responsible for interacting with the training recipient.

[0467] An "emotion engine" is software that analyzes voice data, facial expression data, and gesture data to recognize the emotional state of the subject.

[0468] An "interface" refers to the operation screen and input device that trainees use to interact with the system.

[0469] A "log" is a record of the system's operation history and data, and in this case, it records the interaction between the training participant and the customer avatar.

[0470] "Feedback" refers to the evaluation and areas for improvement provided to the training participants, and is provided based on information analyzed from the logs.

[0471] A "scenario" is a sequence of events or dialogue used in a specific situation or setting, and is used to set the scene for training.

[0472] This invention is a system that combines a virtual reality system for customer service training with an emotion engine that recognizes the emotions of users. Specific embodiments will be described below.

[0473] Structure and program overview

[0474] This system allows trainees to input customer persona information and interact with a customer avatar generated based on that information. The system uses a smartphone or a head-mounted display (VR device) as hardware, and includes a server for generating real-time responses, an emotion engine for recognizing emotions, and generative artificial intelligence.

[0475] Hardware and Software Used

[0476] Smartphone or head-mounted display (VR device): A device that allows users to interact in a virtual space.

[0477] Server: A computer that processes data, analyzes it, and generates responses.

[0478] Emotion engine: Software that analyzes voice data, facial expression data, and gesture data to recognize the user's emotions.

[0479] Generative artificial intelligence: Software that generates customer avatars based on persona information and generates responses in real time based on user sentiment.

[0480] Data processing and calculation

[0481] 1. Login and Persona Settings

[0482] The user logs in to the smartphone app and enters their age, gender, hobbies, and characteristics on the persona setting screen. This persona information is sent to the server and stored in the database.

[0483] 2. Customer avatar generation

[0484] The server passes the persona information to a generation AI to generate the appearance, clothing, and behavior of the customer avatar, and the generated data is then appropriately prepared for the VR environment.

[0485] 3. Setting the situation

[0486] The user selects a training situation on the app screen, and the server loads the corresponding script and scenario.

[0487] 4. Simulation and real-time response

[0488] The user puts on the VR device and begins interacting with a customer avatar in a virtual store.

[0489] The smartphone or VR device collects the user's voice data, facial expression data, and gesture data and transmits them to a server in real time.

[0490] The server's emotion engine analyzes the data and recognizes the user's emotions.

[0491] Based on the results of emotion recognition, the generative AI model generates a response for the customer avatar, which the server then sends to the device, allowing the avatar to respond in real time.

[0492] Specific examples

[0493] Customer Service Training Scenarios

[0494] 1. The user enters "Female in her 30s, gentle personality, retail store worker, positive" on the "Persona Settings" screen and submits.

[0495] 2. The server generates a "gentle customer avatar, a woman in her 30s" based on the persona information.

[0496] 3. The user selects and sets the situation "Explanation of a new product."

[0497] 4. The user wearing the VR device speaks to the avatar, asking, "What are the features of this new product?" The camera captures the user's facial expressions and sends them to the server along with the audio data.

[0498] 5. The server analyzes the user's facial expressions and uses an emotion engine to recognize "interest." The generative AI model generates a response, such as "This new product is compact, lightweight, and made from eco-friendly materials." The server then sends the response to the device, where the customer avatar responds with voice and animation.

[0499] Prompt Sentence Examples

[0500] Persona information: Female in her 30s, gentle personality, retail store worker, positive

[0501] Situation: New product description

[0502] User Sentiment: Interest

[0503] What it produces: This new product is compact, lightweight and made from eco-friendly materials.

[0504] This allows users to improve their skills in dealing with real customers.

[0505] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0506] Step 1:

[0507] A user logs into the system and enters persona information

[0508] Input: The user enters persona information such as age, gender, hobbies, and characteristics.

[0509] How it works: The user enters persona information (e.g., female in her 30s, gentle personality, retail worker, positive) through the interface and clicks the "Submit" button.

[0510] Output: Persona information is sent to the system and stored on the server.

[0511] Step 2:

[0512] The server generates a customer avatar based on the persona information

[0513] Input: Saved persona information.

[0514] How it works: The server retrieves persona information and uses generative artificial intelligence (generative AI model) to generate the appearance, clothing, and behavior of a customer avatar.

[0515] Output: The generated customer avatar is prepared for the virtual reality environment.

[0516] Step 3:

[0517] Users set training situations

[0518] Input: Select a training situation (e.g., explaining a new product).

[0519] How it works: The user selects a specific situation on the situation setting screen and presses the "Set Situation" button.

[0520] Output: The scripts and scenarios corresponding to the selected situations are loaded by the server.

[0521] Step 4:

[0522] The user puts on the VR device and starts the simulation.

[0523] Input: Customer avatar and situation script generated based on persona information.

[0524] How it works: A user puts on a VR device and begins interacting with a customer avatar in a virtual store.

[0525] Output: The interaction is initiated by the user speaking to the customer avatar.

[0526] Step 5:

[0527] The device collects the user's voice, facial expressions, and gestures and sends them to the server.

[0528] Input: User's voice data, facial expression data captured by the camera, and gesture data.

[0529] Actions: The device's sensors capture the user's voice, facial expressions, and gestures and transmit them to the server in real time.

[0530] Output: The user's voice, facial expression, and gesture data are sent to the server.

[0531] Step 6:

[0532] The server recognizes the user's emotions using an emotion engine

[0533] Input: Collected voice data, facial expression data, and gesture data.

[0534] How it works: The server's emotion engine analyzes the collected data and recognizes the user's emotions (e.g., interest, frustration).

[0535] Output: The analyzed emotion information is passed to the generative AI model.

[0536] Step 7:

[0537] The server generates a response for the customer avatar based on the emotion recognition results.

[0538] Input: Emotion recognition results, generative artificial intelligence model.

[0539] How it works: The server uses the generative AI model to generate an appropriate customer avatar response based on the recognized emotion (e.g., "This new product is compact, lightweight, and made from eco-friendly materials.").

[0540] Output: The generated response data is sent to the terminal.

[0541] Step 8:

[0542] The device displays the response to the customer avatar

[0543] Input: The response data of the generated customer avatar.

[0544] How it works: The device displays and plays the generated response data with animation and audio, and the customer avatar responds in real time.

[0545] Output: The user confirms the generated response and continues the interaction.

[0546] Step 9:

[0547] The server logs the interaction and provides feedback to the trainee.

[0548] Input: All interaction data for the user and customer avatar.

[0549] How it works: The server logs interactions and later analyzes them to provide feedback to the user.

[0550] Output: The user receives feedback including the results of the training and areas for improvement.

[0551] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0552] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0553] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0554] [Second embodiment]

[0555] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0556] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0557] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0558] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0559] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0560] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0561] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0562] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0563] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0564] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0565] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0566] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0567] MODE FOR CARRYING OUT THE INVENTION

[0568] The virtual reality system for customer service training of the present invention is implemented as follows.

[0569] Customer persona creation

[0570] 1. The user logs in to the system and accesses the persona setting screen.

[0571] 2. The user enters the following persona information into the form:

[0572] age

[0573] sex

[0574] hobby

[0575] Characteristics (e.g., many complaints, high purchasing intent, etc.)

[0576] 3. After entering the information, the user clicks the "Submit" button.

[0577] 4. The server receives the persona information and stores it in a database.

[0578] Customer avatar generation

[0579] 1. The server calls the generative AI model using the saved persona information.

[0580] 2. Generative AI generates the appearance, clothing, and behavior of a customer avatar based on the input persona information.

[0581] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[0582] Setting the situation

[0583] 1. The user accesses the situation setting screen.

[0584] 2. The user selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[0585] 3. The user presses the "Set Situation" button.

[0586] 4. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[0587] Simulation and Real-Time Response

[0588] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[0589] 2. When the user speaks to the customer avatar,

[0590] The device collects the audio and transmits it to a server in real time.

[0591] 3. The server receives the voice data and converts it into text using a speech analysis engine.

[0592] 4. The server uses generative AI to generate a response for the customer avatar based on the analyzed text.

[0593] 5. The server sends the generated response data to the terminal.

[0594] 6. The terminal displays the response as a voice and animation of the customer avatar.

[0595] Providing an interactive training experience

[0596] 1. Users continuously interact with the customer avatar and experience various situations.

[0597] 2. The server logs the user's interactions with the customer avatar for later analysis.

[0598] 3. After the training is completed, the server generates feedback from the analyzed logs and provides it to the user.

[0599] Specific examples

[0600] For example, let's take the situation of handling a complaint at a fashion retail store.

[0601] 1. The user persona is set as "a woman in her 30s who makes many complaints and is knowledgeable about brands."

[0602] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[0603] 3. The user starts the simulation in the situation of "handling a complaint."

[0604] 4. When the user says, "There is a problem with this product," the device sends the speech to the server.

[0605] 5. The server analyzes the voice and uses generative AI to generate a real-time response such as "Sorry, can you tell me more?", which is displayed on the device.

[0606] 6. Users can then ask further questions about the responses to practice realistic customer interactions.

[0607] In this way, users can receive training for a variety of situations through the virtual reality system, improving their actual customer service skills.

[0608] The processing flow will be explained below.

[0609] Step 1:

[0610] A user logs in to the system and accesses the persona configuration screen.

[0611] Step 2:

[0612] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[0613] Step 3:

[0614] The server receives the persona information and stores it in a database.

[0615] Step 4:

[0616] The server calls the generative AI model based on the stored persona information.

[0617] Step 5:

[0618] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[0619] Step 6:

[0620] The server prepares the generated customer avatar data for the virtual reality environment.

[0621] Step 7:

[0622] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[0623] Step 8:

[0624] The user presses the "Set Situation" button.

[0625] Step 9:

[0626] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[0627] Step 10:

[0628] The user puts on the VR device (terminal) and enters the simulation environment.

[0629] Step 11:

[0630] The user speaks to the customer avatar.

[0631] Step 12:

[0632] The device collects the user's voice and transmits it to the server in real time.

[0633] Step 13:

[0634] The server analyzes the received voice data and converts it into text.

[0635] Step 14:

[0636] The server uses a generative AI to generate a response for the customer avatar based on the analyzed text.

[0637] Step 15:

[0638] The server transmits the generated response data to the terminal.

[0639] Step 16:

[0640] The terminal displays the response as audio and animation of a customer avatar.

[0641] Step 17:

[0642] The user and customer avatar continuously interact and experience various situations.

[0643] Step 18:

[0644] The server logs the interactions between the user and the customer avatar.

[0645] Step 19:

[0646] The server analyzes the log after the training is completed and provides feedback to the user.

[0647] Example 1

[0648] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0649] Traditional customer service training has limited resources and flexibility to effectively improve real-world customer interaction skills. For example, it is difficult to perfectly recreate different customer personas and diverse situations in a real-world training environment, resulting in a lack of practical training. Additionally, feedback on customer interactions is generally not immediate. This creates the problem of taking a long time to acquire appropriate customer interaction skills.

[0650] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0651] In this invention, the server includes: means for providing an interface for a trainee to input customer persona information; means for using a generative AI model to generate a customer avatar based on the persona information; means for the trainee to use a virtual reality device to interact with the customer avatar; means for using a server to generate responses using a voice analysis engine and the generative AI model to generate responses for the customer avatar in real time and provide the responses to the virtual reality device; and means for recording a log of the interactions and providing feedback to the trainee. This effectively recreates different customer personas and a variety of situations, allowing the trainee to acquire practical customer service skills in real time and receive immediate feedback.

[0652] The "Virtual Reality System for Customer Service Training" is a system that allows trainees to train customer service skills using a virtual reality environment.

[0653] "Training Recipient" means an individual receiving training to improve their customer service skills.

[0654] "Persona information" refers to attribute information such as the age, gender, hobbies, and characteristics of a virtual customer.

[0655] "Interface" refers to a screen or input device that allows a user to input and operate persona information.

[0656] "Generative AI model" refers to a model that uses artificial intelligence technology to generate customer avatars based on input persona information.

[0657] "Virtual Reality Device" means a device (e.g., a VR headset) used by a trainee to interact with a customer avatar in a virtual reality environment.

[0658] "Server" refers to a computer system used to manage the entire system, process data, call generative AI models, record logs, and provide feedback.

[0659] "Speech analysis engine" refers to technology or software for converting voice data into text.

[0660] "Customer avatar" refers to a 3D model of a virtual customer generated based on persona information.

[0661] "Situation" refers to a specific scene or situation in customer service training.

[0662] "Response" refers to the reply or reaction that the customer avatar gives to the training recipient's input.

[0663] "Log" refers to a record of interactions between a user and a customer avatar during training.

[0664] "Feedback" refers to evaluation and advice regarding the user's performance provided after the training is completed.

[0665] The virtual reality system for customer service training of this invention is implemented as follows: A user first logs in to the system and accesses an interface for inputting persona information. Here, the user inputs and submits persona information such as age, gender, hobbies, and characteristics. The server receives this information and stores it in a database.

[0666] The server then invokes a generative AI model based on the stored persona information to generate a customer avatar. This generative AI model uses OpenAI's GPT-3, for example. The generated avatar's appearance, clothing, and behavior are customized based on the persona information. The server then prepares this avatar data for the virtual reality environment.

[0667] The user then accesses the situation setting screen and selects a specific situation (e.g., handling a complaint, explaining a new product, etc.). The server loads the script or scenario corresponding to the selected situation and completes preparations.

[0668] The user then puts on a VR device (e.g., Oculus Rift) and enters the simulation environment. When the user speaks to the customer avatar, the device (VR device) collects the speech and transmits it to the server in real time. The server receives the speech and converts it into text using a speech analysis engine (e.g., Google Speech-to-Text).

[0669] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The generated response is sent to the device in real time and displayed as the voice and animation of the customer avatar. The user can continue to interact with this response and experience various situations.

[0670] The server records the interactions between the user and the customer avatar as logs, and after the training is completed, the server generates and provides feedback to the user from the analyzed logs, which allows the user to effectively improve their customer service skills.

[0671] Specific examples

[0672] For example, consider a situation where a customer is handling a complaint at a fashion retailer. The user sets the persona as "a woman in her 30s who makes many complaints and is knowledgeable about brands." The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior. The user then starts the simulation in the "complaint handling" situation.

[0673] When a user says, "There's a problem with this product," the device sends the speech to a server. The server analyzes the speech and uses a generative AI model to generate a real-time response, such as "I'm sorry. Could you please tell me more?", which is displayed on the device. The user can then ask further questions about the response, practicing realistic customer interactions.

[0674] Prompt Sentence Examples

[0675] "Generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand."

[0676] "Please load a script for a complaint handling situation."

[0677] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0678] Step 1:

[0679] A user logs into the system:

[0680] Input: Username and Password

[0681] What happens: A user enters their username and password on the login screen and clicks the "Login" button.

[0682] Data processing: The server receives the entered authentication information and checks it against a database.

[0683] Output: If authentication is successful, display the user's dashboard.

[0684] Step 2:

[0685] User accesses the Persona settings screen:

[0686] Input: Persona information (age, gender, hobbies, characteristics)

[0687] Action: The user selects the "Persona Settings" menu from the dashboard and proceeds to the settings screen.

[0688] Output: Displays an interface for entering persona information.

[0689] Step 3:

[0690] User enters persona information and submits:

[0691] Input: Values ​​corresponding to age, gender, hobbies, and characteristics

[0692] How it works: The user enters their persona information and clicks the "Submit" button.

[0693] Data processing: The server receives the entered data and performs validation (checking the accuracy and completeness of the data).

[0694] Output: Save the persona information that passed validation to the database.

[0695] Step 4:

[0696] The server invokes the generative AI model to generate the customer avatar:

[0697] Input: Saved persona information

[0698] How it works: The server calls the generative AI model based on the persona information, generates a prompt (e.g., "Please generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand"), and sends it.

[0699] Data Computation: A generative AI model generates the appearance, clothing, and behavior of a customer avatar based on the prompt text.

[0700] Output: The generated customer avatar data is sent back to the server.

[0701] Step 5:

[0702] The server prepares the customer avatar data for the VR environment:

[0703] Input: Generated avatar data

[0704] How it works: The server converts the received avatar data into a format suitable for the virtual reality environment.

[0705] Output: Avatar data that can be used in a virtual reality environment

[0706] Step 6:

[0707] User accesses the situation settings screen and selects a situation:

[0708] Input: Specific situation (e.g., handling a complaint, explaining a new product, etc.)

[0709] How it works: The user accesses the situation setup screen, selects a situation from the drop-down list, and presses the "Set Situation" button.

[0710] Output: Loads the scripts and scenarios corresponding to the selected situations into the server.

[0711] Step 7:

[0712] The user puts on the VR device and enters the simulated environment:

[0713] Input: Set situation

[0714] How it works: The user puts on a compatible VR device (e.g., Oculus Rift) and enters the simulated environment.

[0715] Output: The state in which the user begins interacting with the customer avatar in the simulated environment.

[0716] Step 8:

[0717] The user speaks to the customer avatar and audio is collected:

[0718] Input: User's voice

[0719] How it works: When a user speaks to a customer avatar, the terminal (VR device) collects the voice and sends it to the server in real time.

[0720] Output: The collected audio data is sent to the server.

[0721] Step 9:

[0722] The server analyzes the audio data and converts it to text:

[0723] Input: Collected audio data

[0724] How it works: The server receives the audio data and converts it to text using a speech analysis engine (e.g., Google Speech-to-Text).

[0725] Data calculation: Converting voice data into text

[0726] Output: Parsed text data

[0727] Step 10:

[0728] The server uses the generative AI model to generate a response for the customer avatar:

[0729] Input: Parsed text data

[0730] How it works: The server uses a generative AI model to generate a response for the customer avatar based on the parsed text.

[0731] Data Calculation: Generating Response Text

[0732] Output: Generated response data

[0733] Step 11:

[0734] The server sends the response data to the device:

[0735] Input: Generated response data

[0736] Operation: The server sends the generated response data to the terminal.

[0737] Output: Response data sent to the terminal

[0738] Step 12:

[0739] The device displays the response as audio and animation of a customer avatar:

[0740] Input: Response data sent

[0741] Action: The device displays the response as audio and animation of the customer avatar, providing feedback to the user.

[0742] Output: Customer avatar response displayed to the user

[0743] Step 13:

[0744] The server logs the interaction and generates feedback:

[0745] Input: User and customer avatar interaction data

[0746] How it works: The server logs interactions and generates feedback from the logs that are analyzed after the training is completed.

[0747] Output: Feedback provided to the user

[0748] (Application example 1)

[0749] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] Modern customer service training lacks practical training tools to improve customer service skills. It's also difficult to recreate realistic situations and allow trainees to gain diverse experience in a manner that closely resembles the real world. Traditional training methods also make it difficult to provide real-time feedback or use scripts tailored to specific situations. As a result, the results of training are often not fully utilized in the workplace.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0752] In this invention, the server includes means for providing an interface for the trainee to input customer persona information, means for using a generative model to generate a customer avatar based on the persona information, means for using a virtual reality device for the trainee to interact with the customer avatar, means for using a data processing device to generate responses of the customer avatar in real time and provide them to the virtual reality device, means for recording a log of the interaction and providing feedback to the trainee, and means for loading scripts and events corresponding to specific situations such as product explanations and order taking in a physical store environment, thereby allowing the trainee to experience realistic customer interaction simulations and improve their response skills and ability to respond.

[0753] An "interface" is a point of contact between a device or software and a user for inputting or outputting information.

[0754] A "generative model" is an algorithm or program that automatically generates customer avatars based on persona information.

[0755] A "virtual reality device" is a device used to interact with a virtual reality environment, including head-mounted displays and VR goggles.

[0756] A "data processing device" is a hardware or software system that analyzes and processes input voice or text data and generates and transmits an appropriate response.

[0757] A "script" is a scenario or script for responding to a specific situation or event.

[0758] "Interaction" refers to the interaction between the trainee and the customer avatar, including conversation and behavioral exchanges.

[0759] "Persona information" refers to information such as age, gender, hobbies, and characteristics that the trainee enters to define the characteristics of a virtual customer.

[0760] A "customer avatar" is a virtual customer character generated based on persona information.

[0761] A "log" is a record of the interaction between the trainee and the customer avatar, and is data that is later used for analysis and feedback.

[0762] "Feedback" refers to the results, evaluations, and advice analyzed based on interaction logs, and is provided to trainees.

[0763] The present invention relates to a customer service training system that utilizes virtual reality. Hereinafter, an embodiment of this system will be described in detail.

[0764] System Configuration

[0765] User Login and Persona Configuration

[0766] The server provides an interface for users to log in and enter persona information (age, gender, hobbies, characteristics, etc.). This interface can be implemented as a web application or a mobile application, for example. The persona information is sent to the server and stored in a database (for example, MySQL or PostgreSQL).

[0767] Customer avatar generation

[0768] The server uses the stored persona information to invoke a generation algorithm that generates a customer avatar based on this information, using a generative AI model such as OpenAI's GPT-4, and then converts the generated customer avatar into a format that can be used in a virtual reality environment.

[0769] Using a Virtual Reality Device

[0770] The user enters the simulated environment wearing a virtual reality device (e.g., a head-mounted display), such as the Oculus Rift or HTC Vive.

[0771] Setting up situations and generating responses

[0772] The user selects a specific situation (e.g., handling a complaint, explaining a new product, accepting an order, etc.) through the situation setting interface, and the server loads the scripts and events corresponding to this situation.

[0773] When a user speaks to the customer avatar, the virtual reality device collects the speech data and sends it to a server. The server converts the speech into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The converted text is analyzed by a generative AI model (e.g., GPT-4) to generate an appropriate response. This response is sent to the user's virtual reality device in real time.

[0774] Interaction recording and feedback

[0775] The server records the user's interactions with the customer avatar as a log, which allows for later analysis and feedback. After the training is completed, the server analyzes the log and provides feedback to the user, including specific advice and evaluations to improve the user's response skills.

[0776] Specific examples

[0777] For example, consider a situation where a new product is being introduced in a physical store environment.

[0778] 1. The user persona is set as "a woman in her 20s, with a high desire to purchase and a keen sense of fashion."

[0779] 2. The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior.

[0780] 3. The user starts the simulation in the situation of "explaining a new product."

[0781] 4. When the user says, "Tell me about this new dress," the server analyzes the speech and generates a real-time response, such as, "This dress has a design that incorporates the latest trends..."

[0782] Prompt Sentence Examples

[0783] User: "Tell me about this new dress."

[0784] Example prompt for a generative AI model (GPT-4): "Describe a new dress for a fashion-conscious female customer in her 20s with a high purchasing intent."

[0785] Generated AI response: "This dress is designed with the latest trends in mind, making it perfect for spring and summer. It's made from premium materials and is incredibly comfortable. Plus, this dress..."

[0786] In this way, users can receive training in a variety of situations through the virtual reality system, improving their actual customer service skills.

[0787] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0788] Step 1:

[0789] A user logs into a customer service training application. The input is the user's login information (e.g., username and password), which the server verifies and authenticates the user. The output is the main interface screen after the user has logged in.

[0790] Step 2:

[0791] The user accesses an interface to input persona information. The input is persona information such as age, gender, hobbies, and characteristics, which the server receives and stores in a database. The output is a confirmation message that the persona information has been saved.

[0792] Step 3:

[0793] The server calls the generative model based on the stored persona information. The input is the persona information (e.g., "female in her 30s, frequent complainer, knowledgeable about brands"), and generates a customer avatar using a generative AI model (e.g., OpenAI GPT-4). The output is the data of the generated customer avatar.

[0794] Step 4:

[0795] The user puts on the virtual reality device and enters the training simulation environment. The input is a start-up signal for the virtual reality device, and the terminal loads the simulation environment. The output is a confirmation screen that the user has entered the virtual reality environment.

[0796] Step 5:

[0797] The user selects a specific situation through the situation setting interface. The input is the situation information (e.g., "Handling a complaint," "Explaining a new product," "Taking an order"), and the server loads the associated scripts and events. The output is a message indicating that the selected situation is ready.

[0798] Step 6:

[0799] When a user speaks to the customer avatar, the virtual reality device collects the voice data and sends it to the server in real time. The input is voice data, and the server converts it into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The output is the voice-to-text data.

[0800] Step 7:

[0801] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The input is the user's textual utterance, and a natural-sounding response is generated through the generative AI model (e.g., GPT-4). The output is the generated response text.

[0802] Step 8:

[0803] The server sends the generated response data to the virtual reality device, where it is displayed as voice and animation of the customer avatar. The input is the response text, which the device converts into voice data using a speech synthesis engine and synchronizes with the avatar's movements. The output is the avatar's response animation.

[0804] Step 9:

[0805] The server logs the interactions between the user and the customer avatar and stores the data for later analysis. The input is the conversation log between the user and the avatar, and the output is the log data stored in the database.

[0806] Step 10:

[0807] After the training is completed, the server analyzes the logs and generates feedback. The input is the saved log data, and the data analysis algorithm evaluates the user's response skills. The output is a feedback report provided to the user.

[0808] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0809] MODE FOR CARRYING OUT THE INVENTION

[0810] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The system is implemented as follows.

[0811] Persona Settings

[0812] 1. A user logs in to the system and accesses the persona configuration screen.

[0813] 2. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[0814] 3. The server receives the persona information and stores it in a database.

[0815] Customer avatar generation

[0816] 1. The server calls the generative AI model using the saved persona information.

[0817] 2. The generation AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[0818] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[0819] Setting the situation

[0820] 1. The user accesses the situation setting screen and selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[0821] 2. The user presses the "Set Situation" button.

[0822] 3. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[0823] Simulation and Real-Time Response

[0824] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[0825] 2. The user speaks to the customer avatar.

[0826] 3. The device collects the user's voice, facial expressions, and gestures using the camera and microphone, and transmits them to the server in real time.

[0827] Emotion Recognition and Response Generation

[0828] 1. The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine.

[0829] 2. Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[0830] 3. The server sends the generated response data to the terminal.

[0831] 4. The device displays the response as audio and animation of the customer avatar.

[0832] Interactive training experience and feedback

[0833] 1. Users continuously interact with the customer avatar and experience various situations.

[0834] 2. The server records the user's interactions with the customer avatar and the emotion recognition logs.

[0835] 3. After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data.

[0836] Specific examples

[0837] For example, let's take the situation of handling a complaint at a restaurant.

[0838] 1. The user persona is set as "a man in his 40s with extensive experience in the restaurant industry who is very prone to dissatisfaction."

[0839] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[0840] 3. The user starts the simulation in the situation of "handling a complaint."

[0841] 4. When the user says, "This dish is cold," and their face turns grim, the device sends the voice and facial expression to the server.

[0842] 5. The server analyzes the voice and facial expression data and recognizes "dissatisfaction" using an emotion engine.

[0843] 6. The server uses a generation AI to generate a response such as, "Sorry. We will bring you a new dish right away," and sends it to the device.

[0844] 7. The terminal displays a customer avatar responding with voice and animation.

[0845] 8. The user continues to ask questions about the response, practicing realistic customer interactions using emotion recognition.

[0846] In this way, this system, which combines an emotion engine, can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[0847] The processing flow will be explained below.

[0848] MODE FOR CARRYING OUT THE INVENTION

[0849] This invention is a virtual reality system for customer service training that combines an emotion engine that recognizes the user's emotions. The specific processing flow is shown below.

[0850] Processing Steps

[0851] Step 1:

[0852] A user logs in to the system and accesses the persona configuration screen.

[0853] Step 2:

[0854] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[0855] Step 3:

[0856] The server receives the persona information and stores it in a database.

[0857] Step 4:

[0858] The server calls the generative AI model based on the stored persona information.

[0859] Step 5:

[0860] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[0861] Step 6:

[0862] The server prepares the generated customer avatar data for the virtual reality environment.

[0863] Step 7:

[0864] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[0865] Step 8:

[0866] The user presses the "Set Situation" button.

[0867] Step 9:

[0868] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[0869] Step 10:

[0870] The user puts on the VR device (terminal) and enters the simulation environment.

[0871] Step 11:

[0872] The user speaks to the customer avatar.

[0873] Step 12:

[0874] The device collects the user's voice, facial expressions, and gestures using a camera and microphone, and transmits the data to a server in real time.

[0875] Step 13:

[0876] The server analyzes the received voice, facial expressions, and gesture data, and recognizes the user's emotions using an emotion engine.

[0877] Step 14:

[0878] Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[0879] Step 15:

[0880] The server transmits the generated response data to the terminal.

[0881] Step 16:

[0882] The terminal displays the response as audio and animation of a customer avatar.

[0883] Step 17:

[0884] Users continuously interact with the customer avatar and experience various situations.

[0885] Step 18:

[0886] The server logs the user's interactions with the customer avatar and the emotion recognition.

[0887] Step 19:

[0888] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[0889] Specific examples

[0890] For example, let's take the situation of handling a complaint at a restaurant.

[0891] Step 1:

[0892] A user logs in to the system and accesses the persona configuration screen.

[0893] Step 2:

[0894] The user enters the persona information as "male in his 40s, has extensive experience in the restaurant industry, and is very prone to dissatisfaction" and clicks the "Submit" button.

[0895] Step 3:

[0896] The server receives the persona information and stores it in a database.

[0897] Step 4:

[0898] The server calls the generative AI model based on the stored persona information.

[0899] Step 5:

[0900] The generation AI generates a customer avatar with specific clothing and behavior based on the persona information entered.

[0901] Step 6:

[0902] The server prepares the generated customer avatar data for the virtual reality environment.

[0903] Step 7:

[0904] The user accesses the situation setting screen and selects the situation "Handling a complaint."

[0905] Step 8:

[0906] The user presses the "Set Situation" button.

[0907] Step 9:

[0908] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[0909] Step 10:

[0910] The user puts on the VR device (terminal) and enters the simulation environment.

[0911] Step 11:

[0912] The user complains to the customer avatar, saying, "This food is cold."

[0913] Step 12:

[0914] The device collects the user's voice and facial expressions and transmits them to a server in real time.

[0915] Step 13:

[0916] The server analyzes the voice, facial expressions, and gesture data it receives, and uses an emotion engine to recognize "dissatisfaction."

[0917] Step 14:

[0918] Based on the results of emotion recognition, the server uses generative AI to generate a response such as, "We're sorry. We'll bring you a new dish right away."

[0919] Step 15:

[0920] The server transmits the generated response data to the terminal.

[0921] Step 16:

[0922] The device displays a customer avatar responding with voice and animation.

[0923] Step 17:

[0924] The user continues to ask further questions based on the responses, practicing realistic customer interactions using emotion recognition.

[0925] Step 18:

[0926] The server logs the user's interactions with the customer avatar and the emotion recognition.

[0927] Step 19:

[0928] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[0929] Example 2

[0930] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0931] Modern customer service training requires practical and interactive training methods. However, traditional training methods have difficulty recreating real-life customer interactions, limiting the opportunities for trainees to demonstrate their actual skills. Furthermore, the lack of a system that can recognize trainees' emotions and reactions in real time and provide adaptive feedback based on that information makes it difficult to achieve deeper learning.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0933] In this invention, the server includes means for collecting and transmitting the voice, facial expressions, and gestures of the trainee to the server, means for analyzing the received voice, facial expressions, and gesture data to recognize emotions, and means for using a generative artificial intelligence to adjust and generate responses of a customer avatar based on the results of the emotion recognition. This makes it possible to recognize the emotions and reactions of the trainee in real time and provide adaptive feedback based on them.

[0934] "Training Recipient" means a person receiving customer service training.

[0935] "Persona information" refers to data such as age, gender, hobbies, and characteristics that are used as basic information when generating a customer avatar.

[0936] "Interface" refers to the screen or operating means used by trainees to enter persona information.

[0937] "Generative AI" is an AI technology for generating the appearance, clothing, and behavior of a customer avatar based on persona information.

[0938] "Virtual reality device" means equipment used by a trainee to interact with a customer avatar in a virtual environment.

[0939] "Server" means a computer system for storing persona information and providing responses of generated customer avatars.

[0940] "Voice, facial expression, and gesture data" refers to information about the voice, facial expression, and body movements of the training subject, and is data used for emotion recognition.

[0941] "Emotion recognition" is a technology that grasps the emotions of trainees from their voices, facial expressions, and gestures.

[0942] A "customer avatar" is a virtual character that is generated based on persona information and interacts with the training subject in a virtual environment.

[0943] "Response" refers to the reply or action that the customer avatar gives in response to an interaction from the training subject.

[0944] "Log" refers to data that records the interactions between the training subject and the customer avatar and the results of emotion recognition.

[0945] "Feedback" refers to information, including evaluations and areas for improvement, that is provided to trainees after the training has been completed.

[0946] "Specific situations" are specific situations or scenes that trainees experience in a virtual environment, such as handling complaints, explaining new products, or accepting orders.

[0947] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The main components of this system include a user terminal, a server, a generative AI model, and a virtual reality device. Specific implementation methods are described below.

[0948] First, the user logs in to the system and accesses the persona setting screen. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. The server receives the submitted persona information and stores it in a database. The server then invokes a generative AI model based on the stored persona information to generate the appearance, clothing, and behavior of the customer avatar. An example of a prompt message for this is as follows:

[0949] "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[0950] The generated customer avatar data is prepared for the virtual reality environment by the server. The user accesses the situation setting screen and selects a specific situation. Examples include "handling a complaint," "explaining a new product," and "accepting an order." When the user presses the "Set Situation" button, the server loads the script or scenario corresponding to the selected situation and completes the preparation.

[0951] Next, the user puts on the VR device (terminal) and enters the simulation environment. When the user speaks to the customer avatar, the terminal collects the user's voice, as well as facial expressions and gestures using a camera and microphone, and sends the collected data to the server in real time. The server analyzes the received voice, facial expressions, and gesture data and recognizes the user's emotions using an emotion engine. For example, if a user says, "This dish is cold," and their face turns grim, the server analyzes the voice and facial expression data and recognizes the emotion as "dissatisfied."

[0952] Based on the emotion recognition results, the server uses a generative AI model to adjust and generate a response for the customer avatar. For example, a response such as "Sorry, we'll bring you a new dish right away" may be generated. The server then sends the generated response data to the device, which then displays the customer avatar's response with voice and animation.

[0953] In this way, the user continuously interacts with the customer avatar and experiences various situations. The server records the interaction between the user and the customer avatar and the emotion recognition log, analyzes the log after the training is completed, and provides the user with feedback including the emotion data.

[0954] For example, in a situation where a customer is handling a complaint at a restaurant, the user sets their persona as "a man in his 40s with extensive restaurant experience who is very prone to becoming dissatisfied." The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior, and the user begins the simulation in the "handling a complaint" situation. When the user says, "This dish is cold," the device sends their voice and facial expression to the server, which analyzes it and generates an appropriate response, which is sent back to the device.

[0955] In this way, the system can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[0956] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0957] Step 1:

[0958] A user logs in to the system and accesses the persona configuration screen. They provide their username and password as input, and the server performs the authentication process. After successful authentication, they are redirected to the persona configuration screen.

[0959] Step 2:

[0960] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. Input data includes age "40", gender "male", hobbies "reading", and characteristics "kind". The server receives the persona information and saves it in the database. Specifically, the server analyzes the persona information sent, generates an SQL statement to insert into the database, connects to the database, and executes the SQL statement.

[0961] Step 3:

[0962] The server calls the generative AI model using the saved persona information. It uses the persona information as input to generate a prompt. This prompt is passed to the generative AI model to generate a customer avatar. As output, appearance information (hair color, clothing style, etc.) and behavior information are obtained. An example prompt is "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[0963] Step 4:

[0964] The server prepares the generated customer avatar data for the virtual reality environment. It uses the generated appearance and behavior information as input data, converts it into a format suitable for the VR environment, and organizes the necessary files. As an output, it obtains customer avatar data for the VR environment.

[0965] Step 5:

[0966] The user accesses the situation setting screen and selects a specific situation from a drop-down list. The user selects a situation (e.g., "Complaint handling") as input data and presses the "Set Situation" button. The server retrieves the script corresponding to the selected scenario from the database and loads it into memory. The scenario data is prepared as output.

[0967] Step 6:

[0968] The user puts on the VR device (terminal) and enters the simulation environment. The input is to start up the VR device and connect it to the system. The terminal starts up the VR environment and places the user in the simulation. This allows the user to start training in the virtual environment.

[0969] Step 7:

[0970] The user speaks to the customer avatar. For example, they may say, "This dish is cold." The device collects the user's voice and simultaneously records their facial expressions and gestures using a camera and microphone. This data is sent to the server in real time. The collected voice, facial expressions, and gesture data is sent to the server as output.

[0971] Step 8:

[0972] The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine. Voice data, facial expression data, and gesture data are used as input, and an emotion recognition algorithm is applied. The output is an emotion result such as "dissatisfied."

[0973] Step 9:

[0974] The server uses generative AI to adjust and generate the customer avatar's response based on the emotion recognition results. The emotion recognition results and generative AI model are used as input. When the generative AI model is sent a prompt such as "How should you respond to an angry customer?", the output generated is a response such as "We're sorry. We'll bring you a new dish right away."

[0975] Step 10:

[0976] The server sends the generated response data to the terminal, uses the generated response data as input and sends it to the terminal, and when the terminal receives the response data, displays the response as voice and animation of the customer avatar.

[0977] Step 11:

[0978] Users continually interact with customer avatars and experience various situations, allowing them to simulate real-life customer service scenarios and improve their response skills.

[0979] Step 12:

[0980] The server records the interaction between the user and the customer avatar and the emotion recognition log. Each interaction and the emotion recognition results are used as input and stored in a database. After the training is completed, the log is analyzed and feedback, including the emotion data, is provided to the user.

[0981] Step 13:

[0982] After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data. Using the recorded log data as input, the server generates a report that includes the user's behavioral trends and areas for improvement. Detailed feedback is provided to the user as output.

[0983] (Application example 2)

[0984] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0985] Traditional customer service training systems are limited to standard scenarios and situations, and are unable to respond to trainees' emotions and reactions in real time. This limits the effectiveness of training and causes problems such as a lack of flexibility and responsiveness required for actual customer interactions. It is also difficult to track trainees' emotions and provide appropriate feedback based on them.

[0986] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0987] In this invention, the server includes means for collecting voice data and facial expression and gesture data captured by a camera of the trainee and transmitting them to the server in real time, means for analyzing the collected data and recognizing the trainee's emotions with an emotion engine, and means for adjusting and generating responses of a customer avatar using generative artificial intelligence based on the results of emotion recognition. This makes it possible to generate responses in real time based on the trainee's emotions and improve customer service skills in brick-and-mortar stores in a manner closer to reality.

[0988] "Training Participant" means a person who participates in a customer service training program and seeks to improve their skills using a virtual reality system.

[0989] "Persona information" refers to personal information such as age, gender, hobbies, and characteristics entered by the training participant, and a customer avatar is generated based on this information.

[0990] "Generative AI" refers to AI that has the ability to generate new data based on input data, and is used to generate the appearance and behavior of customer avatars.

[0991] A "virtual reality device" is a device that displays a virtual reality space and allows users to interact with that space. Examples include head-mounted displays and VR goggles.

[0992] A "server" is a computer that processes and stores data over a network, and is used here for generating responses for customer avatars, recording logs, and analyzing emotions.

[0993] A "customer avatar" is a virtual person generated within a virtual reality system based on persona information, and is responsible for interacting with the training recipient.

[0994] An "emotion engine" is software that analyzes voice data, facial expression data, and gesture data to recognize the emotional state of the subject.

[0995] An "interface" refers to the operation screen and input device that trainees use to interact with the system.

[0996] A "log" is a record of the system's operation history and data, and in this case, it records the interaction between the training participant and the customer avatar.

[0997] "Feedback" refers to the evaluation and areas for improvement provided to the training participants, and is provided based on information analyzed from the logs.

[0998] A "scenario" is a sequence of events or dialogue used in a specific situation or setting, and is used to set the scene for training.

[0999] This invention is a system that combines a virtual reality system for customer service training with an emotion engine that recognizes the emotions of users. Specific embodiments will be described below.

[1000] Structure and program overview

[1001] This system allows trainees to input customer persona information and interact with a customer avatar generated based on that information. The system uses a smartphone or a head-mounted display (VR device) as hardware, and includes a server for generating real-time responses, an emotion engine for recognizing emotions, and generative artificial intelligence.

[1002] Hardware and Software Used

[1003] Smartphone or head-mounted display (VR device): A device that allows users to interact in a virtual space.

[1004] Server: A computer that processes data, analyzes it, and generates responses.

[1005] Emotion engine: Software that analyzes voice data, facial expression data, and gesture data to recognize the user's emotions.

[1006] Generative artificial intelligence: Software that generates customer avatars based on persona information and generates responses in real time based on user sentiment.

[1007] Data processing and calculation

[1008] 1. Login and Persona Settings

[1009] The user logs in to the smartphone app and enters their age, gender, hobbies, and characteristics on the persona setting screen. This persona information is sent to the server and stored in the database.

[1010] 2. Customer avatar generation

[1011] The server passes the persona information to a generation AI to generate the appearance, clothing, and behavior of the customer avatar, and the generated data is then appropriately prepared for the VR environment.

[1012] 3. Setting the situation

[1013] The user selects a training situation on the app screen, and the server loads the corresponding script and scenario.

[1014] 4. Simulation and real-time response

[1015] The user puts on the VR device and begins interacting with a customer avatar in a virtual store.

[1016] The smartphone or VR device collects the user's voice data, facial expression data, and gesture data and transmits them to a server in real time.

[1017] The server's emotion engine analyzes the data and recognizes the user's emotions.

[1018] Based on the results of emotion recognition, the generative AI model generates a response for the customer avatar, which the server then sends to the device, allowing the avatar to respond in real time.

[1019] Specific examples

[1020] Customer Service Training Scenarios

[1021] 1. The user enters "Female in her 30s, gentle personality, retail store worker, positive" on the "Persona Settings" screen and submits.

[1022] 2. The server generates a "gentle customer avatar, a woman in her 30s" based on the persona information.

[1023] 3. The user selects and sets the situation "Explanation of a new product."

[1024] 4. The user wearing the VR device speaks to the avatar, asking, "What are the features of this new product?" The camera captures the user's facial expressions and sends them to the server along with the audio data.

[1025] 5. The server analyzes the user's facial expressions and uses an emotion engine to recognize "interest." The generative AI model generates a response, such as "This new product is compact, lightweight, and made from eco-friendly materials." The server then sends the response to the device, where the customer avatar responds with voice and animation.

[1026] Prompt Sentence Examples

[1027] Persona information: Female in her 30s, gentle personality, retail store worker, positive

[1028] Situation: New product description

[1029] User Sentiment: Interest

[1030] What it produces: This new product is compact, lightweight and made from eco-friendly materials.

[1031] This allows users to improve their skills in dealing with real customers.

[1032] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1033] Step 1:

[1034] A user logs into the system and enters persona information

[1035] Input: The user enters persona information such as age, gender, hobbies, and characteristics.

[1036] How it works: The user enters persona information (e.g., female in her 30s, gentle personality, retail worker, positive) through the interface and clicks the "Submit" button.

[1037] Output: Persona information is sent to the system and stored on the server.

[1038] Step 2:

[1039] The server generates a customer avatar based on the persona information

[1040] Input: Saved persona information.

[1041] How it works: The server retrieves persona information and uses generative artificial intelligence (generative AI model) to generate the appearance, clothing, and behavior of a customer avatar.

[1042] Output: The generated customer avatar is prepared for the virtual reality environment.

[1043] Step 3:

[1044] Users set training situations

[1045] Input: Select a training situation (e.g., explaining a new product).

[1046] How it works: The user selects a specific situation on the situation setting screen and presses the "Set Situation" button.

[1047] Output: The scripts and scenarios corresponding to the selected situations are loaded by the server.

[1048] Step 4:

[1049] The user puts on the VR device and starts the simulation.

[1050] Input: Customer avatar and situation script generated based on persona information.

[1051] How it works: A user puts on a VR device and begins interacting with a customer avatar in a virtual store.

[1052] Output: The interaction is initiated by the user speaking to the customer avatar.

[1053] Step 5:

[1054] The device collects the user's voice, facial expressions, and gestures and sends them to the server.

[1055] Input: User's voice data, facial expression data captured by the camera, and gesture data.

[1056] Actions: The device's sensors capture the user's voice, facial expressions, and gestures and transmit them to the server in real time.

[1057] Output: The user's voice, facial expression, and gesture data are sent to the server.

[1058] Step 6:

[1059] The server recognizes the user's emotions using an emotion engine

[1060] Input: Collected voice data, facial expression data, and gesture data.

[1061] How it works: The server's emotion engine analyzes the collected data and recognizes the user's emotions (e.g., interest, frustration).

[1062] Output: The analyzed emotion information is passed to the generative AI model.

[1063] Step 7:

[1064] The server generates a response for the customer avatar based on the emotion recognition results.

[1065] Input: Emotion recognition results, generative artificial intelligence model.

[1066] How it works: The server uses the generative AI model to generate an appropriate customer avatar response based on the recognized emotion (e.g., "This new product is compact, lightweight, and made from eco-friendly materials.").

[1067] Output: The generated response data is sent to the terminal.

[1068] Step 8:

[1069] The device displays the response to the customer avatar

[1070] Input: The response data of the generated customer avatar.

[1071] How it works: The device displays and plays the generated response data with animation and audio, and the customer avatar responds in real time.

[1072] Output: The user confirms the generated response and continues the interaction.

[1073] Step 9:

[1074] The server logs the interaction and provides feedback to the trainee.

[1075] Input: All interaction data for the user and customer avatar.

[1076] How it works: The server logs interactions and later analyzes them to provide feedback to the user.

[1077] Output: The user receives feedback including the results of the training and areas for improvement.

[1078] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1079] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1080] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1081] [Third embodiment]

[1082] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1083] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1084] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1085] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1086] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1087] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1088] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1089] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1090] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1091] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1092] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1093] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1094] MODE FOR CARRYING OUT THE INVENTION

[1095] The virtual reality system for customer service training of the present invention is implemented as follows.

[1096] Customer persona creation

[1097] 1. The user logs in to the system and accesses the persona setting screen.

[1098] 2. The user enters the following persona information into the form:

[1099] age

[1100] sex

[1101] hobby

[1102] Characteristics (e.g., many complaints, high purchasing intent, etc.)

[1103] 3. After entering the information, the user clicks the "Submit" button.

[1104] 4. The server receives the persona information and stores it in a database.

[1105] Customer avatar generation

[1106] 1. The server calls the generative AI model using the saved persona information.

[1107] 2. Generative AI generates the appearance, clothing, and behavior of a customer avatar based on the input persona information.

[1108] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[1109] Setting the situation

[1110] 1. The user accesses the situation setting screen.

[1111] 2. The user selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[1112] 3. The user presses the "Set Situation" button.

[1113] 4. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[1114] Simulation and Real-Time Response

[1115] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[1116] 2. When the user speaks to the customer avatar,

[1117] The device collects the audio and transmits it to a server in real time.

[1118] 3. The server receives the voice data and converts it into text using a speech analysis engine.

[1119] 4. The server uses generative AI to generate a response for the customer avatar based on the analyzed text.

[1120] 5. The server sends the generated response data to the terminal.

[1121] 6. The terminal displays the response as a voice and animation of the customer avatar.

[1122] Providing an interactive training experience

[1123] 1. Users continuously interact with the customer avatar and experience various situations.

[1124] 2. The server logs the user's interactions with the customer avatar for later analysis.

[1125] 3. After the training is completed, the server generates feedback from the analyzed logs and provides it to the user.

[1126] Specific examples

[1127] For example, let's take the situation of handling a complaint at a fashion retail store.

[1128] 1. The user persona is set as "a woman in her 30s who makes many complaints and is knowledgeable about brands."

[1129] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[1130] 3. The user starts the simulation in the situation of "handling a complaint."

[1131] 4. When the user says, "There is a problem with this product," the device sends the speech to the server.

[1132] 5. The server analyzes the voice and uses generative AI to generate a real-time response such as "Sorry, can you tell me more?", which is displayed on the device.

[1133] 6. Users can then ask further questions about the responses to practice realistic customer interactions.

[1134] In this way, users can receive training for a variety of situations through the virtual reality system, improving their actual customer service skills.

[1135] The processing flow will be explained below.

[1136] Step 1:

[1137] A user logs in to the system and accesses the persona configuration screen.

[1138] Step 2:

[1139] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[1140] Step 3:

[1141] The server receives the persona information and stores it in a database.

[1142] Step 4:

[1143] The server calls the generative AI model based on the stored persona information.

[1144] Step 5:

[1145] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[1146] Step 6:

[1147] The server prepares the generated customer avatar data for the virtual reality environment.

[1148] Step 7:

[1149] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[1150] Step 8:

[1151] The user presses the "Set Situation" button.

[1152] Step 9:

[1153] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[1154] Step 10:

[1155] The user puts on the VR device (terminal) and enters the simulation environment.

[1156] Step 11:

[1157] The user speaks to the customer avatar.

[1158] Step 12:

[1159] The device collects the user's voice and transmits it to the server in real time.

[1160] Step 13:

[1161] The server analyzes the received voice data and converts it into text.

[1162] Step 14:

[1163] The server uses a generative AI to generate a response for the customer avatar based on the analyzed text.

[1164] Step 15:

[1165] The server transmits the generated response data to the terminal.

[1166] Step 16:

[1167] The terminal displays the response as audio and animation of a customer avatar.

[1168] Step 17:

[1169] The user and customer avatar continuously interact and experience various situations.

[1170] Step 18:

[1171] The server logs the interactions between the user and the customer avatar.

[1172] Step 19:

[1173] The server analyzes the log after the training is completed and provides feedback to the user.

[1174] Example 1

[1175] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1176] Traditional customer service training has limited resources and flexibility to effectively improve real-world customer interaction skills. For example, it is difficult to perfectly recreate different customer personas and diverse situations in a real-world training environment, resulting in a lack of practical training. Additionally, feedback on customer interactions is generally not immediate. This creates the problem of taking a long time to acquire appropriate customer interaction skills.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1178] In this invention, the server includes: means for providing an interface for a trainee to input customer persona information; means for using a generative AI model to generate a customer avatar based on the persona information; means for the trainee to use a virtual reality device to interact with the customer avatar; means for using a server to generate responses using a voice analysis engine and the generative AI model to generate responses for the customer avatar in real time and provide the responses to the virtual reality device; and means for recording a log of the interactions and providing feedback to the trainee. This effectively recreates different customer personas and a variety of situations, allowing the trainee to acquire practical customer service skills in real time and receive immediate feedback.

[1179] The "Virtual Reality System for Customer Service Training" is a system that allows trainees to train customer service skills using a virtual reality environment.

[1180] "Training Recipient" means an individual receiving training to improve their customer service skills.

[1181] "Persona information" refers to attribute information such as the age, gender, hobbies, and characteristics of a virtual customer.

[1182] "Interface" refers to a screen or input device that allows a user to input and operate persona information.

[1183] "Generative AI model" refers to a model that uses artificial intelligence technology to generate customer avatars based on input persona information.

[1184] "Virtual Reality Device" means a device (e.g., a VR headset) used by a trainee to interact with a customer avatar in a virtual reality environment.

[1185] "Server" refers to a computer system used to manage the entire system, process data, call generative AI models, record logs, and provide feedback.

[1186] "Speech analysis engine" refers to technology or software for converting voice data into text.

[1187] "Customer avatar" refers to a 3D model of a virtual customer generated based on persona information.

[1188] "Situation" refers to a specific scene or situation in customer service training.

[1189] "Response" refers to the reply or reaction that the customer avatar gives to the training recipient's input.

[1190] "Log" refers to a record of interactions between a user and a customer avatar during training.

[1191] "Feedback" refers to evaluation and advice regarding the user's performance provided after the training is completed.

[1192] The virtual reality system for customer service training of this invention is implemented as follows: A user first logs in to the system and accesses an interface for inputting persona information. Here, the user inputs and submits persona information such as age, gender, hobbies, and characteristics. The server receives this information and stores it in a database.

[1193] The server then invokes a generative AI model based on the stored persona information to generate a customer avatar. This generative AI model uses OpenAI's GPT-3, for example. The generated avatar's appearance, clothing, and behavior are customized based on the persona information. The server then prepares this avatar data for the virtual reality environment.

[1194] The user then accesses the situation setting screen and selects a specific situation (e.g., handling a complaint, explaining a new product, etc.). The server loads the script or scenario corresponding to the selected situation and completes preparations.

[1195] The user then puts on a VR device (e.g., Oculus Rift) and enters the simulation environment. When the user speaks to the customer avatar, the device (VR device) collects the speech and transmits it to the server in real time. The server receives the speech and converts it into text using a speech analysis engine (e.g., Google Speech-to-Text).

[1196] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The generated response is sent to the device in real time and displayed as the voice and animation of the customer avatar. The user can continue to interact with this response and experience various situations.

[1197] The server records the interactions between the user and the customer avatar as logs, and after the training is completed, the server generates and provides feedback to the user from the analyzed logs, which allows the user to effectively improve their customer service skills.

[1198] Specific examples

[1199] For example, consider a situation where a customer is handling a complaint at a fashion retailer. The user sets the persona as "a woman in her 30s who makes many complaints and is knowledgeable about brands." The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior. The user then starts the simulation in the "complaint handling" situation.

[1200] When a user says, "There's a problem with this product," the device sends the speech to a server. The server analyzes the speech and uses a generative AI model to generate a real-time response, such as "I'm sorry. Could you please tell me more?", which is displayed on the device. The user can then ask further questions about the response, practicing realistic customer interactions.

[1201] Prompt Sentence Examples

[1202] "Generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand."

[1203] "Please load a script for a complaint handling situation."

[1204] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1205] Step 1:

[1206] A user logs into the system:

[1207] Input: Username and Password

[1208] What happens: A user enters their username and password on the login screen and clicks the "Login" button.

[1209] Data processing: The server receives the entered authentication information and checks it against a database.

[1210] Output: If authentication is successful, display the user's dashboard.

[1211] Step 2:

[1212] User accesses the Persona settings screen:

[1213] Input: Persona information (age, gender, hobbies, characteristics)

[1214] Action: The user selects the "Persona Settings" menu from the dashboard and proceeds to the settings screen.

[1215] Output: Displays an interface for entering persona information.

[1216] Step 3:

[1217] User enters persona information and submits:

[1218] Input: Values ​​corresponding to age, gender, hobbies, and characteristics

[1219] How it works: The user enters their persona information and clicks the "Submit" button.

[1220] Data processing: The server receives the entered data and performs validation (checking the accuracy and completeness of the data).

[1221] Output: Save the persona information that passed validation to the database.

[1222] Step 4:

[1223] The server invokes the generative AI model to generate the customer avatar:

[1224] Input: Saved persona information

[1225] How it works: The server calls the generative AI model based on the persona information, generates a prompt (e.g., "Please generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand"), and sends it.

[1226] Data Computation: A generative AI model generates the appearance, clothing, and behavior of a customer avatar based on the prompt text.

[1227] Output: The generated customer avatar data is sent back to the server.

[1228] Step 5:

[1229] The server prepares the customer avatar data for the VR environment:

[1230] Input: Generated avatar data

[1231] How it works: The server converts the received avatar data into a format suitable for the virtual reality environment.

[1232] Output: Avatar data that can be used in a virtual reality environment

[1233] Step 6:

[1234] User accesses the situation settings screen and selects a situation:

[1235] Input: Specific situation (e.g., handling a complaint, explaining a new product, etc.)

[1236] How it works: The user accesses the situation setup screen, selects a situation from the drop-down list, and presses the "Set Situation" button.

[1237] Output: Loads the scripts and scenarios corresponding to the selected situations into the server.

[1238] Step 7:

[1239] The user puts on the VR device and enters the simulated environment:

[1240] Input: Set situation

[1241] How it works: The user puts on a compatible VR device (e.g., Oculus Rift) and enters the simulated environment.

[1242] Output: The state in which the user begins interacting with the customer avatar in the simulated environment.

[1243] Step 8:

[1244] The user speaks to the customer avatar and audio is collected:

[1245] Input: User's voice

[1246] How it works: When a user speaks to a customer avatar, the terminal (VR device) collects the voice and sends it to the server in real time.

[1247] Output: The collected audio data is sent to the server.

[1248] Step 9:

[1249] The server analyzes the audio data and converts it to text:

[1250] Input: Collected audio data

[1251] How it works: The server receives the audio data and converts it to text using a speech analysis engine (e.g., Google Speech-to-Text).

[1252] Data calculation: Converting voice data into text

[1253] Output: Parsed text data

[1254] Step 10:

[1255] The server uses the generative AI model to generate a response for the customer avatar:

[1256] Input: Parsed text data

[1257] How it works: The server uses a generative AI model to generate a response for the customer avatar based on the parsed text.

[1258] Data Calculation: Generating Response Text

[1259] Output: Generated response data

[1260] Step 11:

[1261] The server sends the response data to the device:

[1262] Input: Generated response data

[1263] Operation: The server sends the generated response data to the terminal.

[1264] Output: Response data sent to the terminal

[1265] Step 12:

[1266] The device displays the response as audio and animation of a customer avatar:

[1267] Input: Response data sent

[1268] Action: The device displays the response as audio and animation of the customer avatar, providing feedback to the user.

[1269] Output: Customer avatar response displayed to the user

[1270] Step 13:

[1271] The server logs the interaction and generates feedback:

[1272] Input: User and customer avatar interaction data

[1273] How it works: The server logs interactions and generates feedback from the logs that are analyzed after the training is completed.

[1274] Output: Feedback provided to the user

[1275] (Application example 1)

[1276] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1277] Modern customer service training lacks practical training tools to improve customer service skills. It's also difficult to recreate realistic situations and allow trainees to gain diverse experience in a manner that closely resembles the real world. Traditional training methods also make it difficult to provide real-time feedback or use scripts tailored to specific situations. As a result, the results of training are often not fully utilized in the workplace.

[1278] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1279] In this invention, the server includes means for providing an interface for the trainee to input customer persona information, means for using a generative model to generate a customer avatar based on the persona information, means for using a virtual reality device for the trainee to interact with the customer avatar, means for using a data processing device to generate responses of the customer avatar in real time and provide them to the virtual reality device, means for recording a log of the interaction and providing feedback to the trainee, and means for loading scripts and events corresponding to specific situations such as product explanations and order taking in a physical store environment, thereby allowing the trainee to experience realistic customer interaction simulations and improve their response skills and ability to respond.

[1280] An "interface" is a point of contact between a device or software and a user for inputting or outputting information.

[1281] A "generative model" is an algorithm or program that automatically generates customer avatars based on persona information.

[1282] A "virtual reality device" is a device used to interact with a virtual reality environment, including head-mounted displays and VR goggles.

[1283] A "data processing device" is a hardware or software system that analyzes and processes input voice or text data and generates and transmits an appropriate response.

[1284] A "script" is a scenario or script for responding to a specific situation or event.

[1285] "Interaction" refers to the interaction between the trainee and the customer avatar, including conversation and behavioral exchanges.

[1286] "Persona information" refers to information such as age, gender, hobbies, and characteristics that the trainee enters to define the characteristics of a virtual customer.

[1287] A "customer avatar" is a virtual customer character generated based on persona information.

[1288] A "log" is a record of the interaction between the trainee and the customer avatar, and is data that is later used for analysis and feedback.

[1289] "Feedback" refers to the results, evaluations, and advice analyzed based on interaction logs, and is provided to trainees.

[1290] The present invention relates to a customer service training system that utilizes virtual reality. Hereinafter, an embodiment of this system will be described in detail.

[1291] System Configuration

[1292] User Login and Persona Configuration

[1293] The server provides an interface for users to log in and enter persona information (age, gender, hobbies, characteristics, etc.). This interface can be implemented as a web application or a mobile application, for example. The persona information is sent to the server and stored in a database (for example, MySQL or PostgreSQL).

[1294] Customer avatar generation

[1295] The server uses the stored persona information to invoke a generation algorithm that generates a customer avatar based on this information, using a generative AI model such as OpenAI's GPT-4, and then converts the generated customer avatar into a format that can be used in a virtual reality environment.

[1296] Using a Virtual Reality Device

[1297] The user enters the simulated environment wearing a virtual reality device (e.g., a head-mounted display), such as the Oculus Rift or HTC Vive.

[1298] Setting up situations and generating responses

[1299] The user selects a specific situation (e.g., handling a complaint, explaining a new product, accepting an order, etc.) through the situation setting interface, and the server loads the scripts and events corresponding to this situation.

[1300] When a user speaks to the customer avatar, the virtual reality device collects the speech data and sends it to a server. The server converts the speech into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The converted text is analyzed by a generative AI model (e.g., GPT-4) to generate an appropriate response. This response is sent to the user's virtual reality device in real time.

[1301] Interaction recording and feedback

[1302] The server records the user's interactions with the customer avatar as a log, which allows for later analysis and feedback. After the training is completed, the server analyzes the log and provides feedback to the user, including specific advice and evaluations to improve the user's response skills.

[1303] Specific examples

[1304] For example, consider a situation where a new product is being introduced in a physical store environment.

[1305] 1. The user persona is set as "a woman in her 20s, with a high desire to purchase and a keen sense of fashion."

[1306] 2. The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior.

[1307] 3. The user starts the simulation in the situation of "explaining a new product."

[1308] 4. When the user says, "Tell me about this new dress," the server analyzes the speech and generates a real-time response, such as, "This dress has a design that incorporates the latest trends..."

[1309] Prompt Sentence Examples

[1310] User: "Tell me about this new dress."

[1311] Example prompt for a generative AI model (GPT-4): "Describe a new dress for a fashion-conscious female customer in her 20s with a high purchasing intent."

[1312] Generated AI response: "This dress is designed with the latest trends in mind, making it perfect for spring and summer. It's made from premium materials and is incredibly comfortable. Plus, this dress..."

[1313] In this way, users can receive training in a variety of situations through the virtual reality system, improving their actual customer service skills.

[1314] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1315] Step 1:

[1316] A user logs into a customer service training application. The input is the user's login information (e.g., username and password), which the server verifies and authenticates the user. The output is the main interface screen after the user has logged in.

[1317] Step 2:

[1318] The user accesses an interface to input persona information. The input is persona information such as age, gender, hobbies, and characteristics, which the server receives and stores in a database. The output is a confirmation message that the persona information has been saved.

[1319] Step 3:

[1320] The server calls the generative model based on the stored persona information. The input is the persona information (e.g., "female in her 30s, frequent complainer, knowledgeable about brands"), and generates a customer avatar using a generative AI model (e.g., OpenAI GPT-4). The output is the data of the generated customer avatar.

[1321] Step 4:

[1322] The user puts on the virtual reality device and enters the training simulation environment. The input is a start-up signal for the virtual reality device, and the terminal loads the simulation environment. The output is a confirmation screen that the user has entered the virtual reality environment.

[1323] Step 5:

[1324] The user selects a specific situation through the situation setting interface. The input is the situation information (e.g., "Handling a complaint," "Explaining a new product," "Taking an order"), and the server loads the associated scripts and events. The output is a message indicating that the selected situation is ready.

[1325] Step 6:

[1326] When a user speaks to the customer avatar, the virtual reality device collects the voice data and sends it to the server in real time. The input is voice data, and the server converts it into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The output is the voice-to-text data.

[1327] Step 7:

[1328] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The input is the user's textual utterance, and a natural-sounding response is generated through the generative AI model (e.g., GPT-4). The output is the generated response text.

[1329] Step 8:

[1330] The server sends the generated response data to the virtual reality device, where it is displayed as voice and animation of the customer avatar. The input is the response text, which the device converts into voice data using a speech synthesis engine and synchronizes with the avatar's movements. The output is the avatar's response animation.

[1331] Step 9:

[1332] The server logs the interactions between the user and the customer avatar and stores the data for later analysis. The input is the conversation log between the user and the avatar, and the output is the log data stored in the database.

[1333] Step 10:

[1334] After the training is completed, the server analyzes the logs and generates feedback. The input is the saved log data, and the data analysis algorithm evaluates the user's response skills. The output is a feedback report provided to the user.

[1335] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1336] MODE FOR CARRYING OUT THE INVENTION

[1337] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The system is implemented as follows.

[1338] Persona Settings

[1339] 1. A user logs in to the system and accesses the persona configuration screen.

[1340] 2. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[1341] 3. The server receives the persona information and stores it in a database.

[1342] Customer avatar generation

[1343] 1. The server calls the generative AI model using the saved persona information.

[1344] 2. The generation AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[1345] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[1346] Setting the situation

[1347] 1. The user accesses the situation setting screen and selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[1348] 2. The user presses the "Set Situation" button.

[1349] 3. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[1350] Simulation and Real-Time Response

[1351] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[1352] 2. The user speaks to the customer avatar.

[1353] 3. The device collects the user's voice, facial expressions, and gestures using the camera and microphone, and transmits them to the server in real time.

[1354] Emotion Recognition and Response Generation

[1355] 1. The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine.

[1356] 2. Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[1357] 3. The server sends the generated response data to the terminal.

[1358] 4. The device displays the response as audio and animation of the customer avatar.

[1359] Interactive training experience and feedback

[1360] 1. Users continuously interact with the customer avatar and experience various situations.

[1361] 2. The server records the user's interactions with the customer avatar and the emotion recognition logs.

[1362] 3. After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data.

[1363] Specific examples

[1364] For example, let's take the situation of handling a complaint at a restaurant.

[1365] 1. The user persona is set as "a man in his 40s with extensive experience in the restaurant industry who is very prone to dissatisfaction."

[1366] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[1367] 3. The user starts the simulation in the situation of "handling a complaint."

[1368] 4. When the user says, "This dish is cold," and their face turns grim, the device sends the voice and facial expression to the server.

[1369] 5. The server analyzes the voice and facial expression data and recognizes "dissatisfaction" using an emotion engine.

[1370] 6. The server uses a generation AI to generate a response such as, "Sorry. We will bring you a new dish right away," and sends it to the device.

[1371] 7. The terminal displays a customer avatar responding with voice and animation.

[1372] 8. The user continues to ask questions about the response, practicing realistic customer interactions using emotion recognition.

[1373] In this way, this system, which combines an emotion engine, can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[1374] The processing flow will be explained below.

[1375] MODE FOR CARRYING OUT THE INVENTION

[1376] This invention is a virtual reality system for customer service training that combines an emotion engine that recognizes the user's emotions. The specific processing flow is shown below.

[1377] Processing Steps

[1378] Step 1:

[1379] A user logs in to the system and accesses the persona configuration screen.

[1380] Step 2:

[1381] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[1382] Step 3:

[1383] The server receives the persona information and stores it in a database.

[1384] Step 4:

[1385] The server calls the generative AI model based on the stored persona information.

[1386] Step 5:

[1387] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[1388] Step 6:

[1389] The server prepares the generated customer avatar data for the virtual reality environment.

[1390] Step 7:

[1391] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[1392] Step 8:

[1393] The user presses the "Set Situation" button.

[1394] Step 9:

[1395] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[1396] Step 10:

[1397] The user puts on the VR device (terminal) and enters the simulation environment.

[1398] Step 11:

[1399] The user speaks to the customer avatar.

[1400] Step 12:

[1401] The device collects the user's voice, facial expressions, and gestures using a camera and microphone, and transmits the data to a server in real time.

[1402] Step 13:

[1403] The server analyzes the received voice, facial expressions, and gesture data, and recognizes the user's emotions using an emotion engine.

[1404] Step 14:

[1405] Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[1406] Step 15:

[1407] The server transmits the generated response data to the terminal.

[1408] Step 16:

[1409] The terminal displays the response as audio and animation of a customer avatar.

[1410] Step 17:

[1411] Users continuously interact with the customer avatar and experience various situations.

[1412] Step 18:

[1413] The server logs the user's interactions with the customer avatar and the emotion recognition.

[1414] Step 19:

[1415] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[1416] Specific examples

[1417] For example, let's take the situation of handling a complaint at a restaurant.

[1418] Step 1:

[1419] A user logs in to the system and accesses the persona configuration screen.

[1420] Step 2:

[1421] The user enters the persona information as "male in his 40s, has extensive experience in the restaurant industry, and is very prone to dissatisfaction" and clicks the "Submit" button.

[1422] Step 3:

[1423] The server receives the persona information and stores it in a database.

[1424] Step 4:

[1425] The server calls the generative AI model based on the stored persona information.

[1426] Step 5:

[1427] The generation AI generates a customer avatar with specific clothing and behavior based on the persona information entered.

[1428] Step 6:

[1429] The server prepares the generated customer avatar data for the virtual reality environment.

[1430] Step 7:

[1431] The user accesses the situation setting screen and selects the situation "Handling a complaint."

[1432] Step 8:

[1433] The user presses the "Set Situation" button.

[1434] Step 9:

[1435] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[1436] Step 10:

[1437] The user puts on the VR device (terminal) and enters the simulation environment.

[1438] Step 11:

[1439] The user complains to the customer avatar, saying, "This food is cold."

[1440] Step 12:

[1441] The device collects the user's voice and facial expressions and transmits them to a server in real time.

[1442] Step 13:

[1443] The server analyzes the voice, facial expressions, and gesture data it receives, and uses an emotion engine to recognize "dissatisfaction."

[1444] Step 14:

[1445] Based on the results of emotion recognition, the server uses generative AI to generate a response such as, "We're sorry. We'll bring you a new dish right away."

[1446] Step 15:

[1447] The server transmits the generated response data to the terminal.

[1448] Step 16:

[1449] The device displays a customer avatar responding with voice and animation.

[1450] Step 17:

[1451] The user continues to ask further questions based on the responses, practicing realistic customer interactions using emotion recognition.

[1452] Step 18:

[1453] The server logs the user's interactions with the customer avatar and the emotion recognition.

[1454] Step 19:

[1455] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[1456] Example 2

[1457] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1458] Modern customer service training requires practical and interactive training methods. However, traditional training methods have difficulty recreating real-life customer interactions, limiting the opportunities for trainees to demonstrate their actual skills. Furthermore, the lack of a system that can recognize trainees' emotions and reactions in real time and provide adaptive feedback based on that information makes it difficult to achieve deeper learning.

[1459] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1460] In this invention, the server includes means for collecting and transmitting the voice, facial expressions, and gestures of the trainee to the server, means for analyzing the received voice, facial expressions, and gesture data to recognize emotions, and means for using a generative artificial intelligence to adjust and generate responses of a customer avatar based on the results of the emotion recognition. This makes it possible to recognize the emotions and reactions of the trainee in real time and provide adaptive feedback based on them.

[1461] "Training Recipient" means a person receiving customer service training.

[1462] "Persona information" refers to data such as age, gender, hobbies, and characteristics that are used as basic information when generating a customer avatar.

[1463] "Interface" refers to the screen or operating means used by trainees to enter persona information.

[1464] "Generative AI" is an AI technology for generating the appearance, clothing, and behavior of a customer avatar based on persona information.

[1465] "Virtual reality device" means equipment used by a trainee to interact with a customer avatar in a virtual environment.

[1466] "Server" means a computer system for storing persona information and providing responses of generated customer avatars.

[1467] "Voice, facial expression, and gesture data" refers to information about the voice, facial expression, and body movements of the training subject, and is data used for emotion recognition.

[1468] "Emotion recognition" is a technology that grasps the emotions of trainees from their voices, facial expressions, and gestures.

[1469] A "customer avatar" is a virtual character that is generated based on persona information and interacts with the training subject in a virtual environment.

[1470] "Response" refers to the reply or action that the customer avatar gives in response to an interaction from the training subject.

[1471] "Log" refers to data that records the interactions between the training subject and the customer avatar and the results of emotion recognition.

[1472] "Feedback" refers to information, including evaluations and areas for improvement, that is provided to trainees after the training has been completed.

[1473] "Specific situations" are specific situations or scenes that trainees experience in a virtual environment, such as handling complaints, explaining new products, or accepting orders.

[1474] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The main components of this system include a user terminal, a server, a generative AI model, and a virtual reality device. Specific implementation methods are described below.

[1475] First, the user logs in to the system and accesses the persona setting screen. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. The server receives the submitted persona information and stores it in a database. The server then invokes a generative AI model based on the stored persona information to generate the appearance, clothing, and behavior of the customer avatar. An example of a prompt message for this is as follows:

[1476] "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[1477] The generated customer avatar data is prepared for the virtual reality environment by the server. The user accesses the situation setting screen and selects a specific situation. Examples include "handling a complaint," "explaining a new product," and "accepting an order." When the user presses the "Set Situation" button, the server loads the script or scenario corresponding to the selected situation and completes the preparation.

[1478] Next, the user puts on the VR device (terminal) and enters the simulation environment. When the user speaks to the customer avatar, the terminal collects the user's voice, as well as facial expressions and gestures using a camera and microphone, and sends the collected data to the server in real time. The server analyzes the received voice, facial expressions, and gesture data and recognizes the user's emotions using an emotion engine. For example, if a user says, "This dish is cold," and their face turns grim, the server analyzes the voice and facial expression data and recognizes the emotion as "dissatisfied."

[1479] Based on the emotion recognition results, the server uses a generative AI model to adjust and generate a response for the customer avatar. For example, a response such as "Sorry, we'll bring you a new dish right away" may be generated. The server then sends the generated response data to the device, which then displays the customer avatar's response with voice and animation.

[1480] In this way, the user continuously interacts with the customer avatar and experiences various situations. The server records the interaction between the user and the customer avatar and the emotion recognition log, analyzes the log after the training is completed, and provides the user with feedback including the emotion data.

[1481] For example, in a situation where a customer is handling a complaint at a restaurant, the user sets their persona as "a man in his 40s with extensive restaurant experience who is very prone to becoming dissatisfied." The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior, and the user begins the simulation in the "handling a complaint" situation. When the user says, "This dish is cold," the device sends their voice and facial expression to the server, which analyzes it and generates an appropriate response, which is sent back to the device.

[1482] In this way, the system can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[1483] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1484] Step 1:

[1485] A user logs in to the system and accesses the persona configuration screen. They provide their username and password as input, and the server performs the authentication process. After successful authentication, they are redirected to the persona configuration screen.

[1486] Step 2:

[1487] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. Input data includes age "40", gender "male", hobbies "reading", and characteristics "kind". The server receives the persona information and saves it in the database. Specifically, the server analyzes the persona information sent, generates an SQL statement to insert into the database, connects to the database, and executes the SQL statement.

[1488] Step 3:

[1489] The server calls the generative AI model using the saved persona information. It uses the persona information as input to generate a prompt. This prompt is passed to the generative AI model to generate a customer avatar. As output, appearance information (hair color, clothing style, etc.) and behavior information are obtained. An example prompt is "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[1490] Step 4:

[1491] The server prepares the generated customer avatar data for the virtual reality environment. It uses the generated appearance and behavior information as input data, converts it into a format suitable for the VR environment, and organizes the necessary files. As an output, it obtains customer avatar data for the VR environment.

[1492] Step 5:

[1493] The user accesses the situation setting screen and selects a specific situation from a drop-down list. The user selects a situation (e.g., "Complaint handling") as input data and presses the "Set Situation" button. The server retrieves the script corresponding to the selected scenario from the database and loads it into memory. The scenario data is prepared as output.

[1494] Step 6:

[1495] The user puts on the VR device (terminal) and enters the simulation environment. The input is to start up the VR device and connect it to the system. The terminal starts up the VR environment and places the user in the simulation. This allows the user to start training in the virtual environment.

[1496] Step 7:

[1497] The user speaks to the customer avatar. For example, they may say, "This dish is cold." The device collects the user's voice and simultaneously records their facial expressions and gestures using a camera and microphone. This data is sent to the server in real time. The collected voice, facial expressions, and gesture data is sent to the server as output.

[1498] Step 8:

[1499] The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine. Voice data, facial expression data, and gesture data are used as input, and an emotion recognition algorithm is applied. The output is an emotion result such as "dissatisfied."

[1500] Step 9:

[1501] The server uses generative AI to adjust and generate the customer avatar's response based on the emotion recognition results. The emotion recognition results and generative AI model are used as input. When the generative AI model is sent a prompt such as "How should you respond to an angry customer?", the output generated is a response such as "We're sorry. We'll bring you a new dish right away."

[1502] Step 10:

[1503] The server sends the generated response data to the terminal, uses the generated response data as input and sends it to the terminal, and when the terminal receives the response data, displays the response as voice and animation of the customer avatar.

[1504] Step 11:

[1505] Users continually interact with customer avatars and experience various situations, allowing them to simulate real-life customer service scenarios and improve their response skills.

[1506] Step 12:

[1507] The server records the interaction between the user and the customer avatar and the emotion recognition log. Each interaction and the emotion recognition results are used as input and stored in a database. After the training is completed, the log is analyzed and feedback, including the emotion data, is provided to the user.

[1508] Step 13:

[1509] After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data. Using the recorded log data as input, the server generates a report that includes the user's behavioral trends and areas for improvement. Detailed feedback is provided to the user as output.

[1510] (Application example 2)

[1511] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1512] Traditional customer service training systems are limited to standard scenarios and situations, and are unable to respond to trainees' emotions and reactions in real time. This limits the effectiveness of training and causes problems such as a lack of flexibility and responsiveness required for actual customer interactions. It is also difficult to track trainees' emotions and provide appropriate feedback based on them.

[1513] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1514] In this invention, the server includes means for collecting voice data and facial expression and gesture data captured by a camera of the trainee and transmitting them to the server in real time, means for analyzing the collected data and recognizing the trainee's emotions with an emotion engine, and means for adjusting and generating responses of a customer avatar using generative artificial intelligence based on the results of emotion recognition. This makes it possible to generate responses in real time based on the trainee's emotions and improve customer service skills in brick-and-mortar stores in a manner closer to reality.

[1515] "Training Participant" means a person who participates in a customer service training program and seeks to improve their skills using a virtual reality system.

[1516] "Persona information" refers to personal information such as age, gender, hobbies, and characteristics entered by the training participant, and a customer avatar is generated based on this information.

[1517] "Generative AI" refers to AI that has the ability to generate new data based on input data, and is used to generate the appearance and behavior of customer avatars.

[1518] A "virtual reality device" is a device that displays a virtual reality space and allows users to interact with that space. Examples include head-mounted displays and VR goggles.

[1519] A "server" is a computer that processes and stores data over a network, and is used here for generating responses for customer avatars, recording logs, and analyzing emotions.

[1520] A "customer avatar" is a virtual person generated within a virtual reality system based on persona information, and is responsible for interacting with the training recipient.

[1521] An "emotion engine" is software that analyzes voice data, facial expression data, and gesture data to recognize the emotional state of the subject.

[1522] An "interface" refers to the operation screen and input device that trainees use to interact with the system.

[1523] A "log" is a record of the system's operation history and data, and in this case, it records the interaction between the training participant and the customer avatar.

[1524] "Feedback" refers to the evaluation and areas for improvement provided to the training participants, and is provided based on information analyzed from the logs.

[1525] A "scenario" is a sequence of events or dialogue used in a specific situation or setting, and is used to set the scene for training.

[1526] This invention is a system that combines a virtual reality system for customer service training with an emotion engine that recognizes the emotions of users. Specific embodiments will be described below.

[1527] Structure and program overview

[1528] This system allows trainees to input customer persona information and interact with a customer avatar generated based on that information. The system uses a smartphone or a head-mounted display (VR device) as hardware, and includes a server for generating real-time responses, an emotion engine for recognizing emotions, and generative artificial intelligence.

[1529] Hardware and Software Used

[1530] Smartphone or head-mounted display (VR device): A device that allows users to interact in a virtual space.

[1531] Server: A computer that processes data, analyzes it, and generates responses.

[1532] Emotion engine: Software that analyzes voice data, facial expression data, and gesture data to recognize the user's emotions.

[1533] Generative artificial intelligence: Software that generates customer avatars based on persona information and generates responses in real time based on user sentiment.

[1534] Data processing and calculation

[1535] 1. Login and Persona Settings

[1536] The user logs in to the smartphone app and enters their age, gender, hobbies, and characteristics on the persona setting screen. This persona information is sent to the server and stored in the database.

[1537] 2. Customer avatar generation

[1538] The server passes the persona information to a generation AI to generate the appearance, clothing, and behavior of the customer avatar, and the generated data is then appropriately prepared for the VR environment.

[1539] 3. Setting the situation

[1540] The user selects a training situation on the app screen, and the server loads the corresponding script and scenario.

[1541] 4. Simulation and real-time response

[1542] The user puts on the VR device and begins interacting with a customer avatar in a virtual store.

[1543] The smartphone or VR device collects the user's voice data, facial expression data, and gesture data and transmits them to a server in real time.

[1544] The server's emotion engine analyzes the data and recognizes the user's emotions.

[1545] Based on the results of emotion recognition, the generative AI model generates a response for the customer avatar, which the server then sends to the device, allowing the avatar to respond in real time.

[1546] Specific examples

[1547] Customer Service Training Scenarios

[1548] 1. The user enters "Female in her 30s, gentle personality, retail store worker, positive" on the "Persona Settings" screen and submits.

[1549] 2. The server generates a "gentle customer avatar, a woman in her 30s" based on the persona information.

[1550] 3. The user selects and sets the situation "Explanation of a new product."

[1551] 4. The user wearing the VR device speaks to the avatar, asking, "What are the features of this new product?" The camera captures the user's facial expressions and sends them to the server along with the audio data.

[1552] 5. The server analyzes the user's facial expressions and uses an emotion engine to recognize "interest." The generative AI model generates a response, such as "This new product is compact, lightweight, and made from eco-friendly materials." The server then sends the response to the device, where the customer avatar responds with voice and animation.

[1553] Prompt Sentence Examples

[1554] Persona information: Female in her 30s, gentle personality, retail store worker, positive

[1555] Situation: New product description

[1556] User Sentiment: Interest

[1557] What it produces: This new product is compact, lightweight and made from eco-friendly materials.

[1558] This allows users to improve their skills in dealing with real customers.

[1559] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1560] Step 1:

[1561] A user logs into the system and enters persona information

[1562] Input: The user enters persona information such as age, gender, hobbies, and characteristics.

[1563] How it works: The user enters persona information (e.g., female in her 30s, gentle personality, retail worker, positive) through the interface and clicks the "Submit" button.

[1564] Output: Persona information is sent to the system and stored on the server.

[1565] Step 2:

[1566] The server generates a customer avatar based on the persona information

[1567] Input: Saved persona information.

[1568] How it works: The server retrieves persona information and uses generative artificial intelligence (generative AI model) to generate the appearance, clothing, and behavior of a customer avatar.

[1569] Output: The generated customer avatar is prepared for the virtual reality environment.

[1570] Step 3:

[1571] Users set training situations

[1572] Input: Select a training situation (e.g., explaining a new product).

[1573] How it works: The user selects a specific situation on the situation setting screen and presses the "Set Situation" button.

[1574] Output: The scripts and scenarios corresponding to the selected situations are loaded by the server.

[1575] Step 4:

[1576] The user puts on the VR device and starts the simulation.

[1577] Input: Customer avatar and situation script generated based on persona information.

[1578] How it works: A user puts on a VR device and begins interacting with a customer avatar in a virtual store.

[1579] Output: The interaction is initiated by the user speaking to the customer avatar.

[1580] Step 5:

[1581] The device collects the user's voice, facial expressions, and gestures and sends them to the server.

[1582] Input: User's voice data, facial expression data captured by the camera, and gesture data.

[1583] Actions: The device's sensors capture the user's voice, facial expressions, and gestures and transmit them to the server in real time.

[1584] Output: The user's voice, facial expression, and gesture data are sent to the server.

[1585] Step 6:

[1586] The server recognizes the user's emotions using an emotion engine

[1587] Input: Collected voice data, facial expression data, and gesture data.

[1588] How it works: The server's emotion engine analyzes the collected data and recognizes the user's emotions (e.g., interest, frustration).

[1589] Output: The analyzed emotion information is passed to the generative AI model.

[1590] Step 7:

[1591] The server generates a response for the customer avatar based on the emotion recognition results.

[1592] Input: Emotion recognition results, generative artificial intelligence model.

[1593] How it works: The server uses the generative AI model to generate an appropriate customer avatar response based on the recognized emotion (e.g., "This new product is compact, lightweight, and made from eco-friendly materials.").

[1594] Output: The generated response data is sent to the terminal.

[1595] Step 8:

[1596] The device displays the response to the customer avatar

[1597] Input: The response data of the generated customer avatar.

[1598] How it works: The device displays and plays the generated response data with animation and audio, and the customer avatar responds in real time.

[1599] Output: The user confirms the generated response and continues the interaction.

[1600] Step 9:

[1601] The server logs the interaction and provides feedback to the trainee.

[1602] Input: All interaction data for the user and customer avatar.

[1603] How it works: The server logs interactions and later analyzes them to provide feedback to the user.

[1604] Output: The user receives feedback including the results of the training and areas for improvement.

[1605] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1606] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1607] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1608] [Fourth embodiment]

[1609] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1610] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1611] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1612] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1613] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1614] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1615] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1616] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1617] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1618] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1619] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1620] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1621] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1622] MODE FOR CARRYING OUT THE INVENTION

[1623] The virtual reality system for customer service training of the present invention is implemented as follows.

[1624] Customer persona creation

[1625] 1. The user logs in to the system and accesses the persona setting screen.

[1626] 2. The user enters the following persona information into the form:

[1627] age

[1628] sex

[1629] hobby

[1630] Characteristics (e.g., many complaints, high purchasing intent, etc.)

[1631] 3. After entering the information, the user clicks the "Submit" button.

[1632] 4. The server receives the persona information and stores it in a database.

[1633] Customer avatar generation

[1634] 1. The server calls the generative AI model using the saved persona information.

[1635] 2. Generative AI generates the appearance, clothing, and behavior of a customer avatar based on the input persona information.

[1636] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[1637] Setting the situation

[1638] 1. The user accesses the situation setting screen.

[1639] 2. The user selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[1640] 3. The user presses the "Set Situation" button.

[1641] 4. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[1642] Simulation and Real-Time Response

[1643] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[1644] 2. When the user speaks to the customer avatar,

[1645] The device collects the audio and transmits it to a server in real time.

[1646] 3. The server receives the voice data and converts it into text using a speech analysis engine.

[1647] 4. The server uses generative AI to generate a response for the customer avatar based on the analyzed text.

[1648] 5. The server sends the generated response data to the terminal.

[1649] 6. The terminal displays the response as a voice and animation of the customer avatar.

[1650] Providing an interactive training experience

[1651] 1. Users continuously interact with the customer avatar and experience various situations.

[1652] 2. The server logs the user's interactions with the customer avatar for later analysis.

[1653] 3. After the training is completed, the server generates feedback from the analyzed logs and provides it to the user.

[1654] Specific examples

[1655] For example, let's take the situation of handling a complaint at a fashion retail store.

[1656] 1. The user persona is set as "a woman in her 30s who makes many complaints and is knowledgeable about brands."

[1657] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[1658] 3. The user starts the simulation in the situation of "handling a complaint."

[1659] 4. When the user says, "There is a problem with this product," the device sends the speech to the server.

[1660] 5. The server analyzes the voice and uses generative AI to generate a real-time response such as "Sorry, can you tell me more?", which is displayed on the device.

[1661] 6. Users can then ask further questions about the responses to practice realistic customer interactions.

[1662] In this way, users can receive training for a variety of situations through the virtual reality system, improving their actual customer service skills.

[1663] The processing flow will be explained below.

[1664] Step 1:

[1665] A user logs in to the system and accesses the persona configuration screen.

[1666] Step 2:

[1667] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[1668] Step 3:

[1669] The server receives the persona information and stores it in a database.

[1670] Step 4:

[1671] The server calls the generative AI model based on the stored persona information.

[1672] Step 5:

[1673] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[1674] Step 6:

[1675] The server prepares the generated customer avatar data for the virtual reality environment.

[1676] Step 7:

[1677] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[1678] Step 8:

[1679] The user presses the "Set Situation" button.

[1680] Step 9:

[1681] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[1682] Step 10:

[1683] The user puts on the VR device (terminal) and enters the simulation environment.

[1684] Step 11:

[1685] The user speaks to the customer avatar.

[1686] Step 12:

[1687] The device collects the user's voice and transmits it to the server in real time.

[1688] Step 13:

[1689] The server analyzes the received voice data and converts it into text.

[1690] Step 14:

[1691] The server uses a generative AI to generate a response for the customer avatar based on the analyzed text.

[1692] Step 15:

[1693] The server transmits the generated response data to the terminal.

[1694] Step 16:

[1695] The terminal displays the response as audio and animation of a customer avatar.

[1696] Step 17:

[1697] The user and customer avatar continuously interact and experience various situations.

[1698] Step 18:

[1699] The server logs the interactions between the user and the customer avatar.

[1700] Step 19:

[1701] The server analyzes the log after the training is completed and provides feedback to the user.

[1702] Example 1

[1703] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1704] Traditional customer service training has limited resources and flexibility to effectively improve real-world customer interaction skills. For example, it is difficult to perfectly recreate different customer personas and diverse situations in a real-world training environment, resulting in a lack of practical training. Additionally, feedback on customer interactions is generally not immediate. This creates the problem of taking a long time to acquire appropriate customer interaction skills.

[1705] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1706] In this invention, the server includes: means for providing an interface for a trainee to input customer persona information; means for using a generative AI model to generate a customer avatar based on the persona information; means for the trainee to use a virtual reality device to interact with the customer avatar; means for using a server to generate responses using a voice analysis engine and the generative AI model to generate responses for the customer avatar in real time and provide the responses to the virtual reality device; and means for recording a log of the interactions and providing feedback to the trainee. This effectively recreates different customer personas and a variety of situations, allowing the trainee to acquire practical customer service skills in real time and receive immediate feedback.

[1707] The "Virtual Reality System for Customer Service Training" is a system that allows trainees to train customer service skills using a virtual reality environment.

[1708] "Training Recipient" means an individual receiving training to improve their customer service skills.

[1709] "Persona information" refers to attribute information such as the age, gender, hobbies, and characteristics of a virtual customer.

[1710] "Interface" refers to a screen or input device that allows a user to input and operate persona information.

[1711] "Generative AI model" refers to a model that uses artificial intelligence technology to generate customer avatars based on input persona information.

[1712] "Virtual Reality Device" means a device (e.g., a VR headset) used by a trainee to interact with a customer avatar in a virtual reality environment.

[1713] "Server" refers to a computer system used to manage the entire system, process data, call generative AI models, record logs, and provide feedback.

[1714] "Speech analysis engine" refers to technology or software for converting voice data into text.

[1715] "Customer avatar" refers to a 3D model of a virtual customer generated based on persona information.

[1716] "Situation" refers to a specific scene or situation in customer service training.

[1717] "Response" refers to the reply or reaction that the customer avatar gives to the training recipient's input.

[1718] "Log" refers to a record of interactions between a user and a customer avatar during training.

[1719] "Feedback" refers to evaluation and advice regarding the user's performance provided after the training is completed.

[1720] The virtual reality system for customer service training of this invention is implemented as follows: A user first logs in to the system and accesses an interface for inputting persona information. Here, the user inputs and submits persona information such as age, gender, hobbies, and characteristics. The server receives this information and stores it in a database.

[1721] The server then invokes a generative AI model based on the stored persona information to generate a customer avatar. This generative AI model uses OpenAI's GPT-3, for example. The generated avatar's appearance, clothing, and behavior are customized based on the persona information. The server then prepares this avatar data for the virtual reality environment.

[1722] The user then accesses the situation setting screen and selects a specific situation (e.g., handling a complaint, explaining a new product, etc.). The server loads the script or scenario corresponding to the selected situation and completes preparations.

[1723] The user then puts on a VR device (e.g., Oculus Rift) and enters the simulation environment. When the user speaks to the customer avatar, the device (VR device) collects the speech and transmits it to the server in real time. The server receives the speech and converts it into text using a speech analysis engine (e.g., Google Speech-to-Text).

[1724] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The generated response is sent to the device in real time and displayed as the voice and animation of the customer avatar. The user can continue to interact with this response and experience various situations.

[1725] The server records the interactions between the user and the customer avatar as logs, and after the training is completed, the server generates and provides feedback to the user from the analyzed logs, which allows the user to effectively improve their customer service skills.

[1726] Specific examples

[1727] For example, consider a situation where a customer is handling a complaint at a fashion retailer. The user sets the persona as "a woman in her 30s who makes many complaints and is knowledgeable about brands." The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior. The user then starts the simulation in the "complaint handling" situation.

[1728] When a user says, "There's a problem with this product," the device sends the speech to a server. The server analyzes the speech and uses a generative AI model to generate a real-time response, such as "I'm sorry. Could you please tell me more?", which is displayed on the device. The user can then ask further questions about the response, practicing realistic customer interactions.

[1729] Prompt Sentence Examples

[1730] "Generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand."

[1731] "Please load a script for a complaint handling situation."

[1732] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1733] Step 1:

[1734] A user logs into the system:

[1735] Input: Username and Password

[1736] What happens: A user enters their username and password on the login screen and clicks the "Login" button.

[1737] Data processing: The server receives the entered authentication information and checks it against a database.

[1738] Output: If authentication is successful, display the user's dashboard.

[1739] Step 2:

[1740] User accesses the Persona settings screen:

[1741] Input: Persona information (age, gender, hobbies, characteristics)

[1742] Action: The user selects the "Persona Settings" menu from the dashboard and proceeds to the settings screen.

[1743] Output: Displays an interface for entering persona information.

[1744] Step 3:

[1745] User enters persona information and submits:

[1746] Input: Values ​​corresponding to age, gender, hobbies, and characteristics

[1747] How it works: The user enters their persona information and clicks the "Submit" button.

[1748] Data processing: The server receives the entered data and performs validation (checking the accuracy and completeness of the data).

[1749] Output: Save the persona information that passed validation to the database.

[1750] Step 4:

[1751] The server invokes the generative AI model to generate the customer avatar:

[1752] Input: Saved persona information

[1753] How it works: The server calls the generative AI model based on the persona information, generates a prompt (e.g., "Please generate a customer avatar for a persona of a woman in her 30s who makes many complaints and is knowledgeable about the brand"), and sends it.

[1754] Data Computation: A generative AI model generates the appearance, clothing, and behavior of a customer avatar based on the prompt text.

[1755] Output: The generated customer avatar data is sent back to the server.

[1756] Step 5:

[1757] The server prepares the customer avatar data for the VR environment:

[1758] Input: Generated avatar data

[1759] How it works: The server converts the received avatar data into a format suitable for the virtual reality environment.

[1760] Output: Avatar data that can be used in a virtual reality environment

[1761] Step 6:

[1762] User accesses the situation settings screen and selects a situation:

[1763] Input: Specific situation (e.g., handling a complaint, explaining a new product, etc.)

[1764] How it works: The user accesses the situation setup screen, selects a situation from the drop-down list, and presses the "Set Situation" button.

[1765] Output: Loads the scripts and scenarios corresponding to the selected situations into the server.

[1766] Step 7:

[1767] The user puts on the VR device and enters the simulated environment:

[1768] Input: Set situation

[1769] How it works: The user puts on a compatible VR device (e.g., Oculus Rift) and enters the simulated environment.

[1770] Output: The state in which the user begins interacting with the customer avatar in the simulated environment.

[1771] Step 8:

[1772] The user speaks to the customer avatar and audio is collected:

[1773] Input: User's voice

[1774] How it works: When a user speaks to a customer avatar, the terminal (VR device) collects the voice and sends it to the server in real time.

[1775] Output: The collected audio data is sent to the server.

[1776] Step 9:

[1777] The server analyzes the audio data and converts it to text:

[1778] Input: Collected audio data

[1779] How it works: The server receives the audio data and converts it to text using a speech analysis engine (e.g., Google Speech-to-Text).

[1780] Data calculation: Converting voice data into text

[1781] Output: Parsed text data

[1782] Step 10:

[1783] The server uses the generative AI model to generate a response for the customer avatar:

[1784] Input: Parsed text data

[1785] How it works: The server uses a generative AI model to generate a response for the customer avatar based on the parsed text.

[1786] Data Calculation: Generating Response Text

[1787] Output: Generated response data

[1788] Step 11:

[1789] The server sends the response data to the device:

[1790] Input: Generated response data

[1791] Operation: The server sends the generated response data to the terminal.

[1792] Output: Response data sent to the terminal

[1793] Step 12:

[1794] The device displays the response as audio and animation of a customer avatar:

[1795] Input: Response data sent

[1796] Action: The device displays the response as audio and animation of the customer avatar, providing feedback to the user.

[1797] Output: Customer avatar response displayed to the user

[1798] Step 13:

[1799] The server logs the interaction and generates feedback:

[1800] Input: User and customer avatar interaction data

[1801] How it works: The server logs interactions and generates feedback from the logs that are analyzed after the training is completed.

[1802] Output: Feedback provided to the user

[1803] (Application example 1)

[1804] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1805] Modern customer service training lacks practical training tools to improve customer service skills. It's also difficult to recreate realistic situations and allow trainees to gain diverse experience in a manner that closely resembles the real world. Traditional training methods also make it difficult to provide real-time feedback or use scripts tailored to specific situations. As a result, the results of training are often not fully utilized in the workplace.

[1806] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1807] In this invention, the server includes means for providing an interface for the trainee to input customer persona information, means for using a generative model to generate a customer avatar based on the persona information, means for using a virtual reality device for the trainee to interact with the customer avatar, means for using a data processing device to generate responses of the customer avatar in real time and provide them to the virtual reality device, means for recording a log of the interaction and providing feedback to the trainee, and means for loading scripts and events corresponding to specific situations such as product explanations and order taking in a physical store environment, thereby allowing the trainee to experience realistic customer interaction simulations and improve their response skills and ability to respond.

[1808] An "interface" is a point of contact between a device or software and a user for inputting or outputting information.

[1809] A "generative model" is an algorithm or program that automatically generates customer avatars based on persona information.

[1810] A "virtual reality device" is a device used to interact with a virtual reality environment, including head-mounted displays and VR goggles.

[1811] A "data processing device" is a hardware or software system that analyzes and processes input voice or text data and generates and transmits an appropriate response.

[1812] A "script" is a scenario or script for responding to a specific situation or event.

[1813] "Interaction" refers to the interaction between the trainee and the customer avatar, including conversation and behavioral exchanges.

[1814] "Persona information" refers to information such as age, gender, hobbies, and characteristics that the trainee enters to define the characteristics of a virtual customer.

[1815] A "customer avatar" is a virtual customer character generated based on persona information.

[1816] A "log" is a record of the interaction between the trainee and the customer avatar, and is data that is later used for analysis and feedback.

[1817] "Feedback" refers to the results, evaluations, and advice analyzed based on interaction logs, and is provided to trainees.

[1818] The present invention relates to a customer service training system that utilizes virtual reality. Hereinafter, an embodiment of this system will be described in detail.

[1819] System Configuration

[1820] User Login and Persona Configuration

[1821] The server provides an interface for users to log in and enter persona information (age, gender, hobbies, characteristics, etc.). This interface can be implemented as a web application or a mobile application, for example. The persona information is sent to the server and stored in a database (for example, MySQL or PostgreSQL).

[1822] Customer avatar generation

[1823] The server uses the stored persona information to invoke a generation algorithm that generates a customer avatar based on this information, using a generative AI model such as OpenAI's GPT-4, and then converts the generated customer avatar into a format that can be used in a virtual reality environment.

[1824] Using a Virtual Reality Device

[1825] The user enters the simulated environment wearing a virtual reality device (e.g., a head-mounted display), such as the Oculus Rift or HTC Vive.

[1826] Setting up situations and generating responses

[1827] The user selects a specific situation (e.g., handling a complaint, explaining a new product, accepting an order, etc.) through the situation setting interface, and the server loads the scripts and events corresponding to this situation.

[1828] When a user speaks to the customer avatar, the virtual reality device collects the speech data and sends it to a server. The server converts the speech into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The converted text is analyzed by a generative AI model (e.g., GPT-4) to generate an appropriate response. This response is sent to the user's virtual reality device in real time.

[1829] Interaction recording and feedback

[1830] The server records the user's interactions with the customer avatar as a log, which allows for later analysis and feedback. After the training is completed, the server analyzes the log and provides feedback to the user, including specific advice and evaluations to improve the user's response skills.

[1831] Specific examples

[1832] For example, consider a situation where a new product is being introduced in a physical store environment.

[1833] 1. The user persona is set as "a woman in her 20s, with a high desire to purchase and a keen sense of fashion."

[1834] 2. The server uses a generative AI model based on this persona information to generate a customer avatar with specific clothing and behavior.

[1835] 3. The user starts the simulation in the situation of "explaining a new product."

[1836] 4. When the user says, "Tell me about this new dress," the server analyzes the speech and generates a real-time response, such as, "This dress has a design that incorporates the latest trends..."

[1837] Prompt Sentence Examples

[1838] User: "Tell me about this new dress."

[1839] Example prompt for a generative AI model (GPT-4): "Describe a new dress for a fashion-conscious female customer in her 20s with a high purchasing intent."

[1840] Generated AI response: "This dress is designed with the latest trends in mind, making it perfect for spring and summer. It's made from premium materials and is incredibly comfortable. Plus, this dress..."

[1841] In this way, users can receive training in a variety of situations through the virtual reality system, improving their actual customer service skills.

[1842] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1843] Step 1:

[1844] A user logs into a customer service training application. The input is the user's login information (e.g., username and password), which the server verifies and authenticates the user. The output is the main interface screen after the user has logged in.

[1845] Step 2:

[1846] The user accesses an interface to input persona information. The input is persona information such as age, gender, hobbies, and characteristics, which the server receives and stores in a database. The output is a confirmation message that the persona information has been saved.

[1847] Step 3:

[1848] The server calls the generative model based on the stored persona information. The input is the persona information (e.g., "female in her 30s, frequent complainer, knowledgeable about brands"), and generates a customer avatar using a generative AI model (e.g., OpenAI GPT-4). The output is the data of the generated customer avatar.

[1849] Step 4:

[1850] The user puts on the virtual reality device and enters the training simulation environment. The input is a start-up signal for the virtual reality device, and the terminal loads the simulation environment. The output is a confirmation screen that the user has entered the virtual reality environment.

[1851] Step 5:

[1852] The user selects a specific situation through the situation setting interface. The input is the situation information (e.g., "Handling a complaint," "Explaining a new product," "Taking an order"), and the server loads the associated scripts and events. The output is a message indicating that the selected situation is ready.

[1853] Step 6:

[1854] When a user speaks to the customer avatar, the virtual reality device collects the voice data and sends it to the server in real time. The input is voice data, and the server converts it into text using a speech analysis engine (e.g., Google Cloud Speech-to-Text). The output is the voice-to-text data.

[1855] Step 7:

[1856] The server uses a generative AI model to generate a response for the customer avatar based on the analyzed text. The input is the user's textual utterance, and a natural-sounding response is generated through the generative AI model (e.g., GPT-4). The output is the generated response text.

[1857] Step 8:

[1858] The server sends the generated response data to the virtual reality device, where it is displayed as voice and animation of the customer avatar. The input is the response text, which the device converts into voice data using a speech synthesis engine and synchronizes with the avatar's movements. The output is the avatar's response animation.

[1859] Step 9:

[1860] The server logs the interactions between the user and the customer avatar and stores the data for later analysis. The input is the conversation log between the user and the avatar, and the output is the log data stored in the database.

[1861] Step 10:

[1862] After the training is completed, the server analyzes the logs and generates feedback. The input is the saved log data, and the data analysis algorithm evaluates the user's response skills. The output is a feedback report provided to the user.

[1863] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1864] MODE FOR CARRYING OUT THE INVENTION

[1865] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The system is implemented as follows.

[1866] Persona Settings

[1867] 1. A user logs in to the system and accesses the persona configuration screen.

[1868] 2. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[1869] 3. The server receives the persona information and stores it in a database.

[1870] Customer avatar generation

[1871] 1. The server calls the generative AI model using the saved persona information.

[1872] 2. The generation AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[1873] 3. The server prepares the generated customer avatar data for the virtual reality environment.

[1874] Setting the situation

[1875] 1. The user accesses the situation setting screen and selects a specific situation from a drop-down list (e.g., handling a complaint, explaining a new product, accepting an order, etc.).

[1876] 2. The user presses the "Set Situation" button.

[1877] 3. The server loads the script or scenario corresponding to the selected situation and is ready to go.

[1878] Simulation and Real-Time Response

[1879] 1. The user puts on the VR device (terminal) and enters the simulation environment.

[1880] 2. The user speaks to the customer avatar.

[1881] 3. The device collects the user's voice, facial expressions, and gestures using the camera and microphone, and transmits them to the server in real time.

[1882] Emotion Recognition and Response Generation

[1883] 1. The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine.

[1884] 2. Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[1885] 3. The server sends the generated response data to the terminal.

[1886] 4. The device displays the response as audio and animation of the customer avatar.

[1887] Interactive training experience and feedback

[1888] 1. Users continuously interact with the customer avatar and experience various situations.

[1889] 2. The server records the user's interactions with the customer avatar and the emotion recognition logs.

[1890] 3. After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data.

[1891] Specific examples

[1892] For example, let's take the situation of handling a complaint at a restaurant.

[1893] 1. The user persona is set as "a man in his 40s with extensive experience in the restaurant industry who is very prone to dissatisfaction."

[1894] 2. The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior.

[1895] 3. The user starts the simulation in the situation of "handling a complaint."

[1896] 4. When the user says, "This dish is cold," and their face turns grim, the device sends the voice and facial expression to the server.

[1897] 5. The server analyzes the voice and facial expression data and recognizes "dissatisfaction" using an emotion engine.

[1898] 6. The server uses a generation AI to generate a response such as, "Sorry. We will bring you a new dish right away," and sends it to the device.

[1899] 7. The terminal displays a customer avatar responding with voice and animation.

[1900] 8. The user continues to ask questions about the response, practicing realistic customer interactions using emotion recognition.

[1901] In this way, this system, which combines an emotion engine, can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[1902] The processing flow will be explained below.

[1903] MODE FOR CARRYING OUT THE INVENTION

[1904] This invention is a virtual reality system for customer service training that combines an emotion engine that recognizes the user's emotions. The specific processing flow is shown below.

[1905] Processing Steps

[1906] Step 1:

[1907] A user logs in to the system and accesses the persona configuration screen.

[1908] Step 2:

[1909] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button.

[1910] Step 3:

[1911] The server receives the persona information and stores it in a database.

[1912] Step 4:

[1913] The server calls the generative AI model based on the stored persona information.

[1914] Step 5:

[1915] The generative AI generates the appearance, clothing, and behavior of a customer avatar based on the persona information entered.

[1916] Step 6:

[1917] The server prepares the generated customer avatar data for the virtual reality environment.

[1918] Step 7:

[1919] The user accesses a situation configuration screen and selects a particular situation from a drop-down list.

[1920] Step 8:

[1921] The user presses the "Set Situation" button.

[1922] Step 9:

[1923] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[1924] Step 10:

[1925] The user puts on the VR device (terminal) and enters the simulation environment.

[1926] Step 11:

[1927] The user speaks to the customer avatar.

[1928] Step 12:

[1929] The device collects the user's voice, facial expressions, and gestures using a camera and microphone, and transmits the data to a server in real time.

[1930] Step 13:

[1931] The server analyzes the received voice, facial expressions, and gesture data, and recognizes the user's emotions using an emotion engine.

[1932] Step 14:

[1933] Based on the results of emotion recognition, the server uses generative AI to adjust and generate the customer avatar's response.

[1934] Step 15:

[1935] The server transmits the generated response data to the terminal.

[1936] Step 16:

[1937] The terminal displays the response as audio and animation of a customer avatar.

[1938] Step 17:

[1939] Users continuously interact with the customer avatar and experience various situations.

[1940] Step 18:

[1941] The server logs the user's interactions with the customer avatar and the emotion recognition.

[1942] Step 19:

[1943] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[1944] Specific examples

[1945] For example, let's take the situation of handling a complaint at a restaurant.

[1946] Step 1:

[1947] A user logs in to the system and accesses the persona configuration screen.

[1948] Step 2:

[1949] The user enters the persona information as "male in his 40s, has extensive experience in the restaurant industry, and is very prone to dissatisfaction" and clicks the "Submit" button.

[1950] Step 3:

[1951] The server receives the persona information and stores it in a database.

[1952] Step 4:

[1953] The server calls the generative AI model based on the stored persona information.

[1954] Step 5:

[1955] The generation AI generates a customer avatar with specific clothing and behavior based on the persona information entered.

[1956] Step 6:

[1957] The server prepares the generated customer avatar data for the virtual reality environment.

[1958] Step 7:

[1959] The user accesses the situation setting screen and selects the situation "Handling a complaint."

[1960] Step 8:

[1961] The user presses the "Set Situation" button.

[1962] Step 9:

[1963] The server loads the scripts and scenarios corresponding to the selected situation and is ready to go.

[1964] Step 10:

[1965] The user puts on the VR device (terminal) and enters the simulation environment.

[1966] Step 11:

[1967] The user complains to the customer avatar, saying, "This food is cold."

[1968] Step 12:

[1969] The device collects the user's voice and facial expressions and transmits them to a server in real time.

[1970] Step 13:

[1971] The server analyzes the voice, facial expressions, and gesture data it receives, and uses an emotion engine to recognize "dissatisfaction."

[1972] Step 14:

[1973] Based on the results of emotion recognition, the server uses generative AI to generate a response such as, "We're sorry. We'll bring you a new dish right away."

[1974] Step 15:

[1975] The server transmits the generated response data to the terminal.

[1976] Step 16:

[1977] The device displays a customer avatar responding with voice and animation.

[1978] Step 17:

[1979] The user continues to ask further questions based on the responses, practicing realistic customer interactions using emotion recognition.

[1980] Step 18:

[1981] The server logs the user's interactions with the customer avatar and the emotion recognition.

[1982] Step 19:

[1983] After the training is completed, the server analyzes the logs and provides the user with feedback, including emotional data.

[1984] Example 2

[1985] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1986] Modern customer service training requires practical and interactive training methods. However, traditional training methods have difficulty recreating real-life customer interactions, limiting the opportunities for trainees to demonstrate their actual skills. Furthermore, the lack of a system that can recognize trainees' emotions and reactions in real time and provide adaptive feedback based on that information makes it difficult to achieve deeper learning.

[1987] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1988] In this invention, the server includes means for collecting and transmitting the voice, facial expressions, and gestures of the trainee to the server, means for analyzing the received voice, facial expressions, and gesture data to recognize emotions, and means for using a generative artificial intelligence to adjust and generate responses of a customer avatar based on the results of the emotion recognition. This makes it possible to recognize the emotions and reactions of the trainee in real time and provide adaptive feedback based on them.

[1989] "Training Recipient" means a person receiving customer service training.

[1990] "Persona information" refers to data such as age, gender, hobbies, and characteristics that are used as basic information when generating a customer avatar.

[1991] "Interface" refers to the screen or operating means used by trainees to enter persona information.

[1992] "Generative AI" is an AI technology for generating the appearance, clothing, and behavior of a customer avatar based on persona information.

[1993] "Virtual reality device" means equipment used by a trainee to interact with a customer avatar in a virtual environment.

[1994] "Server" means a computer system for storing persona information and providing responses of generated customer avatars.

[1995] "Voice, facial expression, and gesture data" refers to information about the voice, facial expression, and body movements of the training subject, and is data used for emotion recognition.

[1996] "Emotion recognition" is a technology that grasps the emotions of trainees from their voices, facial expressions, and gestures.

[1997] A "customer avatar" is a virtual character that is generated based on persona information and interacts with the training subject in a virtual environment.

[1998] "Response" refers to the reply or action that the customer avatar gives in response to an interaction from the training subject.

[1999] "Log" refers to data that records the interactions between the training subject and the customer avatar and the results of emotion recognition.

[2000] "Feedback" refers to information, including evaluations and areas for improvement, that is provided to trainees after the training has been completed.

[2001] "Specific situations" are specific situations or scenes that trainees experience in a virtual environment, such as handling complaints, explaining new products, or accepting orders.

[2002] This invention combines a virtual reality system for customer service training with an emotion engine that recognizes user emotions. The main components of this system include a user terminal, a server, a generative AI model, and a virtual reality device. Specific implementation methods are described below.

[2003] First, the user logs in to the system and accesses the persona setting screen. The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. The server receives the submitted persona information and stores it in a database. The server then invokes a generative AI model based on the stored persona information to generate the appearance, clothing, and behavior of the customer avatar. An example of a prompt message for this is as follows:

[2004] "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[2005] The generated customer avatar data is prepared for the virtual reality environment by the server. The user accesses the situation setting screen and selects a specific situation. Examples include "handling a complaint," "explaining a new product," and "accepting an order." When the user presses the "Set Situation" button, the server loads the script or scenario corresponding to the selected situation and completes the preparation.

[2006] Next, the user puts on the VR device (terminal) and enters the simulation environment. When the user speaks to the customer avatar, the terminal collects the user's voice, as well as facial expressions and gestures using a camera and microphone, and sends the collected data to the server in real time. The server analyzes the received voice, facial expressions, and gesture data and recognizes the user's emotions using an emotion engine. For example, if a user says, "This dish is cold," and their face turns grim, the server analyzes the voice and facial expression data and recognizes the emotion as "dissatisfied."

[2007] Based on the emotion recognition results, the server uses a generative AI model to adjust and generate a response for the customer avatar. For example, a response such as "Sorry, we'll bring you a new dish right away" may be generated. The server then sends the generated response data to the device, which then displays the customer avatar's response with voice and animation.

[2008] In this way, the user continuously interacts with the customer avatar and experiences various situations. The server records the interaction between the user and the customer avatar and the emotion recognition log, analyzes the log after the training is completed, and provides the user with feedback including the emotion data.

[2009] For example, in a situation where a customer is handling a complaint at a restaurant, the user sets their persona as "a man in his 40s with extensive restaurant experience who is very prone to becoming dissatisfied." The server uses generative AI based on this persona information to generate a customer avatar with specific clothing and behavior, and the user begins the simulation in the "handling a complaint" situation. When the user says, "This dish is cold," the device sends their voice and facial expression to the server, which analyzes it and generates an appropriate response, which is sent back to the device.

[2010] In this way, the system can provide a realistic training experience by recognizing the user's emotions in real time and generating responses based on them.

[2011] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2012] Step 1:

[2013] A user logs in to the system and accesses the persona configuration screen. They provide their username and password as input, and the server performs the authentication process. After successful authentication, they are redirected to the persona configuration screen.

[2014] Step 2:

[2015] The user enters persona information (age, gender, hobbies, characteristics, etc.) and clicks the "Submit" button. Input data includes age "40", gender "male", hobbies "reading", and characteristics "kind". The server receives the persona information and saves it in the database. Specifically, the server analyzes the persona information sent, generates an SQL statement to insert into the database, connects to the database, and executes the SQL statement.

[2016] Step 3:

[2017] The server calls the generative AI model using the saved persona information. It uses the persona information as input to generate a prompt. This prompt is passed to the generative AI model to generate a customer avatar. As output, appearance information (hair color, clothing style, etc.) and behavior information are obtained. An example prompt is "Please generate a customer avatar for a man in his 40s who enjoys reading and has a kind personality."

[2018] Step 4:

[2019] The server prepares the generated customer avatar data for the virtual reality environment. It uses the generated appearance and behavior information as input data, converts it into a format suitable for the VR environment, and organizes the necessary files. As an output, it obtains customer avatar data for the VR environment.

[2020] Step 5:

[2021] The user accesses the situation setting screen and selects a specific situation from a drop-down list. The user selects a situation (e.g., "Complaint handling") as input data and presses the "Set Situation" button. The server retrieves the script corresponding to the selected scenario from the database and loads it into memory. The scenario data is prepared as output.

[2022] Step 6:

[2023] The user puts on the VR device (terminal) and enters the simulation environment. The input is to start up the VR device and connect it to the system. The terminal starts up the VR environment and places the user in the simulation. This allows the user to start training in the virtual environment.

[2024] Step 7:

[2025] The user speaks to the customer avatar. For example, they may say, "This dish is cold." The device collects the user's voice and simultaneously records their facial expressions and gestures using a camera and microphone. This data is sent to the server in real time. The collected voice, facial expressions, and gesture data is sent to the server as output.

[2026] Step 8:

[2027] The server analyzes the received voice, facial expression, and gesture data, and recognizes the user's emotions using an emotion engine. Voice data, facial expression data, and gesture data are used as input, and an emotion recognition algorithm is applied. The output is an emotion result such as "dissatisfied."

[2028] Step 9:

[2029] The server uses generative AI to adjust and generate the customer avatar's response based on the emotion recognition results. The emotion recognition results and generative AI model are used as input. When the generative AI model is sent a prompt such as "How should you respond to an angry customer?", the output generated is a response such as "We're sorry. We'll bring you a new dish right away."

[2030] Step 10:

[2031] The server sends the generated response data to the terminal, uses the generated response data as input and sends it to the terminal, and when the terminal receives the response data, displays the response as voice and animation of the customer avatar.

[2032] Step 11:

[2033] Users continually interact with customer avatars and experience various situations, allowing them to simulate real-life customer service scenarios and improve their response skills.

[2034] Step 12:

[2035] The server records the interaction between the user and the customer avatar and the emotion recognition log. Each interaction and the emotion recognition results are used as input and stored in a database. After the training is completed, the log is analyzed and feedback, including the emotion data, is provided to the user.

[2036] Step 13:

[2037] After the training is completed, the server analyzes the logs and provides feedback to the user, including emotional data. Using the recorded log data as input, the server generates a report that includes the user's behavioral trends and areas for improvement. Detailed feedback is provided to the user as output.

[2038] (Application example 2)

[2039] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2040] Traditional customer service training systems are limited to standard scenarios and situations, and are unable to respond to trainees' emotions and reactions in real time. This limits the effectiveness of training and causes problems such as a lack of flexibility and responsiveness required for actual customer interactions. It is also difficult to track trainees' emotions and provide appropriate feedback based on them.

[2041] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2042] In this invention, the server includes means for collecting voice data and facial expression and gesture data captured by a camera of the trainee and transmitting them to the server in real time, means for analyzing the collected data and recognizing the trainee's emotions with an emotion engine, and means for adjusting and generating responses of a customer avatar using generative artificial intelligence based on the results of emotion recognition. This makes it possible to generate responses in real time based on the trainee's emotions and improve customer service skills in brick-and-mortar stores in a manner closer to reality.

[2043] "Training Participant" means a person who participates in a customer service training program and seeks to improve their skills using a virtual reality system.

[2044] "Persona information" refers to personal information such as age, gender, hobbies, and characteristics entered by the training participant, and a customer avatar is generated based on this information.

[2045] "Generative AI" refers to AI that has the ability to generate new data based on input data, and is used to generate the appearance and behavior of customer avatars.

[2046] A "virtual reality device" is a device that displays a virtual reality space and allows users to interact with that space. Examples include head-mounted displays and VR goggles.

[2047] A "server" is a computer that processes and stores data over a network, and is used here for generating responses for customer avatars, recording logs, and analyzing emotions.

[2048] A "customer avatar" is a virtual person generated within a virtual reality system based on persona information, and is responsible for interacting with the training recipient.

[2049] An "emotion engine" is software that analyzes voice data, facial expression data, and gesture data to recognize the emotional state of the subject.

[2050] An "interface" refers to the operation screen and input device that trainees use to interact with the system.

[2051] A "log" is a record of the system's operation history and data, and in this case, it records the interaction between the training participant and the customer avatar.

[2052] "Feedback" refers to the evaluation and areas for improvement provided to the training participants, and is provided based on information analyzed from the logs.

[2053] A "scenario" is a sequence of events or dialogue used in a specific situation or setting, and is used to set the scene for training.

[2054] This invention is a system that combines a virtual reality system for customer service training with an emotion engine that recognizes the emotions of users. Specific embodiments will be described below.

[2055] Structure and program overview

[2056] This system allows trainees to input customer persona information and interact with a customer avatar generated based on that information. The system uses a smartphone or a head-mounted display (VR device) as hardware, and includes a server for generating real-time responses, an emotion engine for recognizing emotions, and generative artificial intelligence.

[2057] Hardware and Software Used

[2058] Smartphone or head-mounted display (VR device): A device that allows users to interact in a virtual space.

[2059] Server: A computer that processes data, analyzes it, and generates responses.

[2060] Emotion engine: Software that analyzes voice data, facial expression data, and gesture data to recognize the user's emotions.

[2061] Generative artificial intelligence: Software that generates customer avatars based on persona information and generates responses in real time based on user sentiment.

[2062] Data processing and calculation

[2063] 1. Login and Persona Settings

[2064] The user logs in to the smartphone app and enters their age, gender, hobbies, and characteristics on the persona setting screen. This persona information is sent to the server and stored in the database.

[2065] 2. Customer avatar generation

[2066] The server passes the persona information to a generation AI to generate the appearance, clothing, and behavior of the customer avatar, and the generated data is then appropriately prepared for the VR environment.

[2067] 3. Setting the situation

[2068] The user selects a training situation on the app screen, and the server loads the corresponding script and scenario.

[2069] 4. Simulation and real-time response

[2070] The user puts on the VR device and begins interacting with a customer avatar in a virtual store.

[2071] The smartphone or VR device collects the user's voice data, facial expression data, and gesture data and transmits them to a server in real time.

[2072] The server's emotion engine analyzes the data and recognizes the user's emotions.

[2073] Based on the results of emotion recognition, the generative AI model generates a response for the customer avatar, which the server then sends to the device, allowing the avatar to respond in real time.

[2074] Specific examples

[2075] Customer Service Training Scenarios

[2076] 1. The user enters "Female in her 30s, gentle personality, retail store worker, positive" on the "Persona Settings" screen and submits.

[2077] 2. The server generates a "gentle customer avatar, a woman in her 30s" based on the persona information.

[2078] 3. The user selects and sets the situation "Explanation of a new product."

[2079] 4. The user wearing the VR device speaks to the avatar, asking, "What are the features of this new product?" The camera captures the user's facial expressions and sends them to the server along with the audio data.

[2080] 5. The server analyzes the user's facial expressions and uses an emotion engine to recognize "interest." The generative AI model generates a response, such as "This new product is compact, lightweight, and made from eco-friendly materials." The server then sends the response to the device, where the customer avatar responds with voice and animation.

[2081] Prompt Sentence Examples

[2082] Persona information: Female in her 30s, gentle personality, retail store worker, positive

[2083] Situation: New product description

[2084] User Sentiment: Interest

[2085] What it produces: This new product is compact, lightweight and made from eco-friendly materials.

[2086] This allows users to improve their skills in dealing with real customers.

[2087] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2088] Step 1:

[2089] A user logs into the system and enters persona information

[2090] Input: The user enters persona information such as age, gender, hobbies, and characteristics.

[2091] How it works: The user enters persona information (e.g., female in her 30s, gentle personality, retail worker, positive) through the interface and clicks the "Submit" button.

[2092] Output: Persona information is sent to the system and stored on the server.

[2093] Step 2:

[2094] The server generates a customer avatar based on the persona information

[2095] Input: Saved persona information.

[2096] How it works: The server retrieves persona information and uses generative artificial intelligence (generative AI model) to generate the appearance, clothing, and behavior of a customer avatar.

[2097] Output: The generated customer avatar is prepared for the virtual reality environment.

[2098] Step 3:

[2099] Users set training situations

[2100] Input: Select a training situation (e.g., explaining a new product).

[2101] How it works: The user selects a specific situation on the situation setting screen and presses the "Set Situation" button.

[2102] Output: The scripts and scenarios corresponding to the selected situations are loaded by the server.

[2103] Step 4:

[2104] The user puts on the VR device and starts the simulation.

[2105] Input: Customer avatar and situation script generated based on persona information.

[2106] How it works: A user puts on a VR device and begins interacting with a customer avatar in a virtual store.

[2107] Output: The interaction is initiated by the user speaking to the customer avatar.

[2108] Step 5:

[2109] The device collects the user's voice, facial expressions, and gestures and sends them to the server.

[2110] Input: User's voice data, facial expression data captured by the camera, and gesture data.

[2111] Actions: The device's sensors capture the user's voice, facial expressions, and gestures and transmit them to the server in real time.

[2112] Output: The user's voice, facial expression, and gesture data are sent to the server.

[2113] Step 6:

[2114] The server recognizes the user's emotions using an emotion engine

[2115] Input: Collected voice data, facial expression data, and gesture data.

[2116] How it works: The server's emotion engine analyzes the collected data and recognizes the user's emotions (e.g., interest, frustration).

[2117] Output: The analyzed emotion information is passed to the generative AI model.

[2118] Step 7:

[2119] The server generates a response for the customer avatar based on the emotion recognition results.

[2120] Input: Emotion recognition results, generative artificial intelligence model.

[2121] How it works: The server uses the generative AI model to generate an appropriate customer avatar response based on the recognized emotion (e.g., "This new product is compact, lightweight, and made from eco-friendly materials.").

[2122] Output: The generated response data is sent to the terminal.

[2123] Step 8:

[2124] The device displays the response to the customer avatar

[2125] Input: The response data of the generated customer avatar.

[2126] How it works: The device displays and plays the generated response data with animation and audio, and the customer avatar responds in real time.

[2127] Output: The user confirms the generated response and continues the interaction.

[2128] Step 9:

[2129] The server logs the interaction and provides feedback to the trainee.

[2130] Input: All interaction data for the user and customer avatar.

[2131] How it works: The server logs interactions and later analyzes them to provide feedback to the user.

[2132] Output: The user receives feedback including the results of the training and areas for improvement.

[2133] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2134] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2135] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2136] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2137] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2138] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2139] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2140] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2141] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2142] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2143] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2144] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2145] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2146] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2147] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2148] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2149] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2150] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2151] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2152] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2153] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2154] The following is further disclosed regarding the above embodiment.

[2155] (Claim 1)

[2156] In a virtual reality system for customer service training,

[2157] a means for providing an interface for a trainee to input customer persona information;

[2158] a means for using generative artificial intelligence to generate a customer avatar based on the persona information;

[2159] a means for using a virtual reality device for the training recipient to interact with the customer avatar;

[2160] means for using a server to generate and provide customer avatar responses in real time to a virtual reality device;

[2161] means for logging said interactions and providing feedback to the training recipient;

[2162] A system including:

[2163] (Claim 2)

[2164] 10. The system of claim 1, wherein the appearance, clothing, and behavior of the generated customer avatar are customized based on the persona information.

[2165] (Claim 3)

[2166] 10. The system of claim 1, further comprising means for loading a script or event corresponding to a particular situation.

[2167] "Example 1"

[2168] (Claim 1)

[2169] a means for providing an interface for a trainee to input customer persona information;

[2170] using a generative AI model to generate customer avatars based on persona information;

[2171] a means for using a virtual reality device for the training recipient to interact with the customer avatar;

[2172] a means for using a server to generate responses using a speech analysis engine and a generative AI model to generate responses for a customer avatar in real time and provide the responses to a virtual reality device;

[2173] means for logging said interactions and providing feedback to the training recipient;

[2174] A system including:

[2175] (Claim 2)

[2176] 10. The system of claim 1, wherein the appearance, clothing, and behavior of the generated customer avatar are customized based on the persona information.

[2177] (Claim 3)

[2178] 10. The system of claim 1, further comprising means for loading a script or scenario corresponding to a particular situation.

[2179] "Application Example 1"

[2180] (Claim 1)

[2181] a means for providing an interface for a trainee to input customer persona information;

[2182] a means for using a generative model to generate a customer avatar based on the persona information;

[2183] a means for using a virtual reality device for the training recipient to interact with the customer avatar;

[2184] means for using a data processing device to generate and provide responses of a customer avatar in real time to a virtual reality device;

[2185] means for logging said interactions and providing feedback to the training recipient;

[2186] and means for loading scripts and events corresponding to specific situations, such as product descriptions and order taking, in a physical store environment.

[2187] A system including:

[2188] (Claim 2)

[2189] 10. The system of claim 1, wherein the appearance, clothing, and behavior of the generated customer avatar are customized based on the persona information.

[2190] (Claim 3)

[2191] 10. The system of claim 1, further comprising means for analyzing and converting the training subject's voice data in real time and generating responses for the customer avatar using a generative AI model based on the analysis results.

[2192] "Example 2: Combining Emotion Engines"

[2193] (Claim 1)

[2194] a means for providing an interface for a trainee to input customer persona information;

[2195] a means for using generative artificial intelligence to generate a customer avatar based on the persona information;

[2196] a means for using a virtual reality device for the training recipient to interact with the customer avatar;

[2197] means for using a server to generate and provide customer avatar responses in real time to a virtual reality device;

[2198] A means for collecting the voice, facial expressions, and gestures of the training recipient and transmitting them to a server;

[2199] A means for analyzing received voice, facial expression, and gesture data to perform emotion recognition;

[2200] a means for using generative artificial intelligence to adjust and generate responses of the customer avatar based on the emotion recognition results;

[2201] means for logging said interactions and providing feedback to the training recipient;

[2202] A system including:

[2203] (Claim 2)

[2204] 10. The system of claim 1, wherein the appearance, clothing, and behavior of the generated customer avatar are customized based on the persona information.

[2205] (Claim 3)

[2206] 10. The system of claim 1, further comprising means for loading a script or event corresponding to a particular situation.

[2207] "Application example 2 when combining emotion engines"

[2208] (Claim 1)

[2209] a means for providing an interface for a trainee to input customer persona information;

[2210] a means for using generative artificial intelligence to generate a customer avatar based on the persona information;

[2211] a means for using a virtual reality device for the training recipient to interact with the customer avatar;

[2212] means for using a server to generate and provide customer avatar responses in real time to a virtual reality device;

[2213] means for logging said interactions and providing feedback to the training recipient;

[2214] A means for collecting voice data and facial expression and gesture data captured by a camera of the training subject and transmitting the collected data to a server in real time;

[2215] A means of analyzing the collected data and recognizing the emotions of the training participants using an emotion engine;

[2216] A means for adjusting and generating responses of the customer avatar using artificial intelligence based on the results of emotion recognition;

[2217] A system including:

[2218] (Claim 2)

[2219] 10. The system of claim 1, wherein the appearance, clothing, and behavior of the generated customer avatar are customized based on the persona information.

[2220] (Claim 3)

[2221] 10. The system of claim 1, further comprising means for loading a script or event corresponding to a particular situation. [Explanation of symbols]

[2222] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. In a virtual reality system for customer service training, a means for providing an interface for a trainee to input customer persona information; a means for using generative artificial intelligence to generate a customer avatar based on the persona information; a means for using a virtual reality device for the training recipient to interact with the customer avatar; means for using a server to generate and provide customer avatar responses in real time to a virtual reality device; means for logging said interactions and providing feedback to the training recipient; A system including:

2. The system of claim 1 , wherein the appearance, clothing, and behavior of the generated customer avatar are customized based on the persona information.

3. The system of claim 1 further comprising means for loading a script or event corresponding to a particular situation.

Images