System

Abstract

A system is provided.SOLUTION: A system comprising: user registration means; means for displaying a plurality of scenario data; generation means for generating a conversation scenario based on scenario data selected by a user; acquisition means for acquiring answer data from the user; analysis means for analyzing the acquired answer data and generating feedback in real time; providing means for providing the generated feedback to the user; and means for storing and analyzing progress data of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The rise of remote work and communication in today's world has led to fewer opportunities for face-to-face communication. This has led to a lack of communication skills, especially in important situations such as interviews and presentations. Many people suffer from a lack of confidence, practice, and feedback in real-life communication situations, which reduces their competitiveness in the labor market. Therefore, there is a need for a way for users to efficiently improve their communication skills through realistic situations. [Means for solving the problem]

[0005] The present invention provides a system including a user registration means, a display means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, and a means for saving and analyzing user progress data. This system allows users to receive real-time feedback through interactive role-playing, efficiently improving their communication skills. Furthermore, the saving and analysis of progress data enables continuous self-evaluation and growth.

[0006] "User registration procedure" refers to the process by which a user enters their information into the system and creates a registration profile.

[0007] "Scenario data" refers to information on various communication scenarios that a user can select and use.

[0008] "Generation means" refers to a system component that automatically generates an actual conversation scenario based on scenario data selected by the user.

[0009] "Acquisition means" refers to the system's function of collecting response data entered by users.

[0010] "Answer data" refers to the response information provided by a user in response to a question posed by the system.

[0011] "Analysis means" refers to the system component that analyzes the acquired response data in real time and generates suggestions for improvement and appropriate feedback.

[0012] "Delivery means" refers to the system's function of delivering the generated feedback to the user.

[0013] "Progress data" refers to data that records the user's learning and skill improvement history.

[0014] "Storage" refers to the process by which the system writes acquired data to a database or other storage medium.

[0015] "Analysis" refers to the process of evaluating a user's progress and areas for improvement based on the stored data. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system supports continuous development by saving and analyzing users' progress.

[0038] Program processing overview

[0039] Initial Setup and User Registration

[0040] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[0041] Scenario Selection

[0042] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[0043] Role-playing

[0044] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to a generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[0045] Real-time feedback

[0046] For each answer, the server analyzes it and generates feedback that is immediately sent to the user, allowing them to refine and submit their answer again.

[0047] Progress Tracking and Analysis

[0048] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[0049] Specific examples

[0050] Here, a specific example is shown in which a user "Sato-san" uses the system to carry out a technical interview scenario.

[0051] 1. Initial setup and user registration:

[0052] Mr. Sato accesses the new registration screen and enters the required information (user name, email address, password).

[0053] The server receives this information and stores it in a database.

[0054] 2. Scenario Selection:

[0055] Sato accesses the dashboard and selects "Technical Interview Scenario."

[0056] The server loads the data for the selected scenario and prepares the generative AI model.

[0057] 3. Role-playing:

[0058] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Sato.

[0059] Sato replies, "I've recently been working on developing a web application."

[0060] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[0061] 4. Real-time feedback:

[0062] Based on the feedback, Sato answers again, "I used React as the front end and Node.js as the back end."

[0063] The server will analyze again and provide more specific feedback.

[0064] 5. Progress Tracking and Analysis:

[0065] After the session ends, the server stores Sato's answer history, feedback history, and progress data.

[0066] The server generates a progress report based on the stored data and sends it to Mr. Sato.

[0067] Sato checks the report and visually confirms his own growth.

[0068] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and solves the problem of improving communication abilities in a remote environment.

[0069] The processing flow will be explained below.

[0070] Specific explanation of program processing

[0071] Initial Setup and User Registration

[0072] Step 1:

[0073] The server establishes a database connection when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[0074] Step 2:

[0075] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[0076] Step 3:

[0077] After the user has completed the input, he clicks the Register button.

[0078] Step 4:

[0079] The terminal transmits the input information to the server.

[0080] Step 5:

[0081] The server receives the information, checks it for format and duplication, and stores it in a database.

[0082] Scenario Selection

[0083] Step 6:

[0084] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[0085] Step 7:

[0086] The terminal displays a list of scenarios to the user.

[0087] Step 8:

[0088] The user selects the scenario of interest and clicks the Select button.

[0089] Step 9:

[0090] The terminal transmits the scenario information selected by the user to the server.

[0091] Step 10:

[0092] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[0093] Role-playing

[0094] Step 11:

[0095] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[0096] Step 12:

[0097] The terminal displays the first question to the user.

[0098] Step 13:

[0099] The user answers the questions by typing or speaking.

[0100] Step 14:

[0101] The terminal transmits the inputted answer to the server.

[0102] Step 15:

[0103] The server passes the answer to a generative AI model, which analyzes it in real time.

[0104] Step 16:

[0105] The server generates appropriate feedback based on the analysis results.

[0106] Step 17:

[0107] The server transmits the generated feedback to the terminal.

[0108] Step 18:

[0109] The device displays feedback to the user.

[0110] Real-time feedback

[0111] Step 19:

[0112] The user improves their answer based on the feedback and re-enters it.

[0113] Step 20:

[0114] The device sends the improved answer to the server.

[0115] Step 21:

[0116] The server then passes the answer back to the generative AI model for analysis.

[0117] Step 22:

[0118] The server generates new feedback and sends it back to the device.

[0119] Step 23:

[0120] The device displays new feedback to the user.

[0121] Progress Tracking and Analysis

[0122] Step 24:

[0123] The server stores the user's answer history, feedback history, and progress data in a database after each session.

[0124] Step 25:

[0125] The server analyzes the user's progress based on the stored data and generates a visual report.

[0126] Step 26:

[0127] The server sends the generated progress report to the terminal.

[0128] Step 27:

[0129] The terminal displays a progress report to the user.

[0130] As described above, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills.

[0131] Example 1

[0132] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0133] Conventional communication skill improvement systems have difficulty providing real-time feedback to users' responses, preventing them from immediately improving their communication skills. Furthermore, they lack the ability to store and analyze progress data, preventing them from effectively supporting users' continuous improvement.

[0134] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0135] In this invention, the server includes user registration means, means for displaying multiple pieces of scenario information, generation means for generating a dialogue scenario based on scenario information selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, transmission means for the user's terminal to transmit input response data to the server, transmission means for the server to transmit the generated feedback to the user's terminal, and display means for the terminal to receive the data and display it to the user. This allows the user to receive instant feedback and continuously improve their communication skills.

[0136] The "user registration means" is a means by which a user inputs information for new registration in the system and stores that information in the database.

[0137] "Scenario information" is information relating to a number of dialogue scenarios that the user can select from, and role-playing is carried out based on this information.

[0138] The "generation means" is a means for generating a dialogue scenario and a prompt sentence based on scenario information selected by the user.

[0139] The "acquisition means" is a means by which the system acquires response data from the user.

[0140] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time based on the analysis results.

[0141] The "means for providing" is a means for transmitting the generated feedback to the user's terminal and displaying it.

[0142] "Progress Data" refers to data including a user's response history, feedback history, progress status, etc., and is used to evaluate a user's growth and improvement.

[0143] The "transmission means" is a means by which the user's terminal transmits response data to the server, and the server transmits the generated feedback to the user's terminal.

[0144] The "display means" is a means for visually displaying to the user the feedback and scenario information received by the user's terminal.

[0145] A "generative AI model" is an artificial intelligence model used to analyze user response data and generate appropriate feedback or the next prompt.

[0146] A "prompt sentence" is dialogue text that includes a question or instruction for the user to follow next.

[0147] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from a variety of scenarios, role-play based on the scenarios, and receive real-time feedback. The system also supports continuous improvement by saving and analyzing the user's progress.

[0148] This system is implemented primarily using the following hardware and software:

[0149] Hardware: Servers, user devices (PCs, smartphones, tablets)

[0150] Software: Database, generative AI model (e.g., GPT-3), front-end framework (e.g., React), back-end framework (e.g., Node.js)

[0151] When the system starts up, the server initializes the database and creates the necessary tables, such as the user table, scenario table, and log table. This allows users to register with the system and prepare it for use. When a user accesses the new registration screen and enters information such as their username, email address, and password, the terminal sends this information to the server. The server receives the data and stores it in the database.

[0152] The server then retrieves available scenario information from the database and sends it to the user's device, allowing the user to select the scenario they are interested in from the displayed list of scenarios. Once the user selects a scenario, the selection information is sent to the server, which then prepares a generative AI model based on the selected scenario.

[0153] When role-playing begins, the server generates questions for the first phase of the selected scenario and sends them to the user. For example, in a technical interview scenario, a question such as "Tell me about a project you recently worked on" is generated. The user enters answers to the questions, and the answer data is sent to the server via the terminal.

[0154] The server passes the received answers to the generative AI model, which analyzes them in real time. The generated feedback is then sent back to the user's device and displayed to the user. Depending on the feedback, the user can refine their answer and send it back to the server. By repeating this process, the user's communication skills can be improved.

[0155] After the session ends, the server saves the user's answer history, feedback history, and progress data. Based on this data, the server analyzes the user's progress and generates a visual report. The report is sent to the user's device, allowing the user to visually check their progress.

[0156] For example, if user "Sato" uses the technical interview scenario,

[0157] 1. Initial setup and user registration:

[0158] Mr. Sato accesses the new registration screen and enters his username, email address, and password.

[0159] The server receives this information and stores it in a database.

[0160] 2. Scenario Selection:

[0161] Sato accesses the dashboard and selects the technical interview scenario.

[0162] The server loads the data for the selected scenario and prepares the generative AI model.

[0163] 3. Role-playing:

[0164] The server generates a question such as "Tell me about a project you've worked on recently" and sends it to Sato.

[0165] Mr. Sato enters the answer and sends it to the server.

[0166] The server analyzes the answers using a generative AI model, generating feedback such as, "Please explain the specific technology stack and role."

[0167] 4. Real-time feedback:

[0168] Sato refines the answer based on the feedback and submits it to the server again.

[0169] The server analyzes the refined answer and provides more specific feedback.

[0170] 5. Progress Tracking and Analysis:

[0171] After the session ends, the server stores Mr. Sato's answer history, feedback history, and progress data in a database.

[0172] A progress report is generated based on the saved data and sent to Sato.

[0173] Sato checks the report and visually confirms his own growth.

[0174] Examples of prompts include:

[0175] "Tell me about a project you've worked on recently."

[0176] "Please explain your specific role in the project and the technology you used."

[0177] "Tell us about a challenge you faced in the project and how you solved it."

[0178] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and aims to improve communication abilities in remote environments.

[0179] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0180] Program flow:

[0181] Step 1: Initialize the database and set up the structure

[0182] The server initializes the database when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[0183] Input: System startup event

[0184] Data manipulation: Creating tables using SQL queries

[0185] Output: Initialized database

[0186] Specific behavior:

[0187] Generate each table with the CREATE TABLE statement

[0188] Insert initial data into the database using INSERT statements as needed

[0189] Step 2: Receiving and storing user registration information

[0190] The user accesses the new registration screen, enters their username, email address, and password, and presses the registration button. The device then sends this data to the server.

[0191] Input: User registration information (user name, email address, password)

[0192] Data processing: data validation and encoding

[0193] Output: User information stored in the database

[0194] Specific behavior:

[0195] The front-end collects form data and sends it to the server via API

[0196] The backend performs input validation (e.g., email address format check)

[0197] The password is hashed and saved in the database using an INSERT statement.

[0198] Step 3: Get and display the scenario list

[0199] The server retrieves available scenario information from the database and sends it to the user's terminal, which then displays the received scenario information.

[0200] Input: Request to get a scenario list

[0201] Data processing: Retrieving scenario information from the database

[0202] Output: Scenario information sent to the user's device

[0203] Specific behavior:

[0204] Retrieve a list of scenarios from the database using a SELECT statement

[0205] The acquired data is encoded in JSON format and sent to the terminal via API.

[0206] The front end displays the scenario list in list format.

[0207] Step 4: Select and submit a scenario

[0208] The user selects the scenario of interest and sends the selection to the server, which then prepares a generative AI model based on the selected scenario.

[0209] Input: ID of the selected scenario

[0210] Data processing: Loading AI models based on scenario ID

[0211] Output: A prepared generative AI model

[0212] Specific behavior:

[0213] The front end gets the ID of the selected scenario and sends it to the server

[0214] The server loads the generative AI model based on the scenario ID and prepares the next prompt.

[0215] Step 5: Generate and submit your question

[0216] The server generates the questions for the first phase of the selected scenario and sends them to the user.

[0217] Input: Data for the selected scenario

[0218] Data processing: prompt sentence generation

[0219] Output: The question sent to the user's device

[0220] Specific behavior:

[0221] Executes dialog generation logic based on scenarios

[0222] Generate an initial question, encode it in JSON format, and send it to the terminal.

[0223] The front end displays the question to the user

[0224] Step 6: User answers and submits

[0225] The user inputs an answer to the question, and the answer data is sent to the server via the terminal.

[0226] Input: User response data

[0227] Data processing: Validation and encoding of response data

[0228] Output: Response data sent to the server

[0229] Specific behavior:

[0230] The front end collects the answers entered by the user

[0231] Response data is sent to the server via API

[0232] Step 7: Analyze responses and generate feedback

[0233] The server passes the received answers to a generative AI model for real-time analysis, and the generated feedback is sent to the user's device.

[0234] Input: User response data

[0235] Data processing: Analysis with generative AI models

[0236] Output: Generated feedback

[0237] Specific behavior:

[0238] Provide answer data as input to the generative AI model

[0239] Obtain analysis results and format them as feedback

[0240] Feedback is encoded in JSON format and sent to the user's device

[0241] The front end displays feedback to the user

[0242] Step 8: Re-enter and submit your improved answers

[0243] The user refines the answer based on the feedback and submits it to the server again.

[0244] Input: Improved response data

[0245] Data processing: Revalidation and encoding

[0246] Output: Improved answer sent to the server

[0247] Specific behavior:

[0248] The user refines the answer and enters it again

[0249] Response data is sent to the server again

[0250] Step 9: Save your data

[0251] After the session ends, the server stores the user's answer history, feedback history, and progress data.

[0252] Input: User session data (answers, feedback)

[0253] Data processing: structuring data

[0254] Output: Session data stored in the database

[0255] Specific behavior:

[0256] Save answer history and feedback history to the log table using INSERT statements

[0257] Step 10: Analyze data and generate reports

[0258] The server analyzes the user's progress based on the stored data and creates a visual report, which is then sent to the user's device for display.

[0259] Input: Saved session data

[0260] Data processing: data analysis and visualization

[0261] Output: Progress report sent to the user's terminal

[0262] Specific behavior:

[0263] Analyzes stored data and generates progress reports

[0264] Use a data visualization library (e.g., D3.js) to generate graphs, etc.

[0265] Encode the report in JSON format and send it to the user's device

[0266] The front end displays the report to the user

[0267] In this way, the system provides a series of processes for users to interactively improve their communication skills.

[0268] (Application example 1)

[0269] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] In today's virtual stores, salespeople's communication skills have a significant impact on the quality of the customer experience. However, traditional training methods make it difficult to provide real-time feedback and encourage continuous skill improvement. Furthermore, due to a lack of mechanisms for accurately tracking and analyzing progress, it is difficult to efficiently support salespeople in improving their skills.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0272] In this invention, the server includes user registration means, means for displaying multiple scenario data, generation means for generating a conversation scenario based on scenario data selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, and means for evaluating the sales skills of salespeople in the virtual store and providing scenarios for training them. This allows salespeople to receive feedback in real time while undergoing training, enabling continuous skill improvement.

[0273] The "user registration means" is a function for registering a new user in the system, and is usually a means for inputting information such as a user name, email address, and password, and storing the information in a database.

[0274] "Means for displaying multiple scenario data" is a function that displays a list of various scenarios that the user can select on the screen, and is a means that allows the user to select which specific scenario to use.

[0275] The "means for generating a conversation scenario" is a function for automatically generating a conversation in accordance with a scenario selected by a user, based on the scenario data.

[0276] The "means for acquiring answer data from the user" is a function for the system to receive answers input by the user to the conversation scenario and store them as data.

[0277] The "analysis means for generating feedback in real time" is a function for analyzing user response data and automatically generating appropriate feedback immediately.

[0278] The "means for providing the generated feedback to the user" is a function for transmitting the generated feedback to the user's terminal and displaying it on the screen.

[0279] The "means for saving and analyzing user progress data" is a function for evaluating the improvement of a user's skills by saving the user's past response data and feedback content and analyzing them as appropriate.

[0280] "Means for providing scenarios for evaluating and training salespeople's response skills in a virtual store" refers to a function for providing scenarios designed to improve salespeople's response skills in a virtual store environment and for conducting training through those scenarios.

[0281] The present invention is a system for improving a user's communication ability, and is particularly designed to evaluate and train salespeople's communication skills in a virtual store. The components and processes of this system are as follows:

[0282] System Components

[0283] 1. User registration method

[0284] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[0285] 2. A way to display multiple scenario data

[0286] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[0287] 3. Conversation scenario generation method

[0288] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[0289] 4. How to obtain response data from users

[0290] The server receives the answer data entered by the user when answering the scenario and stores it in a database.

[0291] 5. Analytics that generate real-time feedback

[0292] The server analyzes each answer using a generative AI model and generates feedback, which is immediately provided to the user, allowing them to refine their answer and submit it again.

[0293] 6. Means for providing generated feedback to users

[0294] The generated feedback is immediately sent to the user's device and displayed to the user, allowing the user to receive feedback in real time.

[0295] 7. A means of storing and analyzing user progress data

[0296] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report that is provided to the user. This report allows the user to visually confirm their own progress.

[0297] 8. A means of providing scenarios to assess and train sales associate skills in a virtual store

[0298] The server provides scenarios in a virtual store environment to improve sales staff skills, including new product introductions, complaint handling, and cross-selling, allowing for training that is tailored to actual sales situations.

[0299] Hardware and software used

[0300] This system uses the following hardware and software:

[0301] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[0302] Software: TensorFlow (library underlying generative AI models), MySQL (database management system), Node.js (server-side programming)

[0303] Specific examples

[0304] Scenario where a user is introducing a new product in a virtual store:

[0305] 1. User Registration

[0306] The user accesses the new registration screen and enters the required information (user name, email address, password). The server receives this information and stores it in the database.

[0307] 2. Scenario Selection

[0308] The user accesses the dashboard and selects the "New Product Introduction Scenario." The server loads the data for the selected scenario and prepares the generative AI model.

[0309] 3. Role-playing

[0310] The server generates the first question, "What are the main features of this product?" and sends it to the user. The user answers, "This product has a high-performance battery and fast charging capabilities." The server then runs this answer through a generative AI model, which generates feedback such as, "Please be specific about battery life and charging time."

[0311] Example prompt sentence:

[0312] "If a customer asks about the features of a new product, how would you explain it?"

[0313] The system allows salespeople to receive real-time feedback and effectively improve their skills in the virtual store.

[0314] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0315] Step 1:

[0316] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). This ensures that data for user registration and scenario selection is correctly saved and managed.

[0317] Input: System boot

[0318] Output: Initialize the database and create tables.

[0319] What happens next: The server connects to the MySQL database and creates the necessary tables using the CREATE statement.

[0320] Step 2:

[0321] The user accesses the new registration screen, enters information such as a user name, email address, and password, and presses the registration button.

[0322] Input: Username, Email Address, Password

[0323] Output: Save registration information to database

[0324] Specific operation: The information entered by the user is sent to the server, and the server saves the information in the database using the INSERT statement.

[0325] Step 3:

[0326] The server retrieves a list of available scenarios from the database and sends it to the user's terminal.

[0327] Input: Request scenario list

[0328] Output: Send scenario list

[0329] Specific operation: The server uses a SELECT statement to retrieve scenario data from the database and sends the retrieved data to the user's device in JSON format.

[0330] Step 4:

[0331] The user selects a scenario of interest from the displayed list of scenarios and transmits the selected information to the server.

[0332] Input: Scenario selection information

[0333] Output: Preparation for selected scenarios

[0334] Specific operation: After receiving the user's selection information, the server retrieves detailed data on the selected scenario from the database and prepares the generative AI model.

[0335] Step 5:

[0336] The server generates questions for the first phase of the selected scenario and sends them to the user, who then inputs the answers, which the device then sends to the server.

[0337] Input: Scenario question generation request, user answer

[0338] Output: Send question, get answer

[0339] Specific operation: The server uses the generative AI model to generate questions based on the scenario and sends them to the user's device. When the user enters an answer, the answer is sent to the server.

[0340] Step 6:

[0341] The server passes the answer to the generative AI model, which analyzes it in real time and generates feedback, which is then sent back to the user's device and displayed to them.

[0342] Input: User response data

[0343] Output: Generated feedback

[0344] Specific operation: The server inputs the answer data into the generative AI model, obtains feedback as the model's analysis result, and sends the feedback to the user's device and displays it to the user.

[0345] Step 7:

[0346] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database.

[0347] Input: Session data

[0348] Output: Saved progress data

[0349] Specific behavior: The server stores the user's answers and feedback data in a database using INSERT or UPDATE statements, for future reference in sessions and for progress analysis.

[0350] Step 8:

[0351] The server analyzes the user's progress based on the stored data, creates a visualized report, and provides it to the user.

[0352] Input: Saved progress data

[0353] Output: Generate and send a progress report

[0354] Specific operation: The server analyzes past response data and feedback, generates a report visualizing the user's growth and areas for improvement, and sends it to the user's device.

[0355] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0356] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[0357] Program processing overview

[0358] Initial Setup and User Registration

[0359] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The server receives this information and saves it in the database.

[0360] Scenario Selection

[0361] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[0362] Role-playing

[0363] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[0364] Use of emotion engine

[0365] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to an analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[0366] Real-time feedback

[0367] For each answer, the server performs analysis and emotion recognition and generates feedback that is instantly sent to the user, allowing them to improve their answer and submit it again.

[0368] Progress Tracking and Analysis

[0369] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[0370] Specific examples

[0371] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[0372] 1. Initial setup and user registration:

[0373] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[0374] The server receives this information and stores it in a database.

[0375] 2. Scenario Selection:

[0376] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[0377] The server loads the data for the selected scenario and prepares the generative AI model.

[0378] 3. Role-playing:

[0379] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[0380] Yamada replies, "I've recently been working on developing a web application."

[0381] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[0382] 4. Use of Emotion Engine:

[0383] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[0384] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[0385] 5. Real-time feedback:

[0386] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[0387] The server will analyze again and provide more specific feedback.

[0388] 6. Progress Tracking and Analysis:

[0389] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[0390] The server generates a progress report based on the stored data and sends it to Yamada.

[0391] Yamada checks the report and visually confirms his own growth.

[0392] In this way, by combining emotion engines, it becomes possible to provide more accurate feedback according to the user's emotions, and to more effectively support the improvement of communication skills.

[0393] The processing flow will be explained below.

[0394] Specific explanation of program processing (including emotion engine)

[0395] Initial Setup and User Registration

[0396] Step 1:

[0397] When the system starts up, the server establishes a database connection and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.).

[0398] Step 2:

[0399] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[0400] Step 3:

[0401] After the user has completed the input, he clicks the Register button.

[0402] Step 4:

[0403] The terminal transmits the input information to the server.

[0404] Step 5:

[0405] The server receives the information, checks it for format and duplication, and stores it in a database.

[0406] Scenario Selection

[0407] Step 6:

[0408] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[0409] Step 7:

[0410] The terminal displays a list of scenarios to the user.

[0411] Step 8:

[0412] The user selects the scenario of interest and clicks the Select button.

[0413] Step 9:

[0414] The terminal transmits the scenario information selected by the user to the server.

[0415] Step 10:

[0416] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[0417] Role-playing

[0418] Step 11:

[0419] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[0420] Step 12:

[0421] The terminal displays the first question to the user.

[0422] Step 13:

[0423] The user answers the questions by typing or speaking.

[0424] Step 14:

[0425] The terminal transmits the inputted answer to the server.

[0426] Step 15:

[0427] The server passes the answer to a generative AI model, which analyzes it in real time.

[0428] Use of emotion engine

[0429] Step 16:

[0430] The server extracts emotion information from the acquired user response data using an emotion engine.

[0431] Step 17:

[0432] The emotion engine uses speech and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's responses.

[0433] Step 18:

[0434] The emotion engine provides the recognized emotion information to the server.

[0435] Real-time feedback

[0436] Step 19:

[0437] The server generates appropriate feedback based on the analysis results, including emotional information.

[0438] Step 20:

[0439] The server transmits the generated feedback to the terminal.

[0440] Step 21:

[0441] The device displays feedback to the user.

[0442] Step 22:

[0443] The user improves their answer based on the feedback and re-enters it.

[0444] Step 23:

[0445] The device sends the improved answer to the server.

[0446] Step 24:

[0447] The server then passes the answer back to the generative AI model for analysis.

[0448] Step 25:

[0449] The server generates new feedback and sends it back to the device.

[0450] Step 26:

[0451] The device displays new feedback to the user.

[0452] Progress Tracking and Analysis

[0453] Step 27:

[0454] After each session, the server stores the user's answer history, feedback history, emotion data, and progress data in a database.

[0455] Step 28:

[0456] The server analyzes the user's progress based on the stored data and generates a visual report.

[0457] Step 29:

[0458] The server sends the generated progress report to the terminal.

[0459] Step 30:

[0460] The terminal displays a progress report to the user.

[0461] Through these steps, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills. By utilizing an emotion engine, the system provides feedback based on the user's emotions, providing a more effective learning experience.

[0462] Example 2

[0463] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0464] In conventional communication improvement systems, feedback to users' responses was uniform and could not be adapted to the user's emotions. As a result, the feedback users received was not optimized for individual situations, making it difficult to improve their communication skills efficiently.

[0465] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0466] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring answer data from the user, an analysis means for analyzing the acquired answer data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, a means for performing emotion analysis on the acquired answer data, and a means for reflecting the results of the emotion analysis in the feedback. This makes it possible to provide individualized feedback according to the user's emotions and more effectively improve communication skills.

[0467] "User registration means" refers to the means by which a user accesses the system, inputs his / her own information, and performs registration.

[0468] The "means for displaying a plurality of scenario data" is a means for visually displaying a plurality of scenarios provided by the system to the user.

[0469] The "generation means" is a means for generating a conversation scenario and questions based on scenario data selected by the user.

[0470] The "acquisition means" is a means for collecting response data from users and transmitting it to the system.

[0471] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time.

[0472] The "means for providing" is a means for providing and displaying the generated feedback to the user.

[0473] The "means for saving and analyzing progress data" refers to a means for saving a user's response history and feedback history in a database and analyzing progress based on that history.

[0474] The "means for performing emotion analysis" is a means for analyzing the emotional information contained in the acquired response data and recognizing specific emotions (joy, anger, sadness, etc.).

[0475] The "means for reflecting the results of emotion analysis in the feedback" refers to a means for adjusting the feedback content based on the results of emotion analysis and providing optimal feedback to the user.

[0476] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[0477] Initial Setup and User Registration

[0478] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the entered information to the server, which then stores the received information in the database. When registration is complete, the server sends a success message to the terminal, which displays it to the user.

[0479] Scenario Selection

[0480] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario and prepares for the next phase.

[0481] Role-playing

[0482] The server generates questions for the first phase of the selected scenario and sends them to the user. The device displays the questions to the user, who then enters answers. The device then sends the entered answers to the server, which passes them to the generative AI model for real-time analysis. The server generates feedback based on the analysis results and sends it to the device. The device then displays the generated feedback to the user.

[0483] Use of emotion engine

[0484] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to the analysis means to improve the accuracy of the feedback. For example, if the user's response indicates a negative emotion, the feedback can use an affiliative expression that corresponds to that emotion.

[0485] Real-time feedback

[0486] For each answer, the server analyzes and recognizes emotions, and generates feedback. The feedback is immediately sent to the user's device, allowing the user to improve their answer and submit it again. This allows users to receive continuous feedback and improve their communication skills.

[0487] Progress Tracking and Analysis

[0488] After the session ends, the server stores the user's answer history, feedback history, emotional data, and progress data in a database. The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually check their own progress.

[0489] Specific examples

[0490] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[0491] 1. Initial setup and user registration:

[0492] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[0493] The server receives this information and stores it in a database.

[0494] 2. Scenario Selection:

[0495] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[0496] The server loads the data for the selected scenario and prepares the generative AI model.

[0497] 3. Role-playing:

[0498] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[0499] Yamada replies, "I've recently been working on developing a web application."

[0500] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[0501] 4. Use of Emotion Engine:

[0502] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[0503] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[0504] 5. Real-time feedback:

[0505] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[0506] The server will analyze again and provide more specific feedback.

[0507] 6. Progress Tracking and Analysis:

[0508] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[0509] The server generates a progress report based on the stored data and sends it to Yamada.

[0510] Yamada checks the report and visually confirms his own growth.

[0511] Prompt Sentence Examples

[0512] Here are some example prompts to input to a generative AI model:

[0513] For the user "Yamada's" most recent answer: "I recently worked on developing a web app," generate feedback that explains the technology stack and specific role. Also, analyze this answer using the emotion engine to derive emotions (e.g., joy, anger, sadness), and adjust the feedback accordingly.

[0514] By making full use of such a complex approach and providing highly accurate feedback that reflects the user's emotions, it is possible to efficiently support the improvement of communication skills.

[0515] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0516] Step 1:

[0517] When the system starts, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). In this step, the server sets the schema of each table and inserts the initial data. The input is the initial system information, and the output is an initialized database.

[0518] Step 2:

[0519] The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the user's input information to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The input is the user's registration information, and the output is a success message.

[0520] Step 3:

[0521] The server retrieves a list of available scenarios from the database and sends it to the terminal. The terminal displays the received scenario list to the user. The input is a request for a scenario list, and the output is the scenario list.

[0522] Step 4:

[0523] The user selects the scenario of interest from the displayed list of scenarios and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario. The input is the scenario selection information, and the output is a notification that the scenario data has been loaded.

[0524] Step 5:

[0525] The server generates questions for the first phase of the selected scenario and sends them to the user. The terminal displays the questions to the user. The user inputs the answers and sends them to the terminal. The input is the user's answers, and the output is the obtained answer data.

[0526] Step 6:

[0527] The server passes the received answers to the generative AI model, which analyzes them in real time. The input is the user's answer data, and the output is the analysis result.

[0528] Step 7:

[0529] The server generates feedback based on the analysis results and sends it to the terminal, which then displays the generated feedback to the user. The input is the analysis results and the output is the feedback message.

[0530] Step 8:

[0531] The server extracts emotional information from the acquired user response data using an emotion engine. The emotion engine uses voice analysis and text analysis to recognize emotions from the user's response. The input is the user's response data, and the output is emotional information.

[0532] Step 9:

[0533] The server adjusts the feedback content based on the results of emotion analysis and generates optimal feedback for the user. The input is emotion information and initial feedback, and the output is the adjusted feedback.

[0534] Step 10:

[0535] The server then sends the generated feedback back to the device, which displays it to the user. The user then inputs a new answer based on the feedback and sends it back to the device. The input is the adjusted feedback, and the output is the new answer.

[0536] Step 11:

[0537] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. The input is the session data, and the output is the stored data.

[0538] Step 12:

[0539] The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually confirm their own growth. The input is the progress data, and the output is the visualized report.

[0540] (Application example 2)

[0541] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0542] Conventional customer service training systems do not provide sufficient feedback to improve users' communication skills, especially when it comes to providing feedback that takes into account the user's emotional information. As a result, the effectiveness of training is limited, and it is difficult to improve skills that are suited to real customer service situations.

[0543] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0544] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on analyzed emotion information. This makes it possible to provide more accurate feedback that reflects the user's emotion information, thereby enabling the improvement of effective communication skills suited to real customer service situations.

[0545] (User registration means) is a function for registering new users in the system and saving their information in the database.

[0546] (Scenario data) is a collection of data that defines virtual conversation situations for users to use in training.

[0547] (Generation means) is a function that automatically generates specific conversation scenarios and questions based on scenario data selected by the user.

[0548] (Acquisition means) is a function for collecting response data entered by users and processing the data within the system.

[0549] (Analysis means) is a function that analyzes the acquired response data and generates feedback in real time.

[0550] (Providing means) is a function that presents the generated feedback to the user and enables the user to make a new response based on the feedback.

[0551] "Progress Data" is a collection of information that indicates the user's response history, feedback history, and other relevant data obtained throughout the training.

[0552] (Emotion analysis means) is a function for extracting emotional information from the user's response data and reflecting that information in the feedback.

[0553] A system that realizes this invention is a system that includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on the analyzed emotion information.

[0554] Hardware and software used

[0555] Hardware:

[0556] head-mounted display

[0557] Smartphone

[0558] software:

[0559] Python

[0560] TextBlob (emotional analysis)

[0561] TfidfVectorizer (text analysis)

[0562] Logistic Regression (generative AI model)

[0563] Processing Description

[0564] When the system starts up, the server initializes the database and creates the necessary tables such as the user table, scenario table, log table, emotion data table, etc. When a user accesses the new registration screen, enters information such as a username, email address, and password, and presses the register button, the server receives this information and saves it in the database.

[0565] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server, which then prepares a generative AI model based on the selected scenario.

[0566] The server generates the first question for the selected scenario and sends it to the user. The user enters the answer, and the device sends the input to the server. The server passes the answer to the generative AI model, analyzes it in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[0567] The server uses an emotion engine to extract emotion information from the acquired user response data. The emotion engine uses TextBlob to recognize emotions from the user's responses. The recognized emotion information is provided to the analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[0568] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress, creates a visualized report, and provides it to the user.

[0569] Specific examples

[0570] Here is a concrete example of how a new customer service representative at a brick-and-mortar store uses the training system.

[0571] 1. Initial setup and user registration:

[0572] The customer service representative accesses the new registration screen and enters the required information (user name, email address, password).

[0573] The server receives this information and stores it in a database.

[0574] 2. Scenario Selection:

[0575] A customer service representative accesses the dashboard and selects the "First Time Customer Scenario."

[0576] The server loads the data for the selected scenario and prepares the generative AI model.

[0577] 3. Role-playing:

[0578] The server generates the first question, "Welcome. What item would you like to purchase today?" and sends it to the customer service representative.

[0579] The customer service representative replies, "I've recently been working on developing a web application."

[0580] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[0581] 4. Use of Emotion Engine:

[0582] The server analyzes the customer service representative's responses using an emotion engine and extracts emotional information.

[0583] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[0584] 5. Real-time feedback:

[0585] Based on the feedback, the customer service representative answers again, "I used React as the front end and Node.js as the back end."

[0586] The server will analyze again and provide more specific feedback.

[0587] 6. Progress Tracking and Analysis:

[0588] After the session ends, the server stores the agent's response history, feedback history, and progress data.

[0589] The server generates a progress report based on the stored data and sends it to the customer service representative.

[0590] Customer service representatives review the reports and visually see their own progress.

[0591] An example of a specific prompt is, "Would you like to train a scenario where you meet a customer for the first time? Please choose a specific situation."

[0592] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0593] Step 1:

[0594] User Registration

[0595] The user accesses the new registration screen, enters information such as a username, email address, and password, and clicks the Register button.

[0596] Input: Information entered by the user, such as username, email address, or password.

[0597] Output: The user information is saved in the database and the user is recorded as a new user.

[0598] What happens: The server takes this information and executes a SQL query to store it in a database.

[0599] Step 2:

[0600] Scenario Selection

[0601] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[0602] Input: Scenario data stored in the database.

[0603] Output: The retrieved scenario list is displayed on the user's terminal.

[0604] Specific operation: The server retrieves scenario data from the scenario table using a SELECT query and sends it to the terminal in JSON format.

[0605] Step 3:

[0606] Scenario selection and generation

[0607] The user selects a scenario of interest from the displayed list of scenarios and transmits the selected information to the server.

[0608] Input: Scenario information selected by the user.

[0609] Output: A conversation scenario generated based on the selected scenario.

[0610] Specific operation: The server receives the selected scenario information, prepares the generative AI model, and generates an initial conversation scenario.

[0611] Step 4:

[0612] Acquiring response data

[0613] The server generates the first question for the selected scenario and sends it to the user.

[0614] Input: Initial question data based on the scenario.

[0615] Output: The question displayed on the user's terminal.

[0616] Specific operation: The server uses the generative AI model to generate a question and sends it to the user's device.

[0617] Step 5:

[0618] Enter and submit response data

[0619] The user enters an answer to the question, and the terminal transmits the answer to the server.

[0620] Input: The answer data entered by the user.

[0621] Output: The response data sent to the server.

[0622] Specific operation: When the user enters an answer and presses the send button, the device sends the answer data to the server using the POST method.

[0623] Step 6:

[0624] Analysis of response data

[0625] The server passes the response data to a generative AI model, which analyzes it and generates feedback in real time.

[0626] Input: The answer data submitted by the user.

[0627] Output: The generated feedback data.

[0628] Specific operation: The server inputs the response data into the generative AI model, performs text analysis and sentiment analysis, and generates feedback.

[0629] Step 7:

[0630] Sentiment analysis and feedback adjustment

[0631] The server analyzes the acquired response data using an emotion engine to extract emotional information, and adjusts the feedback accordingly.

[0632] Input: User response data and sentiment engine analysis results.

[0633] Output: Feedback tailored based on emotional information.

[0634] Specific operation: The server uses TextBlob to perform sentiment analysis and reflects the sentiment information in the feedback.

[0635] Step 8:

[0636] Providing feedback

[0637] The server transmits the generated feedback back to the user's terminal and displays it to the user.

[0638] Input: Calibrated feedback data.

[0639] Output: Feedback displayed on the user's device.

[0640] Specific behavior: The server sends feedback data in JSON format to the device, and the device displays it.

[0641] Step 9:

[0642] Save and analyze progress data

[0643] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database, and generates a progress report based on this data.

[0644] Input: User answer history, feedback history, sentiment data, and progress data.

[0645] Output: A progress report for the user.

[0646] Specific operation: The server stores the data in a database, performs data analysis based on the stored data, and generates a visualized report.

[0647] Specific prompt examples

[0648] "Do you want to train a scenario for a first-time customer encounter? Pick a specific situation."

[0649] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0650] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0651] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0652] [Second embodiment]

[0653] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0654] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0655] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0656] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0657] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0658] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0659] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0660] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0661] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0662] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0663] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0664] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0665] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system supports continuous development by saving and analyzing users' progress.

[0666] Program processing overview

[0667] Initial Setup and User Registration

[0668] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[0669] Scenario Selection

[0670] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[0671] Role-playing

[0672] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to a generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[0673] Real-time feedback

[0674] For each answer, the server analyzes it and generates feedback that is immediately sent to the user, allowing them to refine and submit their answer again.

[0675] Progress Tracking and Analysis

[0676] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[0677] Specific examples

[0678] Here, a specific example is shown in which a user "Sato-san" uses the system to carry out a technical interview scenario.

[0679] 1. Initial setup and user registration:

[0680] Mr. Sato accesses the new registration screen and enters the required information (user name, email address, password).

[0681] The server receives this information and stores it in a database.

[0682] 2. Scenario Selection:

[0683] Sato accesses the dashboard and selects "Technical Interview Scenario."

[0684] The server loads the data for the selected scenario and prepares the generative AI model.

[0685] 3. Role-playing:

[0686] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Sato.

[0687] Sato replies, "I've recently been working on developing a web application."

[0688] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[0689] 4. Real-time feedback:

[0690] Based on the feedback, Sato answers again, "I used React as the front end and Node.js as the back end."

[0691] The server will analyze again and provide more specific feedback.

[0692] 5. Progress Tracking and Analysis:

[0693] After the session ends, the server stores Sato's answer history, feedback history, and progress data.

[0694] The server generates a progress report based on the stored data and sends it to Mr. Sato.

[0695] Sato checks the report and visually confirms his own growth.

[0696] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and solves the problem of improving communication abilities in a remote environment.

[0697] The processing flow will be explained below.

[0698] Specific explanation of program processing

[0699] Initial Setup and User Registration

[0700] Step 1:

[0701] The server establishes a database connection when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[0702] Step 2:

[0703] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[0704] Step 3:

[0705] After the user has completed the input, he clicks the Register button.

[0706] Step 4:

[0707] The terminal transmits the input information to the server.

[0708] Step 5:

[0709] The server receives the information, checks it for format and duplication, and stores it in a database.

[0710] Scenario Selection

[0711] Step 6:

[0712] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[0713] Step 7:

[0714] The terminal displays a list of scenarios to the user.

[0715] Step 8:

[0716] The user selects the scenario of interest and clicks the Select button.

[0717] Step 9:

[0718] The terminal transmits the scenario information selected by the user to the server.

[0719] Step 10:

[0720] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[0721] Role-playing

[0722] Step 11:

[0723] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[0724] Step 12:

[0725] The terminal displays the first question to the user.

[0726] Step 13:

[0727] The user answers the questions by typing or speaking.

[0728] Step 14:

[0729] The terminal transmits the inputted answer to the server.

[0730] Step 15:

[0731] The server passes the answer to a generative AI model, which analyzes it in real time.

[0732] Step 16:

[0733] The server generates appropriate feedback based on the analysis results.

[0734] Step 17:

[0735] The server transmits the generated feedback to the terminal.

[0736] Step 18:

[0737] The device displays feedback to the user.

[0738] Real-time feedback

[0739] Step 19:

[0740] The user improves their answer based on the feedback and re-enters it.

[0741] Step 20:

[0742] The device sends the improved answer to the server.

[0743] Step 21:

[0744] The server then passes the answer back to the generative AI model for analysis.

[0745] Step 22:

[0746] The server generates new feedback and sends it back to the device.

[0747] Step 23:

[0748] The device displays new feedback to the user.

[0749] Progress Tracking and Analysis

[0750] Step 24:

[0751] The server stores the user's answer history, feedback history, and progress data in a database after each session.

[0752] Step 25:

[0753] The server analyzes the user's progress based on the stored data and generates a visual report.

[0754] Step 26:

[0755] The server sends the generated progress report to the terminal.

[0756] Step 27:

[0757] The terminal displays a progress report to the user.

[0758] As described above, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills.

[0759] Example 1

[0760] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0761] Conventional communication skill improvement systems have difficulty providing real-time feedback to users' responses, preventing them from immediately improving their communication skills. Furthermore, they lack the ability to store and analyze progress data, preventing them from effectively supporting users' continuous improvement.

[0762] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0763] In this invention, the server includes user registration means, means for displaying multiple pieces of scenario information, generation means for generating a dialogue scenario based on scenario information selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, transmission means for the user's terminal to transmit input response data to the server, transmission means for the server to transmit the generated feedback to the user's terminal, and display means for the terminal to receive the data and display it to the user. This allows the user to receive instant feedback and continuously improve their communication skills.

[0764] The "user registration means" is a means by which a user inputs information for new registration in the system and stores that information in the database.

[0765] "Scenario information" is information relating to a number of dialogue scenarios that the user can select from, and role-playing is carried out based on this information.

[0766] The "generation means" is a means for generating a dialogue scenario and a prompt sentence based on scenario information selected by the user.

[0767] The "acquisition means" is a means by which the system acquires response data from the user.

[0768] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time based on the analysis results.

[0769] The "means for providing" is a means for transmitting the generated feedback to the user's terminal and displaying it.

[0770] "Progress Data" refers to data including a user's response history, feedback history, progress status, etc., and is used to evaluate a user's growth and improvement.

[0771] The "transmission means" is a means by which the user's terminal transmits response data to the server, and the server transmits the generated feedback to the user's terminal.

[0772] The "display means" is a means for visually displaying to the user the feedback and scenario information received by the user's terminal.

[0773] A "generative AI model" is an artificial intelligence model used to analyze user response data and generate appropriate feedback or the next prompt.

[0774] A "prompt sentence" is dialogue text that includes a question or instruction for the user to follow next.

[0775] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from a variety of scenarios, role-play based on the scenarios, and receive real-time feedback. The system also supports continuous improvement by saving and analyzing the user's progress.

[0776] This system is implemented primarily using the following hardware and software:

[0777] Hardware: Servers, user devices (PCs, smartphones, tablets)

[0778] Software: Database, generative AI model (e.g., GPT-3), front-end framework (e.g., React), back-end framework (e.g., Node.js)

[0779] When the system starts up, the server initializes the database and creates the necessary tables, such as the user table, scenario table, and log table. This allows users to register with the system and prepare it for use. When a user accesses the new registration screen and enters information such as their username, email address, and password, the terminal sends this information to the server. The server receives the data and stores it in the database.

[0780] The server then retrieves available scenario information from the database and sends it to the user's device, allowing the user to select the scenario they are interested in from the displayed list of scenarios. Once the user selects a scenario, the selection information is sent to the server, which then prepares a generative AI model based on the selected scenario.

[0781] When role-playing begins, the server generates questions for the first phase of the selected scenario and sends them to the user. For example, in a technical interview scenario, a question such as "Tell me about a project you recently worked on" is generated. The user enters answers to the questions, and the answer data is sent to the server via the terminal.

[0782] The server passes the received answers to the generative AI model, which analyzes them in real time. The generated feedback is then sent back to the user's device and displayed to the user. Depending on the feedback, the user can refine their answer and send it back to the server. By repeating this process, the user's communication skills can be improved.

[0783] After the session ends, the server saves the user's answer history, feedback history, and progress data. Based on this data, the server analyzes the user's progress and generates a visual report. The report is sent to the user's device, allowing the user to visually check their progress.

[0784] For example, if user "Sato" uses the technical interview scenario,

[0785] 1. Initial setup and user registration:

[0786] Mr. Sato accesses the new registration screen and enters his username, email address, and password.

[0787] The server receives this information and stores it in a database.

[0788] 2. Scenario Selection:

[0789] Sato accesses the dashboard and selects the technical interview scenario.

[0790] The server loads the data for the selected scenario and prepares the generative AI model.

[0791] 3. Role-playing:

[0792] The server generates a question such as "Tell me about a project you've worked on recently" and sends it to Sato.

[0793] Mr. Sato enters the answer and sends it to the server.

[0794] The server analyzes the answers using a generative AI model, generating feedback such as, "Please explain the specific technology stack and role."

[0795] 4. Real-time feedback:

[0796] Sato refines the answer based on the feedback and submits it to the server again.

[0797] The server analyzes the refined answer and provides more specific feedback.

[0798] 5. Progress Tracking and Analysis:

[0799] After the session ends, the server stores Mr. Sato's answer history, feedback history, and progress data in a database.

[0800] A progress report is generated based on the saved data and sent to Sato.

[0801] Sato checks the report and visually confirms his own growth.

[0802] Examples of prompts include:

[0803] "Tell me about a project you've worked on recently."

[0804] "Please explain your specific role in the project and the technology you used."

[0805] "Tell us about a challenge you faced in the project and how you solved it."

[0806] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and aims to improve communication abilities in remote environments.

[0807] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0808] Program flow:

[0809] Step 1: Initialize the database and set up the structure

[0810] The server initializes the database when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[0811] Input: System startup event

[0812] Data manipulation: Creating tables using SQL queries

[0813] Output: Initialized database

[0814] Specific behavior:

[0815] Generate each table with the CREATE TABLE statement

[0816] Insert initial data into the database using INSERT statements as needed

[0817] Step 2: Receiving and storing user registration information

[0818] The user accesses the new registration screen, enters their username, email address, and password, and presses the registration button. The device then sends this data to the server.

[0819] Input: User registration information (user name, email address, password)

[0820] Data processing: data validation and encoding

[0821] Output: User information stored in the database

[0822] Specific behavior:

[0823] The front-end collects form data and sends it to the server via API

[0824] The backend performs input validation (e.g., email address format check)

[0825] The password is hashed and saved in the database using an INSERT statement.

[0826] Step 3: Get and display the scenario list

[0827] The server retrieves available scenario information from the database and sends it to the user's terminal, which then displays the received scenario information.

[0828] Input: Request to get a scenario list

[0829] Data processing: Retrieving scenario information from the database

[0830] Output: Scenario information sent to the user's device

[0831] Specific behavior:

[0832] Retrieve a list of scenarios from the database using a SELECT statement

[0833] The acquired data is encoded in JSON format and sent to the terminal via API.

[0834] The front end displays the scenario list in list format.

[0835] Step 4: Select and submit a scenario

[0836] The user selects the scenario of interest and sends the selection to the server, which then prepares a generative AI model based on the selected scenario.

[0837] Input: ID of the selected scenario

[0838] Data processing: Loading AI models based on scenario ID

[0839] Output: A prepared generative AI model

[0840] Specific behavior:

[0841] The front end gets the ID of the selected scenario and sends it to the server

[0842] The server loads the generative AI model based on the scenario ID and prepares the next prompt.

[0843] Step 5: Generate and submit your question

[0844] The server generates the questions for the first phase of the selected scenario and sends them to the user.

[0845] Input: Data for the selected scenario

[0846] Data processing: prompt sentence generation

[0847] Output: The question sent to the user's device

[0848] Specific behavior:

[0849] Executes dialog generation logic based on scenarios

[0850] Generate an initial question, encode it in JSON format, and send it to the terminal.

[0851] The front end displays the question to the user

[0852] Step 6: User answers and submits

[0853] The user inputs an answer to the question, and the answer data is sent to the server via the terminal.

[0854] Input: User response data

[0855] Data processing: Validation and encoding of response data

[0856] Output: Response data sent to the server

[0857] Specific behavior:

[0858] The front end collects the answers entered by the user

[0859] Response data is sent to the server via API

[0860] Step 7: Analyze responses and generate feedback

[0861] The server passes the received answers to a generative AI model for real-time analysis, and the generated feedback is sent to the user's device.

[0862] Input: User response data

[0863] Data processing: Analysis with generative AI models

[0864] Output: Generated feedback

[0865] Specific behavior:

[0866] Provide answer data as input to the generative AI model

[0867] Obtain analysis results and format them as feedback

[0868] Feedback is encoded in JSON format and sent to the user's device

[0869] The front end displays feedback to the user

[0870] Step 8: Re-enter and submit your improved answers

[0871] The user refines the answer based on the feedback and submits it to the server again.

[0872] Input: Improved response data

[0873] Data processing: Revalidation and encoding

[0874] Output: Improved answer sent to the server

[0875] Specific behavior:

[0876] The user refines the answer and enters it again

[0877] Response data is sent to the server again

[0878] Step 9: Save your data

[0879] After the session ends, the server stores the user's answer history, feedback history, and progress data.

[0880] Input: User session data (answers, feedback)

[0881] Data processing: structuring data

[0882] Output: Session data stored in the database

[0883] Specific behavior:

[0884] Save answer history and feedback history to the log table using INSERT statements

[0885] Step 10: Analyze data and generate reports

[0886] The server analyzes the user's progress based on the stored data and creates a visual report, which is then sent to the user's device for display.

[0887] Input: Saved session data

[0888] Data processing: data analysis and visualization

[0889] Output: Progress report sent to the user's terminal

[0890] Specific behavior:

[0891] Analyzes stored data and generates progress reports

[0892] Use a data visualization library (e.g., D3.js) to generate graphs, etc.

[0893] Encode the report in JSON format and send it to the user's device

[0894] The front end displays the report to the user

[0895] In this way, the system provides a series of processes for users to interactively improve their communication skills.

[0896] (Application example 1)

[0897] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0898] In today's virtual stores, salespeople's communication skills have a significant impact on the quality of the customer experience. However, traditional training methods make it difficult to provide real-time feedback and encourage continuous skill improvement. Furthermore, due to a lack of mechanisms for accurately tracking and analyzing progress, it is difficult to efficiently support salespeople in improving their skills.

[0899] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0900] In this invention, the server includes user registration means, means for displaying multiple scenario data, generation means for generating a conversation scenario based on scenario data selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, and means for evaluating the sales skills of salespeople in the virtual store and providing scenarios for training them. This allows salespeople to receive feedback in real time while undergoing training, enabling continuous skill improvement.

[0901] The "user registration means" is a function for registering a new user in the system, and is usually a means for inputting information such as a user name, email address, and password, and storing the information in a database.

[0902] "Means for displaying multiple scenario data" is a function that displays a list of various scenarios that the user can select on the screen, and is a means that allows the user to select which specific scenario to use.

[0903] The "means for generating a conversation scenario" is a function for automatically generating a conversation in accordance with a scenario selected by a user, based on the scenario data.

[0904] The "means for acquiring answer data from the user" is a function for the system to receive answers input by the user to the conversation scenario and store them as data.

[0905] The "analysis means for generating feedback in real time" is a function for analyzing user response data and automatically generating appropriate feedback immediately.

[0906] The "means for providing the generated feedback to the user" is a function for transmitting the generated feedback to the user's terminal and displaying it on the screen.

[0907] The "means for saving and analyzing user progress data" is a function for evaluating the improvement of a user's skills by saving the user's past response data and feedback content and analyzing them as appropriate.

[0908] "Means for providing scenarios for evaluating and training salespeople's response skills in a virtual store" refers to a function for providing scenarios designed to improve salespeople's response skills in a virtual store environment and for conducting training through those scenarios.

[0909] The present invention is a system for improving a user's communication ability, and is particularly designed to evaluate and train salespeople's communication skills in a virtual store. The components and processes of this system are as follows:

[0910] System Components

[0911] 1. User registration method

[0912] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[0913] 2. A way to display multiple scenario data

[0914] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[0915] 3. Conversation scenario generation method

[0916] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[0917] 4. How to obtain response data from users

[0918] The server receives the answer data entered by the user when answering the scenario and stores it in a database.

[0919] 5. Analytics that generate real-time feedback

[0920] The server analyzes each answer using a generative AI model and generates feedback, which is immediately provided to the user, allowing them to refine their answer and submit it again.

[0921] 6. Means for providing generated feedback to users

[0922] The generated feedback is immediately sent to the user's device and displayed to the user, allowing the user to receive feedback in real time.

[0923] 7. A means of storing and analyzing user progress data

[0924] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report that is provided to the user. This report allows the user to visually confirm their own progress.

[0925] 8. A means of providing scenarios to assess and train sales associate skills in a virtual store

[0926] The server provides scenarios in a virtual store environment to improve sales staff skills, including new product introductions, complaint handling, and cross-selling, allowing for training that is tailored to actual sales situations.

[0927] Hardware and software used

[0928] This system uses the following hardware and software:

[0929] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[0930] Software: TensorFlow (library underlying generative AI models), MySQL (database management system), Node.js (server-side programming)

[0931] Specific examples

[0932] Scenario where a user is introducing a new product in a virtual store:

[0933] 1. User Registration

[0934] The user accesses the new registration screen and enters the required information (user name, email address, password). The server receives this information and stores it in the database.

[0935] 2. Scenario Selection

[0936] The user accesses the dashboard and selects the "New Product Introduction Scenario." The server loads the data for the selected scenario and prepares the generative AI model.

[0937] 3. Role-playing

[0938] The server generates the first question, "What are the main features of this product?" and sends it to the user. The user answers, "This product has a high-performance battery and fast charging capabilities." The server then runs this answer through a generative AI model, which generates feedback such as, "Please be specific about battery life and charging time."

[0939] Example prompt sentence:

[0940] "If a customer asks about the features of a new product, how would you explain it?"

[0941] The system allows salespeople to receive real-time feedback and effectively improve their skills in the virtual store.

[0942] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0943] Step 1:

[0944] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). This ensures that data for user registration and scenario selection is correctly saved and managed.

[0945] Input: System boot

[0946] Output: Initialize the database and create tables.

[0947] What happens next: The server connects to the MySQL database and creates the necessary tables using the CREATE statement.

[0948] Step 2:

[0949] The user accesses the new registration screen, enters information such as a user name, email address, and password, and presses the registration button.

[0950] Input: Username, Email Address, Password

[0951] Output: Save registration information to database

[0952] Specific operation: The information entered by the user is sent to the server, and the server saves the information in the database using the INSERT statement.

[0953] Step 3:

[0954] The server retrieves a list of available scenarios from the database and sends it to the user's terminal.

[0955] Input: Request scenario list

[0956] Output: Send scenario list

[0957] Specific operation: The server uses a SELECT statement to retrieve scenario data from the database and sends the retrieved data to the user's device in JSON format.

[0958] Step 4:

[0959] The user selects a scenario of interest from the displayed list of scenarios and transmits the selected information to the server.

[0960] Input: Scenario selection information

[0961] Output: Preparation for selected scenarios

[0962] Specific operation: After receiving the user's selection information, the server retrieves detailed data on the selected scenario from the database and prepares the generative AI model.

[0963] Step 5:

[0964] The server generates questions for the first phase of the selected scenario and sends them to the user, who then inputs the answers, which the device then sends to the server.

[0965] Input: Scenario question generation request, user answer

[0966] Output: Send question, get answer

[0967] Specific operation: The server uses the generative AI model to generate questions based on the scenario and sends them to the user's device. When the user enters an answer, the answer is sent to the server.

[0968] Step 6:

[0969] The server passes the answer to the generative AI model, which analyzes it in real time and generates feedback, which is then sent back to the user's device and displayed to them.

[0970] Input: User response data

[0971] Output: Generated feedback

[0972] Specific operation: The server inputs the answer data into the generative AI model, obtains feedback as the model's analysis result, and sends the feedback to the user's device and displays it to the user.

[0973] Step 7:

[0974] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database.

[0975] Input: Session data

[0976] Output: Saved progress data

[0977] Specific behavior: The server stores the user's answers and feedback data in a database using INSERT or UPDATE statements, for future reference in sessions and for progress analysis.

[0978] Step 8:

[0979] The server analyzes the user's progress based on the stored data, creates a visualized report, and provides it to the user.

[0980] Input: Saved progress data

[0981] Output: Generate and send a progress report

[0982] Specific operation: The server analyzes past response data and feedback, generates a report visualizing the user's growth and areas for improvement, and sends it to the user's device.

[0983] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0984] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[0985] Program processing overview

[0986] Initial Setup and User Registration

[0987] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The server receives this information and saves it in the database.

[0988] Scenario Selection

[0989] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[0990] Role-playing

[0991] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[0992] Use of emotion engine

[0993] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to an analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[0994] Real-time feedback

[0995] For each answer, the server performs analysis and emotion recognition and generates feedback that is instantly sent to the user, allowing them to improve their answer and submit it again.

[0996] Progress Tracking and Analysis

[0997] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[0998] Specific examples

[0999] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[1000] 1. Initial setup and user registration:

[1001] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[1002] The server receives this information and stores it in a database.

[1003] 2. Scenario Selection:

[1004] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[1005] The server loads the data for the selected scenario and prepares the generative AI model.

[1006] 3. Role-playing:

[1007] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[1008] Yamada replies, "I've recently been working on developing a web application."

[1009] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1010] 4. Use of Emotion Engine:

[1011] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[1012] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[1013] 5. Real-time feedback:

[1014] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[1015] The server will analyze again and provide more specific feedback.

[1016] 6. Progress Tracking and Analysis:

[1017] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[1018] The server generates a progress report based on the stored data and sends it to Yamada.

[1019] Yamada checks the report and visually confirms his own growth.

[1020] In this way, by combining emotion engines, it becomes possible to provide more accurate feedback according to the user's emotions, and to more effectively support the improvement of communication skills.

[1021] The processing flow will be explained below.

[1022] Specific explanation of program processing (including emotion engine)

[1023] Initial Setup and User Registration

[1024] Step 1:

[1025] When the system starts up, the server establishes a database connection and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.).

[1026] Step 2:

[1027] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[1028] Step 3:

[1029] After the user has completed the input, he clicks the Register button.

[1030] Step 4:

[1031] The terminal transmits the input information to the server.

[1032] Step 5:

[1033] The server receives the information, checks it for format and duplication, and stores it in a database.

[1034] Scenario Selection

[1035] Step 6:

[1036] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[1037] Step 7:

[1038] The terminal displays a list of scenarios to the user.

[1039] Step 8:

[1040] The user selects the scenario of interest and clicks the Select button.

[1041] Step 9:

[1042] The terminal transmits the scenario information selected by the user to the server.

[1043] Step 10:

[1044] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[1045] Role-playing

[1046] Step 11:

[1047] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[1048] Step 12:

[1049] The terminal displays the first question to the user.

[1050] Step 13:

[1051] The user answers the questions by typing or speaking.

[1052] Step 14:

[1053] The terminal transmits the inputted answer to the server.

[1054] Step 15:

[1055] The server passes the answer to a generative AI model, which analyzes it in real time.

[1056] Use of emotion engine

[1057] Step 16:

[1058] The server extracts emotion information from the acquired user response data using an emotion engine.

[1059] Step 17:

[1060] The emotion engine uses speech and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's responses.

[1061] Step 18:

[1062] The emotion engine provides the recognized emotion information to the server.

[1063] Real-time feedback

[1064] Step 19:

[1065] The server generates appropriate feedback based on the analysis results, including emotional information.

[1066] Step 20:

[1067] The server transmits the generated feedback to the terminal.

[1068] Step 21:

[1069] The device displays feedback to the user.

[1070] Step 22:

[1071] The user improves their answer based on the feedback and re-enters it.

[1072] Step 23:

[1073] The device sends the improved answer to the server.

[1074] Step 24:

[1075] The server then passes the answer back to the generative AI model for analysis.

[1076] Step 25:

[1077] The server generates new feedback and sends it back to the device.

[1078] Step 26:

[1079] The device displays new feedback to the user.

[1080] Progress Tracking and Analysis

[1081] Step 27:

[1082] After each session, the server stores the user's answer history, feedback history, emotion data, and progress data in a database.

[1083] Step 28:

[1084] The server analyzes the user's progress based on the stored data and generates a visual report.

[1085] Step 29:

[1086] The server sends the generated progress report to the terminal.

[1087] Step 30:

[1088] The terminal displays a progress report to the user.

[1089] Through these steps, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills. By utilizing an emotion engine, the system provides feedback based on the user's emotions, providing a more effective learning experience.

[1090] Example 2

[1091] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1092] In conventional communication improvement systems, feedback to users' responses was uniform and could not be adapted to the user's emotions. As a result, the feedback users received was not optimized for individual situations, making it difficult to improve their communication skills efficiently.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1094] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring answer data from the user, an analysis means for analyzing the acquired answer data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, a means for performing emotion analysis on the acquired answer data, and a means for reflecting the results of the emotion analysis in the feedback. This makes it possible to provide individualized feedback according to the user's emotions and more effectively improve communication skills.

[1095] "User registration means" refers to the means by which a user accesses the system, inputs his / her own information, and performs registration.

[1096] The "means for displaying a plurality of scenario data" is a means for visually displaying a plurality of scenarios provided by the system to the user.

[1097] The "generation means" is a means for generating a conversation scenario and questions based on scenario data selected by the user.

[1098] The "acquisition means" is a means for collecting response data from users and transmitting it to the system.

[1099] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time.

[1100] The "means for providing" is a means for providing and displaying the generated feedback to the user.

[1101] The "means for saving and analyzing progress data" refers to a means for saving a user's response history and feedback history in a database and analyzing progress based on that history.

[1102] The "means for performing emotion analysis" is a means for analyzing the emotional information contained in the acquired response data and recognizing specific emotions (joy, anger, sadness, etc.).

[1103] The "means for reflecting the results of emotion analysis in the feedback" refers to a means for adjusting the feedback content based on the results of emotion analysis and providing optimal feedback to the user.

[1104] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[1105] Initial Setup and User Registration

[1106] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the entered information to the server, which then stores the received information in the database. When registration is complete, the server sends a success message to the terminal, which displays it to the user.

[1107] Scenario Selection

[1108] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario and prepares for the next phase.

[1109] Role-playing

[1110] The server generates questions for the first phase of the selected scenario and sends them to the user. The device displays the questions to the user, who then enters answers. The device then sends the entered answers to the server, which passes them to the generative AI model for real-time analysis. The server generates feedback based on the analysis results and sends it to the device. The device then displays the generated feedback to the user.

[1111] Use of emotion engine

[1112] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to the analysis means to improve the accuracy of the feedback. For example, if the user's response indicates a negative emotion, the feedback can use an affiliative expression that corresponds to that emotion.

[1113] Real-time feedback

[1114] For each answer, the server analyzes and recognizes emotions, and generates feedback. The feedback is immediately sent to the user's device, allowing the user to improve their answer and submit it again. This allows users to receive continuous feedback and improve their communication skills.

[1115] Progress Tracking and Analysis

[1116] After the session ends, the server stores the user's answer history, feedback history, emotional data, and progress data in a database. The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually check their own progress.

[1117] Specific examples

[1118] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[1119] 1. Initial setup and user registration:

[1120] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[1121] The server receives this information and stores it in a database.

[1122] 2. Scenario Selection:

[1123] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[1124] The server loads the data for the selected scenario and prepares the generative AI model.

[1125] 3. Role-playing:

[1126] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[1127] Yamada replies, "I've recently been working on developing a web application."

[1128] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1129] 4. Use of Emotion Engine:

[1130] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[1131] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[1132] 5. Real-time feedback:

[1133] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[1134] The server will analyze again and provide more specific feedback.

[1135] 6. Progress Tracking and Analysis:

[1136] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[1137] The server generates a progress report based on the stored data and sends it to Yamada.

[1138] Yamada checks the report and visually confirms his own growth.

[1139] Prompt Sentence Examples

[1140] Here are some example prompts to input to a generative AI model:

[1141] For the user "Yamada's" most recent answer: "I recently worked on developing a web app," generate feedback that explains the technology stack and specific role. Also, analyze this answer using the emotion engine to derive emotions (e.g., joy, anger, sadness), and adjust the feedback accordingly.

[1142] By making full use of such a complex approach and providing highly accurate feedback that reflects the user's emotions, it is possible to efficiently support the improvement of communication skills.

[1143] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1144] Step 1:

[1145] When the system starts, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). In this step, the server sets the schema of each table and inserts the initial data. The input is the initial system information, and the output is an initialized database.

[1146] Step 2:

[1147] The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the user's input information to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The input is the user's registration information, and the output is a success message.

[1148] Step 3:

[1149] The server retrieves a list of available scenarios from the database and sends it to the terminal. The terminal displays the received scenario list to the user. The input is a request for a scenario list, and the output is the scenario list.

[1150] Step 4:

[1151] The user selects the scenario of interest from the displayed list of scenarios and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario. The input is the scenario selection information, and the output is a notification that the scenario data has been loaded.

[1152] Step 5:

[1153] The server generates questions for the first phase of the selected scenario and sends them to the user. The terminal displays the questions to the user. The user inputs the answers and sends them to the terminal. The input is the user's answers, and the output is the obtained answer data.

[1154] Step 6:

[1155] The server passes the received answers to the generative AI model, which analyzes them in real time. The input is the user's answer data, and the output is the analysis result.

[1156] Step 7:

[1157] The server generates feedback based on the analysis results and sends it to the terminal, which then displays the generated feedback to the user. The input is the analysis results and the output is the feedback message.

[1158] Step 8:

[1159] The server extracts emotional information from the acquired user response data using an emotion engine. The emotion engine uses voice analysis and text analysis to recognize emotions from the user's response. The input is the user's response data, and the output is emotional information.

[1160] Step 9:

[1161] The server adjusts the feedback content based on the results of emotion analysis and generates optimal feedback for the user. The input is emotion information and initial feedback, and the output is the adjusted feedback.

[1162] Step 10:

[1163] The server then sends the generated feedback back to the device, which displays it to the user. The user then inputs a new answer based on the feedback and sends it back to the device. The input is the adjusted feedback, and the output is the new answer.

[1164] Step 11:

[1165] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. The input is the session data, and the output is the stored data.

[1166] Step 12:

[1167] The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually confirm their own growth. The input is the progress data, and the output is the visualized report.

[1168] (Application example 2)

[1169] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1170] Conventional customer service training systems do not provide sufficient feedback to improve users' communication skills, especially when it comes to providing feedback that takes into account the user's emotional information. As a result, the effectiveness of training is limited, and it is difficult to improve skills that are suited to real customer service situations.

[1171] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1172] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on analyzed emotion information. This makes it possible to provide more accurate feedback that reflects the user's emotion information, thereby enabling the improvement of effective communication skills suited to real customer service situations.

[1173] (User registration means) is a function for registering new users in the system and saving their information in the database.

[1174] (Scenario data) is a collection of data that defines virtual conversation situations for users to use in training.

[1175] (Generation means) is a function that automatically generates specific conversation scenarios and questions based on scenario data selected by the user.

[1176] (Acquisition means) is a function for collecting response data entered by users and processing the data within the system.

[1177] (Analysis means) is a function that analyzes the acquired response data and generates feedback in real time.

[1178] (Providing means) is a function that presents the generated feedback to the user and enables the user to make a new response based on the feedback.

[1179] "Progress Data" is a collection of information that indicates the user's response history, feedback history, and other relevant data obtained throughout the training.

[1180] (Emotion analysis means) is a function for extracting emotional information from the user's response data and reflecting that information in the feedback.

[1181] A system that realizes this invention is a system that includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on the analyzed emotion information.

[1182] Hardware and software used

[1183] Hardware:

[1184] head-mounted display

[1185] Smartphone

[1186] software:

[1187] Python

[1188] TextBlob (emotional analysis)

[1189] TfidfVectorizer (text analysis)

[1190] Logistic Regression (generative AI model)

[1191] Processing Description

[1192] When the system starts up, the server initializes the database and creates the necessary tables such as the user table, scenario table, log table, emotion data table, etc. When a user accesses the new registration screen, enters information such as a username, email address, and password, and presses the register button, the server receives this information and saves it in the database.

[1193] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server, which then prepares a generative AI model based on the selected scenario.

[1194] The server generates the first question for the selected scenario and sends it to the user. The user enters the answer, and the device sends the input to the server. The server passes the answer to the generative AI model, analyzes it in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[1195] The server uses an emotion engine to extract emotion information from the acquired user response data. The emotion engine uses TextBlob to recognize emotions from the user's responses. The recognized emotion information is provided to the analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[1196] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress, creates a visualized report, and provides it to the user.

[1197] Specific examples

[1198] Here is a concrete example of how a new customer service representative at a brick-and-mortar store uses the training system.

[1199] 1. Initial setup and user registration:

[1200] The customer service representative accesses the new registration screen and enters the required information (user name, email address, password).

[1201] The server receives this information and stores it in a database.

[1202] 2. Scenario Selection:

[1203] A customer service representative accesses the dashboard and selects the "First Time Customer Scenario."

[1204] The server loads the data for the selected scenario and prepares the generative AI model.

[1205] 3. Role-playing:

[1206] The server generates the first question, "Welcome. What item would you like to purchase today?" and sends it to the customer service representative.

[1207] The customer service representative replies, "I've recently been working on developing a web application."

[1208] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1209] 4. Use of Emotion Engine:

[1210] The server analyzes the customer service representative's responses using an emotion engine and extracts emotional information.

[1211] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[1212] 5. Real-time feedback:

[1213] Based on the feedback, the customer service representative answers again, "I used React as the front end and Node.js as the back end."

[1214] The server will analyze again and provide more specific feedback.

[1215] 6. Progress Tracking and Analysis:

[1216] After the session ends, the server stores the agent's response history, feedback history, and progress data.

[1217] The server generates a progress report based on the stored data and sends it to the customer service representative.

[1218] Customer service representatives review the reports and visually see their own progress.

[1219] An example of a specific prompt is, "Would you like to train a scenario where you meet a customer for the first time? Please choose a specific situation."

[1220] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1221] Step 1:

[1222] User Registration

[1223] The user accesses the new registration screen, enters information such as a username, email address, and password, and clicks the Register button.

[1224] Input: Information entered by the user, such as username, email address, or password.

[1225] Output: The user information is saved in the database and the user is recorded as a new user.

[1226] What happens: The server takes this information and executes a SQL query to store it in a database.

[1227] Step 2:

[1228] Scenario Selection

[1229] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[1230] Input: Scenario data stored in the database.

[1231] Output: The retrieved scenario list is displayed on the user's terminal.

[1232] Specific operation: The server retrieves scenario data from the scenario table using a SELECT query and sends it to the terminal in JSON format.

[1233] Step 3:

[1234] Scenario selection and generation

[1235] The user selects a scenario of interest from the displayed list of scenarios and transmits the selected information to the server.

[1236] Input: Scenario information selected by the user.

[1237] Output: A conversation scenario generated based on the selected scenario.

[1238] Specific operation: The server receives the selected scenario information, prepares the generative AI model, and generates an initial conversation scenario.

[1239] Step 4:

[1240] Acquiring response data

[1241] The server generates the first question for the selected scenario and sends it to the user.

[1242] Input: Initial question data based on the scenario.

[1243] Output: The question displayed on the user's terminal.

[1244] Specific operation: The server uses the generative AI model to generate a question and sends it to the user's device.

[1245] Step 5:

[1246] Enter and submit response data

[1247] The user enters an answer to the question, and the terminal transmits the answer to the server.

[1248] Input: The answer data entered by the user.

[1249] Output: The response data sent to the server.

[1250] Specific operation: When the user enters an answer and presses the send button, the device sends the answer data to the server using the POST method.

[1251] Step 6:

[1252] Analysis of response data

[1253] The server passes the response data to a generative AI model, which analyzes it and generates feedback in real time.

[1254] Input: The answer data submitted by the user.

[1255] Output: The generated feedback data.

[1256] Specific operation: The server inputs the response data into the generative AI model, performs text analysis and sentiment analysis, and generates feedback.

[1257] Step 7:

[1258] Sentiment analysis and feedback adjustment

[1259] The server analyzes the acquired response data using an emotion engine to extract emotional information, and adjusts the feedback accordingly.

[1260] Input: User response data and sentiment engine analysis results.

[1261] Output: Feedback tailored based on emotional information.

[1262] Specific operation: The server uses TextBlob to perform sentiment analysis and reflects the sentiment information in the feedback.

[1263] Step 8:

[1264] Providing feedback

[1265] The server transmits the generated feedback back to the user's terminal and displays it to the user.

[1266] Input: Calibrated feedback data.

[1267] Output: Feedback displayed on the user's device.

[1268] Specific behavior: The server sends feedback data in JSON format to the device, and the device displays it.

[1269] Step 9:

[1270] Save and analyze progress data

[1271] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database, and generates a progress report based on this data.

[1272] Input: User answer history, feedback history, sentiment data, and progress data.

[1273] Output: A progress report for the user.

[1274] Specific operation: The server stores the data in a database, performs data analysis based on the stored data, and generates a visualized report.

[1275] Specific prompt examples

[1276] "Do you want to train a scenario for a first-time customer encounter? Pick a specific situation."

[1277] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1278] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1279] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1280] [Third embodiment]

[1281] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1282] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1283] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1284] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1285] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1286] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1287] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1288] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1289] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1290] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1291] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1292] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1293] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system supports continuous development by saving and analyzing users' progress.

[1294] Program processing overview

[1295] Initial Setup and User Registration

[1296] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[1297] Scenario Selection

[1298] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[1299] Role-playing

[1300] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to a generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[1301] Real-time feedback

[1302] For each answer, the server analyzes it and generates feedback that is immediately sent to the user, allowing them to refine and submit their answer again.

[1303] Progress Tracking and Analysis

[1304] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[1305] Specific examples

[1306] Here, a specific example is shown in which a user "Sato-san" uses the system to carry out a technical interview scenario.

[1307] 1. Initial setup and user registration:

[1308] Mr. Sato accesses the new registration screen and enters the required information (user name, email address, password).

[1309] The server receives this information and stores it in a database.

[1310] 2. Scenario Selection:

[1311] Sato accesses the dashboard and selects "Technical Interview Scenario."

[1312] The server loads the data for the selected scenario and prepares the generative AI model.

[1313] 3. Role-playing:

[1314] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Sato.

[1315] Sato replies, "I've recently been working on developing a web application."

[1316] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1317] 4. Real-time feedback:

[1318] Based on the feedback, Sato answers again, "I used React as the front end and Node.js as the back end."

[1319] The server will analyze again and provide more specific feedback.

[1320] 5. Progress Tracking and Analysis:

[1321] After the session ends, the server stores Sato's answer history, feedback history, and progress data.

[1322] The server generates a progress report based on the stored data and sends it to Mr. Sato.

[1323] Sato checks the report and visually confirms his own growth.

[1324] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and solves the problem of improving communication abilities in a remote environment.

[1325] The processing flow will be explained below.

[1326] Specific explanation of program processing

[1327] Initial Setup and User Registration

[1328] Step 1:

[1329] The server establishes a database connection when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[1330] Step 2:

[1331] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[1332] Step 3:

[1333] After the user has completed the input, he clicks the Register button.

[1334] Step 4:

[1335] The terminal transmits the input information to the server.

[1336] Step 5:

[1337] The server receives the information, checks it for format and duplication, and stores it in a database.

[1338] Scenario Selection

[1339] Step 6:

[1340] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[1341] Step 7:

[1342] The terminal displays a list of scenarios to the user.

[1343] Step 8:

[1344] The user selects the scenario of interest and clicks the Select button.

[1345] Step 9:

[1346] The terminal transmits the scenario information selected by the user to the server.

[1347] Step 10:

[1348] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[1349] Role-playing

[1350] Step 11:

[1351] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[1352] Step 12:

[1353] The terminal displays the first question to the user.

[1354] Step 13:

[1355] The user answers the questions by typing or speaking.

[1356] Step 14:

[1357] The terminal transmits the inputted answer to the server.

[1358] Step 15:

[1359] The server passes the answer to a generative AI model, which analyzes it in real time.

[1360] Step 16:

[1361] The server generates appropriate feedback based on the analysis results.

[1362] Step 17:

[1363] The server transmits the generated feedback to the terminal.

[1364] Step 18:

[1365] The device displays feedback to the user.

[1366] Real-time feedback

[1367] Step 19:

[1368] The user improves their answer based on the feedback and re-enters it.

[1369] Step 20:

[1370] The device sends the improved answer to the server.

[1371] Step 21:

[1372] The server then passes the answer back to the generative AI model for analysis.

[1373] Step 22:

[1374] The server generates new feedback and sends it back to the device.

[1375] Step 23:

[1376] The device displays new feedback to the user.

[1377] Progress Tracking and Analysis

[1378] Step 24:

[1379] The server stores the user's answer history, feedback history, and progress data in a database after each session.

[1380] Step 25:

[1381] The server analyzes the user's progress based on the stored data and generates a visual report.

[1382] Step 26:

[1383] The server sends the generated progress report to the terminal.

[1384] Step 27:

[1385] The terminal displays a progress report to the user.

[1386] As described above, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills.

[1387] Example 1

[1388] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1389] Conventional communication skill improvement systems have difficulty providing real-time feedback to users' responses, preventing them from immediately improving their communication skills. Furthermore, they lack the ability to store and analyze progress data, preventing them from effectively supporting users' continuous improvement.

[1390] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1391] In this invention, the server includes user registration means, means for displaying multiple pieces of scenario information, generation means for generating a dialogue scenario based on scenario information selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, transmission means for the user's terminal to transmit input response data to the server, transmission means for the server to transmit the generated feedback to the user's terminal, and display means for the terminal to receive the data and display it to the user. This allows the user to receive instant feedback and continuously improve their communication skills.

[1392] The "user registration means" is a means by which a user inputs information for new registration in the system and stores that information in the database.

[1393] "Scenario information" is information relating to a number of dialogue scenarios that the user can select from, and role-playing is carried out based on this information.

[1394] The "generation means" is a means for generating a dialogue scenario and a prompt sentence based on scenario information selected by the user.

[1395] The "acquisition means" is a means by which the system acquires response data from the user.

[1396] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time based on the analysis results.

[1397] The "means for providing" is a means for transmitting the generated feedback to the user's terminal and displaying it.

[1398] "Progress Data" refers to data including a user's response history, feedback history, progress status, etc., and is used to evaluate a user's growth and improvement.

[1399] The "transmission means" is a means by which the user's terminal transmits response data to the server, and the server transmits the generated feedback to the user's terminal.

[1400] The "display means" is a means for visually displaying to the user the feedback and scenario information received by the user's terminal.

[1401] A "generative AI model" is an artificial intelligence model used to analyze user response data and generate appropriate feedback or the next prompt.

[1402] A "prompt sentence" is dialogue text that includes a question or instruction for the user to follow next.

[1403] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from a variety of scenarios, role-play based on the scenarios, and receive real-time feedback. The system also supports continuous improvement by saving and analyzing the user's progress.

[1404] This system is implemented primarily using the following hardware and software:

[1405] Hardware: Servers, user devices (PCs, smartphones, tablets)

[1406] Software: Database, generative AI model (e.g., GPT-3), front-end framework (e.g., React), back-end framework (e.g., Node.js)

[1407] When the system starts up, the server initializes the database and creates the necessary tables, such as the user table, scenario table, and log table. This allows users to register with the system and prepare it for use. When a user accesses the new registration screen and enters information such as their username, email address, and password, the terminal sends this information to the server. The server receives the data and stores it in the database.

[1408] The server then retrieves available scenario information from the database and sends it to the user's device, allowing the user to select the scenario they are interested in from the displayed list of scenarios. Once the user selects a scenario, the selection information is sent to the server, which then prepares a generative AI model based on the selected scenario.

[1409] When role-playing begins, the server generates questions for the first phase of the selected scenario and sends them to the user. For example, in a technical interview scenario, a question such as "Tell me about a project you recently worked on" is generated. The user enters answers to the questions, and the answer data is sent to the server via the terminal.

[1410] The server passes the received answers to the generative AI model, which analyzes them in real time. The generated feedback is then sent back to the user's device and displayed to the user. Depending on the feedback, the user can refine their answer and send it back to the server. By repeating this process, the user's communication skills can be improved.

[1411] After the session ends, the server saves the user's answer history, feedback history, and progress data. Based on this data, the server analyzes the user's progress and generates a visual report. The report is sent to the user's device, allowing the user to visually check their progress.

[1412] For example, if user "Sato" uses the technical interview scenario,

[1413] 1. Initial setup and user registration:

[1414] Mr. Sato accesses the new registration screen and enters his username, email address, and password.

[1415] The server receives this information and stores it in a database.

[1416] 2. Scenario Selection:

[1417] Sato accesses the dashboard and selects the technical interview scenario.

[1418] The server loads the data for the selected scenario and prepares the generative AI model.

[1419] 3. Role-playing:

[1420] The server generates a question such as "Tell me about a project you've worked on recently" and sends it to Sato.

[1421] Mr. Sato enters the answer and sends it to the server.

[1422] The server analyzes the answers using a generative AI model, generating feedback such as, "Please explain the specific technology stack and role."

[1423] 4. Real-time feedback:

[1424] Sato refines the answer based on the feedback and submits it to the server again.

[1425] The server analyzes the refined answer and provides more specific feedback.

[1426] 5. Progress Tracking and Analysis:

[1427] After the session ends, the server stores Mr. Sato's answer history, feedback history, and progress data in a database.

[1428] A progress report is generated based on the saved data and sent to Sato.

[1429] Sato checks the report and visually confirms his own growth.

[1430] Examples of prompts include:

[1431] "Tell me about a project you've worked on recently."

[1432] "Please explain your specific role in the project and the technology you used."

[1433] "Tell us about a challenge you faced in the project and how you solved it."

[1434] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and aims to improve communication abilities in remote environments.

[1435] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1436] Program flow:

[1437] Step 1: Initialize the database and set up the structure

[1438] The server initializes the database when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[1439] Input: System startup event

[1440] Data manipulation: Creating tables using SQL queries

[1441] Output: Initialized database

[1442] Specific behavior:

[1443] Generate each table with the CREATE TABLE statement

[1444] Insert initial data into the database using INSERT statements as needed

[1445] Step 2: Receiving and storing user registration information

[1446] The user accesses the new registration screen, enters their username, email address, and password, and presses the registration button. The device then sends this data to the server.

[1447] Input: User registration information (user name, email address, password)

[1448] Data processing: data validation and encoding

[1449] Output: User information stored in the database

[1450] Specific behavior:

[1451] The front-end collects form data and sends it to the server via API

[1452] The backend performs input validation (e.g., email address format check)

[1453] The password is hashed and saved in the database using an INSERT statement.

[1454] Step 3: Get and display the scenario list

[1455] The server retrieves available scenario information from the database and sends it to the user's terminal, which then displays the received scenario information.

[1456] Input: Request to get a scenario list

[1457] Data processing: Retrieving scenario information from the database

[1458] Output: Scenario information sent to the user's device

[1459] Specific behavior:

[1460] Retrieve a list of scenarios from the database using a SELECT statement

[1461] The acquired data is encoded in JSON format and sent to the terminal via API.

[1462] The front end displays the scenario list in list format.

[1463] Step 4: Select and submit a scenario

[1464] The user selects the scenario of interest and sends the selection to the server, which then prepares a generative AI model based on the selected scenario.

[1465] Input: ID of the selected scenario

[1466] Data processing: Loading AI models based on scenario ID

[1467] Output: A prepared generative AI model

[1468] Specific behavior:

[1469] The front end gets the ID of the selected scenario and sends it to the server

[1470] The server loads the generative AI model based on the scenario ID and prepares the next prompt.

[1471] Step 5: Generate and submit your question

[1472] The server generates the questions for the first phase of the selected scenario and sends them to the user.

[1473] Input: Data for the selected scenario

[1474] Data processing: prompt sentence generation

[1475] Output: The question sent to the user's device

[1476] Specific behavior:

[1477] Executes dialog generation logic based on scenarios

[1478] Generate an initial question, encode it in JSON format, and send it to the terminal.

[1479] The front end displays the question to the user

[1480] Step 6: User answers and submits

[1481] The user inputs an answer to the question, and the answer data is sent to the server via the terminal.

[1482] Input: User response data

[1483] Data processing: Validation and encoding of response data

[1484] Output: Response data sent to the server

[1485] Specific behavior:

[1486] The front end collects the answers entered by the user

[1487] Response data is sent to the server via API

[1488] Step 7: Analyze responses and generate feedback

[1489] The server passes the received answers to a generative AI model for real-time analysis, and the generated feedback is sent to the user's device.

[1490] Input: User response data

[1491] Data processing: Analysis with generative AI models

[1492] Output: Generated feedback

[1493] Specific behavior:

[1494] Provide answer data as input to the generative AI model

[1495] Obtain analysis results and format them as feedback

[1496] Feedback is encoded in JSON format and sent to the user's device

[1497] The front end displays feedback to the user

[1498] Step 8: Re-enter and submit your improved answers

[1499] The user refines the answer based on the feedback and submits it to the server again.

[1500] Input: Improved response data

[1501] Data processing: Revalidation and encoding

[1502] Output: Improved answer sent to the server

[1503] Specific behavior:

[1504] The user refines the answer and enters it again

[1505] Response data is sent to the server again

[1506] Step 9: Save your data

[1507] After the session ends, the server stores the user's answer history, feedback history, and progress data.

[1508] Input: User session data (answers, feedback)

[1509] Data processing: structuring data

[1510] Output: Session data stored in the database

[1511] Specific behavior:

[1512] Save answer history and feedback history to the log table using INSERT statements

[1513] Step 10: Analyze data and generate reports

[1514] The server analyzes the user's progress based on the stored data and creates a visual report, which is then sent to the user's device for display.

[1515] Input: Saved session data

[1516] Data processing: data analysis and visualization

[1517] Output: Progress report sent to the user's terminal

[1518] Specific behavior:

[1519] Analyzes stored data and generates progress reports

[1520] Use a data visualization library (e.g., D3.js) to generate graphs, etc.

[1521] Encode the report in JSON format and send it to the user's device

[1522] The front end displays the report to the user

[1523] In this way, the system provides a series of processes for users to interactively improve their communication skills.

[1524] (Application example 1)

[1525] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1526] In today's virtual stores, salespeople's communication skills have a significant impact on the quality of the customer experience. However, traditional training methods make it difficult to provide real-time feedback and encourage continuous skill improvement. Furthermore, due to a lack of mechanisms for accurately tracking and analyzing progress, it is difficult to efficiently support salespeople in improving their skills.

[1527] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1528] In this invention, the server includes user registration means, means for displaying multiple scenario data, generation means for generating a conversation scenario based on scenario data selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, and means for evaluating the sales skills of salespeople in the virtual store and providing scenarios for training them. This allows salespeople to receive feedback in real time while undergoing training, enabling continuous skill improvement.

[1529] The "user registration means" is a function for registering a new user in the system, and is usually a means for inputting information such as a user name, email address, and password, and storing the information in a database.

[1530] "Means for displaying multiple scenario data" is a function that displays a list of various scenarios that the user can select on the screen, and is a means that allows the user to select which specific scenario to use.

[1531] The "means for generating a conversation scenario" is a function for automatically generating a conversation in accordance with a scenario selected by a user, based on the scenario data.

[1532] The "means for acquiring answer data from the user" is a function for the system to receive answers input by the user to the conversation scenario and store them as data.

[1533] The "analysis means for generating feedback in real time" is a function for analyzing user response data and automatically generating appropriate feedback immediately.

[1534] The "means for providing the generated feedback to the user" is a function for transmitting the generated feedback to the user's terminal and displaying it on the screen.

[1535] The "means for saving and analyzing user progress data" is a function for evaluating the improvement of a user's skills by saving the user's past response data and feedback content and analyzing them as appropriate.

[1536] "Means for providing scenarios for evaluating and training salespeople's response skills in a virtual store" refers to a function for providing scenarios designed to improve salespeople's response skills in a virtual store environment and for conducting training through those scenarios.

[1537] The present invention is a system for improving a user's communication ability, and is particularly designed to evaluate and train salespeople's communication skills in a virtual store. The components and processes of this system are as follows:

[1538] System Components

[1539] 1. User registration method

[1540] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[1541] 2. A way to display multiple scenario data

[1542] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[1543] 3. Conversation scenario generation method

[1544] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[1545] 4. How to obtain response data from users

[1546] The server receives the answer data entered by the user when answering the scenario and stores it in a database.

[1547] 5. Analytics that generate real-time feedback

[1548] The server analyzes each answer using a generative AI model and generates feedback, which is immediately provided to the user, allowing them to refine their answer and submit it again.

[1549] 6. Means for providing generated feedback to users

[1550] The generated feedback is immediately sent to the user's device and displayed to the user, allowing the user to receive feedback in real time.

[1551] 7. A means of storing and analyzing user progress data

[1552] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report that is provided to the user. This report allows the user to visually confirm their own progress.

[1553] 8. A means of providing scenarios to assess and train sales associate skills in a virtual store

[1554] The server provides scenarios in a virtual store environment to improve sales staff skills, including new product introductions, complaint handling, and cross-selling, allowing for training that is tailored to actual sales situations.

[1555] Hardware and software used

[1556] This system uses the following hardware and software:

[1557] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[1558] Software: TensorFlow (library underlying generative AI models), MySQL (database management system), Node.js (server-side programming)

[1559] Specific examples

[1560] Scenario where a user is introducing a new product in a virtual store:

[1561] 1. User Registration

[1562] The user accesses the new registration screen and enters the required information (user name, email address, password). The server receives this information and stores it in the database.

[1563] 2. Scenario Selection

[1564] The user accesses the dashboard and selects the "New Product Introduction Scenario." The server loads the data for the selected scenario and prepares the generative AI model.

[1565] 3. Role-playing

[1566] The server generates the first question, "What are the main features of this product?" and sends it to the user. The user answers, "This product has a high-performance battery and fast charging capabilities." The server then runs this answer through a generative AI model, which generates feedback such as, "Please be specific about battery life and charging time."

[1567] Example prompt sentence:

[1568] "If a customer asks about the features of a new product, how would you explain it?"

[1569] The system allows salespeople to receive real-time feedback and effectively improve their skills in the virtual store.

[1570] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1571] Step 1:

[1572] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). This ensures that data for user registration and scenario selection is correctly saved and managed.

[1573] Input: System boot

[1574] Output: Initialize the database and create tables.

[1575] What happens next: The server connects to the MySQL database and creates the necessary tables using the CREATE statement.

[1576] Step 2:

[1577] The user accesses the new registration screen, enters information such as a user name, email address, and password, and presses the registration button.

[1578] Input: Username, Email Address, Password

[1579] Output: Save registration information to database

[1580] Specific operation: The information entered by the user is sent to the server, and the server saves the information in the database using the INSERT statement.

[1581] Step 3:

[1582] The server retrieves a list of available scenarios from the database and sends it to the user's terminal.

[1583] Input: Request scenario list

[1584] Output: Send scenario list

[1585] Specific operation: The server uses a SELECT statement to retrieve scenario data from the database and sends the retrieved data to the user's device in JSON format.

[1586] Step 4:

[1587] The user selects a scenario of interest from the displayed list of scenarios and transmits the selected information to the server.

[1588] Input: Scenario selection information

[1589] Output: Preparation for selected scenarios

[1590] Specific operation: After receiving the user's selection information, the server retrieves detailed data on the selected scenario from the database and prepares the generative AI model.

[1591] Step 5:

[1592] The server generates questions for the first phase of the selected scenario and sends them to the user, who then inputs the answers, which the device then sends to the server.

[1593] Input: Scenario question generation request, user answer

[1594] Output: Send question, get answer

[1595] Specific operation: The server uses the generative AI model to generate questions based on the scenario and sends them to the user's device. When the user enters an answer, the answer is sent to the server.

[1596] Step 6:

[1597] The server passes the answer to the generative AI model, which analyzes it in real time and generates feedback, which is then sent back to the user's device and displayed to them.

[1598] Input: User response data

[1599] Output: Generated feedback

[1600] Specific operation: The server inputs the answer data into the generative AI model, obtains feedback as the model's analysis result, and sends the feedback to the user's device and displays it to the user.

[1601] Step 7:

[1602] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database.

[1603] Input: Session data

[1604] Output: Saved progress data

[1605] Specific behavior: The server stores the user's answers and feedback data in a database using INSERT or UPDATE statements, for future reference in sessions and for progress analysis.

[1606] Step 8:

[1607] The server analyzes the user's progress based on the stored data, creates a visualized report, and provides it to the user.

[1608] Input: Saved progress data

[1609] Output: Generate and send a progress report

[1610] Specific operation: The server analyzes past response data and feedback, generates a report visualizing the user's growth and areas for improvement, and sends it to the user's device.

[1611] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1612] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[1613] Program processing overview

[1614] Initial Setup and User Registration

[1615] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The server receives this information and saves it in the database.

[1616] Scenario Selection

[1617] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[1618] Role-playing

[1619] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[1620] Use of emotion engine

[1621] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to an analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[1622] Real-time feedback

[1623] For each answer, the server performs analysis and emotion recognition and generates feedback that is instantly sent to the user, allowing them to improve their answer and submit it again.

[1624] Progress Tracking and Analysis

[1625] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[1626] Specific examples

[1627] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[1628] 1. Initial setup and user registration:

[1629] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[1630] The server receives this information and stores it in a database.

[1631] 2. Scenario Selection:

[1632] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[1633] The server loads the data for the selected scenario and prepares the generative AI model.

[1634] 3. Role-playing:

[1635] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[1636] Yamada replies, "I've recently been working on developing a web application."

[1637] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1638] 4. Use of Emotion Engine:

[1639] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[1640] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[1641] 5. Real-time feedback:

[1642] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[1643] The server will analyze again and provide more specific feedback.

[1644] 6. Progress Tracking and Analysis:

[1645] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[1646] The server generates a progress report based on the stored data and sends it to Yamada.

[1647] Yamada checks the report and visually confirms his own growth.

[1648] In this way, by combining emotion engines, it becomes possible to provide more accurate feedback according to the user's emotions, and to more effectively support the improvement of communication skills.

[1649] The processing flow will be explained below.

[1650] Specific explanation of program processing (including emotion engine)

[1651] Initial Setup and User Registration

[1652] Step 1:

[1653] When the system starts up, the server establishes a database connection and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.).

[1654] Step 2:

[1655] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[1656] Step 3:

[1657] After the user has completed the input, he clicks the Register button.

[1658] Step 4:

[1659] The terminal transmits the input information to the server.

[1660] Step 5:

[1661] The server receives the information, checks it for format and duplication, and stores it in a database.

[1662] Scenario Selection

[1663] Step 6:

[1664] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[1665] Step 7:

[1666] The terminal displays a list of scenarios to the user.

[1667] Step 8:

[1668] The user selects the scenario of interest and clicks the Select button.

[1669] Step 9:

[1670] The terminal transmits the scenario information selected by the user to the server.

[1671] Step 10:

[1672] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[1673] Role-playing

[1674] Step 11:

[1675] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[1676] Step 12:

[1677] The terminal displays the first question to the user.

[1678] Step 13:

[1679] The user answers the questions by typing or speaking.

[1680] Step 14:

[1681] The terminal transmits the inputted answer to the server.

[1682] Step 15:

[1683] The server passes the answer to a generative AI model, which analyzes it in real time.

[1684] Use of emotion engine

[1685] Step 16:

[1686] The server extracts emotion information from the acquired user response data using an emotion engine.

[1687] Step 17:

[1688] The emotion engine uses speech and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's responses.

[1689] Step 18:

[1690] The emotion engine provides the recognized emotion information to the server.

[1691] Real-time feedback

[1692] Step 19:

[1693] The server generates appropriate feedback based on the analysis results, including emotional information.

[1694] Step 20:

[1695] The server transmits the generated feedback to the terminal.

[1696] Step 21:

[1697] The device displays feedback to the user.

[1698] Step 22:

[1699] The user improves their answer based on the feedback and re-enters it.

[1700] Step 23:

[1701] The device sends the improved answer to the server.

[1702] Step 24:

[1703] The server then passes the answer back to the generative AI model for analysis.

[1704] Step 25:

[1705] The server generates new feedback and sends it back to the device.

[1706] Step 26:

[1707] The device displays new feedback to the user.

[1708] Progress Tracking and Analysis

[1709] Step 27:

[1710] After each session, the server stores the user's answer history, feedback history, emotion data, and progress data in a database.

[1711] Step 28:

[1712] The server analyzes the user's progress based on the stored data and generates a visual report.

[1713] Step 29:

[1714] The server sends the generated progress report to the terminal.

[1715] Step 30:

[1716] The terminal displays a progress report to the user.

[1717] Through these steps, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills. By utilizing an emotion engine, the system provides feedback based on the user's emotions, providing a more effective learning experience.

[1718] Example 2

[1719] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1720] In conventional communication improvement systems, feedback to users' responses was uniform and could not be adapted to the user's emotions. As a result, the feedback users received was not optimized for individual situations, making it difficult to improve their communication skills efficiently.

[1721] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1722] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring answer data from the user, an analysis means for analyzing the acquired answer data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, a means for performing emotion analysis on the acquired answer data, and a means for reflecting the results of the emotion analysis in the feedback. This makes it possible to provide individualized feedback according to the user's emotions and more effectively improve communication skills.

[1723] "User registration means" refers to the means by which a user accesses the system, inputs his / her own information, and performs registration.

[1724] The "means for displaying a plurality of scenario data" is a means for visually displaying a plurality of scenarios provided by the system to the user.

[1725] The "generation means" is a means for generating a conversation scenario and questions based on scenario data selected by the user.

[1726] The "acquisition means" is a means for collecting response data from users and transmitting it to the system.

[1727] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time.

[1728] The "means for providing" is a means for providing and displaying the generated feedback to the user.

[1729] The "means for saving and analyzing progress data" refers to a means for saving a user's response history and feedback history in a database and analyzing progress based on that history.

[1730] The "means for performing emotion analysis" is a means for analyzing the emotional information contained in the acquired response data and recognizing specific emotions (joy, anger, sadness, etc.).

[1731] The "means for reflecting the results of emotion analysis in the feedback" refers to a means for adjusting the feedback content based on the results of emotion analysis and providing optimal feedback to the user.

[1732] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[1733] Initial Setup and User Registration

[1734] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the entered information to the server, which then stores the received information in the database. When registration is complete, the server sends a success message to the terminal, which displays it to the user.

[1735] Scenario Selection

[1736] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario and prepares for the next phase.

[1737] Role-playing

[1738] The server generates questions for the first phase of the selected scenario and sends them to the user. The device displays the questions to the user, who then enters answers. The device then sends the entered answers to the server, which passes them to the generative AI model for real-time analysis. The server generates feedback based on the analysis results and sends it to the device. The device then displays the generated feedback to the user.

[1739] Use of emotion engine

[1740] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to the analysis means to improve the accuracy of the feedback. For example, if the user's response indicates a negative emotion, the feedback can use an affiliative expression that corresponds to that emotion.

[1741] Real-time feedback

[1742] For each answer, the server analyzes and recognizes emotions, and generates feedback. The feedback is immediately sent to the user's device, allowing the user to improve their answer and submit it again. This allows users to receive continuous feedback and improve their communication skills.

[1743] Progress Tracking and Analysis

[1744] After the session ends, the server stores the user's answer history, feedback history, emotional data, and progress data in a database. The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually check their own progress.

[1745] Specific examples

[1746] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[1747] 1. Initial setup and user registration:

[1748] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[1749] The server receives this information and stores it in a database.

[1750] 2. Scenario Selection:

[1751] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[1752] The server loads the data for the selected scenario and prepares the generative AI model.

[1753] 3. Role-playing:

[1754] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[1755] Yamada replies, "I've recently been working on developing a web application."

[1756] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1757] 4. Use of Emotion Engine:

[1758] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[1759] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[1760] 5. Real-time feedback:

[1761] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[1762] The server will analyze again and provide more specific feedback.

[1763] 6. Progress Tracking and Analysis:

[1764] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[1765] The server generates a progress report based on the stored data and sends it to Yamada.

[1766] Yamada checks the report and visually confirms his own growth.

[1767] Prompt Sentence Examples

[1768] Here are some example prompts to input to a generative AI model:

[1769] For the user "Yamada's" most recent answer: "I recently worked on developing a web app," generate feedback that explains the technology stack and specific role. Also, analyze this answer using the emotion engine to derive emotions (e.g., joy, anger, sadness), and adjust the feedback accordingly.

[1770] By making full use of such a complex approach and providing highly accurate feedback that reflects the user's emotions, it is possible to efficiently support the improvement of communication skills.

[1771] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1772] Step 1:

[1773] When the system starts, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). In this step, the server sets the schema of each table and inserts the initial data. The input is the initial system information, and the output is an initialized database.

[1774] Step 2:

[1775] The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the user's input information to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The input is the user's registration information, and the output is a success message.

[1776] Step 3:

[1777] The server retrieves a list of available scenarios from the database and sends it to the terminal. The terminal displays the received scenario list to the user. The input is a request for a scenario list, and the output is the scenario list.

[1778] Step 4:

[1779] The user selects the scenario of interest from the displayed list of scenarios and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario. The input is the scenario selection information, and the output is a notification that the scenario data has been loaded.

[1780] Step 5:

[1781] The server generates questions for the first phase of the selected scenario and sends them to the user. The terminal displays the questions to the user. The user inputs the answers and sends them to the terminal. The input is the user's answers, and the output is the obtained answer data.

[1782] Step 6:

[1783] The server passes the received answers to the generative AI model, which analyzes them in real time. The input is the user's answer data, and the output is the analysis result.

[1784] Step 7:

[1785] The server generates feedback based on the analysis results and sends it to the terminal, which then displays the generated feedback to the user. The input is the analysis results and the output is the feedback message.

[1786] Step 8:

[1787] The server extracts emotional information from the acquired user response data using an emotion engine. The emotion engine uses voice analysis and text analysis to recognize emotions from the user's response. The input is the user's response data, and the output is emotional information.

[1788] Step 9:

[1789] The server adjusts the feedback content based on the results of emotion analysis and generates optimal feedback for the user. The input is emotion information and initial feedback, and the output is the adjusted feedback.

[1790] Step 10:

[1791] The server then sends the generated feedback back to the device, which displays it to the user. The user then inputs a new answer based on the feedback and sends it back to the device. The input is the adjusted feedback, and the output is the new answer.

[1792] Step 11:

[1793] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. The input is the session data, and the output is the stored data.

[1794] Step 12:

[1795] The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually confirm their own growth. The input is the progress data, and the output is the visualized report.

[1796] (Application example 2)

[1797] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1798] Conventional customer service training systems do not provide sufficient feedback to improve users' communication skills, especially when it comes to providing feedback that takes into account the user's emotional information. As a result, the effectiveness of training is limited, and it is difficult to improve skills that are suited to real customer service situations.

[1799] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1800] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on analyzed emotion information. This makes it possible to provide more accurate feedback that reflects the user's emotion information, thereby enabling the improvement of effective communication skills suited to real customer service situations.

[1801] (User registration means) is a function for registering new users in the system and saving their information in the database.

[1802] (Scenario data) is a collection of data that defines virtual conversation situations for users to use in training.

[1803] (Generation means) is a function that automatically generates specific conversation scenarios and questions based on scenario data selected by the user.

[1804] (Acquisition means) is a function for collecting response data entered by users and processing the data within the system.

[1805] (Analysis means) is a function that analyzes the acquired response data and generates feedback in real time.

[1806] (Providing means) is a function that presents the generated feedback to the user and enables the user to make a new response based on the feedback.

[1807] "Progress Data" is a collection of information that indicates the user's response history, feedback history, and other relevant data obtained throughout the training.

[1808] (Emotion analysis means) is a function for extracting emotional information from the user's response data and reflecting that information in the feedback.

[1809] A system that realizes this invention is a system that includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on the analyzed emotion information.

[1810] Hardware and software used

[1811] Hardware:

[1812] head-mounted display

[1813] Smartphone

[1814] software:

[1815] Python

[1816] TextBlob (emotional analysis)

[1817] TfidfVectorizer (text analysis)

[1818] Logistic Regression (generative AI model)

[1819] Processing Description

[1820] When the system starts up, the server initializes the database and creates the necessary tables such as the user table, scenario table, log table, emotion data table, etc. When a user accesses the new registration screen, enters information such as a username, email address, and password, and presses the register button, the server receives this information and saves it in the database.

[1821] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server, which then prepares a generative AI model based on the selected scenario.

[1822] The server generates the first question for the selected scenario and sends it to the user. The user enters the answer, and the device sends the input to the server. The server passes the answer to the generative AI model, analyzes it in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[1823] The server uses an emotion engine to extract emotion information from the acquired user response data. The emotion engine uses TextBlob to recognize emotions from the user's responses. The recognized emotion information is provided to the analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[1824] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress, creates a visualized report, and provides it to the user.

[1825] Specific examples

[1826] Here is a concrete example of how a new customer service representative at a brick-and-mortar store uses the training system.

[1827] 1. Initial setup and user registration:

[1828] The customer service representative accesses the new registration screen and enters the required information (user name, email address, password).

[1829] The server receives this information and stores it in a database.

[1830] 2. Scenario Selection:

[1831] A customer service representative accesses the dashboard and selects the "First Time Customer Scenario."

[1832] The server loads the data for the selected scenario and prepares the generative AI model.

[1833] 3. Role-playing:

[1834] The server generates the first question, "Welcome. What item would you like to purchase today?" and sends it to the customer service representative.

[1835] The customer service representative replies, "I've recently been working on developing a web application."

[1836] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1837] 4. Use of Emotion Engine:

[1838] The server analyzes the customer service representative's responses using an emotion engine and extracts emotional information.

[1839] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[1840] 5. Real-time feedback:

[1841] Based on the feedback, the customer service representative answers again, "I used React as the front end and Node.js as the back end."

[1842] The server will analyze again and provide more specific feedback.

[1843] 6. Progress Tracking and Analysis:

[1844] After the session ends, the server stores the agent's response history, feedback history, and progress data.

[1845] The server generates a progress report based on the stored data and sends it to the customer service representative.

[1846] Customer service representatives review the reports and visually see their own progress.

[1847] An example of a specific prompt is, "Would you like to train a scenario where you meet a customer for the first time? Please choose a specific situation."

[1848] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1849] Step 1:

[1850] User Registration

[1851] The user accesses the new registration screen, enters information such as a username, email address, and password, and clicks the Register button.

[1852] Input: Information entered by the user, such as username, email address, or password.

[1853] Output: The user information is saved in the database and the user is recorded as a new user.

[1854] What happens: The server takes this information and executes a SQL query to store it in a database.

[1855] Step 2:

[1856] Scenario Selection

[1857] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[1858] Input: Scenario data stored in the database.

[1859] Output: The retrieved scenario list is displayed on the user's terminal.

[1860] Specific operation: The server retrieves scenario data from the scenario table using a SELECT query and sends it to the terminal in JSON format.

[1861] Step 3:

[1862] Scenario selection and generation

[1863] The user selects a scenario of interest from the displayed list of scenarios and transmits the selected information to the server.

[1864] Input: Scenario information selected by the user.

[1865] Output: A conversation scenario generated based on the selected scenario.

[1866] Specific operation: The server receives the selected scenario information, prepares the generative AI model, and generates an initial conversation scenario.

[1867] Step 4:

[1868] Acquiring response data

[1869] The server generates the first question for the selected scenario and sends it to the user.

[1870] Input: Initial question data based on the scenario.

[1871] Output: The question displayed on the user's terminal.

[1872] Specific operation: The server uses the generative AI model to generate a question and sends it to the user's device.

[1873] Step 5:

[1874] Enter and submit response data

[1875] The user enters an answer to the question, and the terminal transmits the answer to the server.

[1876] Input: The answer data entered by the user.

[1877] Output: The response data sent to the server.

[1878] Specific operation: When the user enters an answer and presses the send button, the device sends the answer data to the server using the POST method.

[1879] Step 6:

[1880] Analysis of response data

[1881] The server passes the response data to a generative AI model, which analyzes it and generates feedback in real time.

[1882] Input: The answer data submitted by the user.

[1883] Output: The generated feedback data.

[1884] Specific operation: The server inputs the response data into the generative AI model, performs text analysis and sentiment analysis, and generates feedback.

[1885] Step 7:

[1886] Sentiment analysis and feedback adjustment

[1887] The server analyzes the acquired response data using an emotion engine to extract emotional information, and adjusts the feedback accordingly.

[1888] Input: User response data and sentiment engine analysis results.

[1889] Output: Feedback tailored based on emotional information.

[1890] Specific operation: The server uses TextBlob to perform sentiment analysis and reflects the sentiment information in the feedback.

[1891] Step 8:

[1892] Providing feedback

[1893] The server transmits the generated feedback back to the user's terminal and displays it to the user.

[1894] Input: Calibrated feedback data.

[1895] Output: Feedback displayed on the user's device.

[1896] Specific behavior: The server sends feedback data in JSON format to the device, and the device displays it.

[1897] Step 9:

[1898] Save and analyze progress data

[1899] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database, and generates a progress report based on this data.

[1900] Input: User answer history, feedback history, sentiment data, and progress data.

[1901] Output: A progress report for the user.

[1902] Specific operation: The server stores the data in a database, performs data analysis based on the stored data, and generates a visualized report.

[1903] Specific prompt examples

[1904] "Do you want to train a scenario for a first-time customer encounter? Pick a specific situation."

[1905] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1906] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1907] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1908] [Fourth embodiment]

[1909] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1910] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1911] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1912] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1913] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1914] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1915] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1916] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1917] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1918] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1919] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1920] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1921] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1922] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system supports continuous development by saving and analyzing users' progress.

[1923] Program processing overview

[1924] Initial Setup and User Registration

[1925] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[1926] Scenario Selection

[1927] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[1928] Role-playing

[1929] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to a generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[1930] Real-time feedback

[1931] For each answer, the server analyzes it and generates feedback that is immediately sent to the user, allowing them to refine and submit their answer again.

[1932] Progress Tracking and Analysis

[1933] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[1934] Specific examples

[1935] Here, a specific example is shown in which a user "Sato-san" uses the system to carry out a technical interview scenario.

[1936] 1. Initial setup and user registration:

[1937] Mr. Sato accesses the new registration screen and enters the required information (user name, email address, password).

[1938] The server receives this information and stores it in a database.

[1939] 2. Scenario Selection:

[1940] Sato accesses the dashboard and selects "Technical Interview Scenario."

[1941] The server loads the data for the selected scenario and prepares the generative AI model.

[1942] 3. Role-playing:

[1943] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Sato.

[1944] Sato replies, "I've recently been working on developing a web application."

[1945] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[1946] 4. Real-time feedback:

[1947] Based on the feedback, Sato answers again, "I used React as the front end and Node.js as the back end."

[1948] The server will analyze again and provide more specific feedback.

[1949] 5. Progress Tracking and Analysis:

[1950] After the session ends, the server stores Sato's answer history, feedback history, and progress data.

[1951] The server generates a progress report based on the stored data and sends it to Mr. Sato.

[1952] Sato checks the report and visually confirms his own growth.

[1953] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and solves the problem of improving communication abilities in a remote environment.

[1954] The processing flow will be explained below.

[1955] Specific explanation of program processing

[1956] Initial Setup and User Registration

[1957] Step 1:

[1958] The server establishes a database connection when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[1959] Step 2:

[1960] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[1961] Step 3:

[1962] After the user has completed the input, he clicks the Register button.

[1963] Step 4:

[1964] The terminal transmits the input information to the server.

[1965] Step 5:

[1966] The server receives the information, checks it for format and duplication, and stores it in a database.

[1967] Scenario Selection

[1968] Step 6:

[1969] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[1970] Step 7:

[1971] The terminal displays a list of scenarios to the user.

[1972] Step 8:

[1973] The user selects the scenario of interest and clicks the Select button.

[1974] Step 9:

[1975] The terminal transmits the scenario information selected by the user to the server.

[1976] Step 10:

[1977] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[1978] Role-playing

[1979] Step 11:

[1980] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[1981] Step 12:

[1982] The terminal displays the first question to the user.

[1983] Step 13:

[1984] The user answers the questions by typing or speaking.

[1985] Step 14:

[1986] The terminal transmits the inputted answer to the server.

[1987] Step 15:

[1988] The server passes the answer to a generative AI model, which analyzes it in real time.

[1989] Step 16:

[1990] The server generates appropriate feedback based on the analysis results.

[1991] Step 17:

[1992] The server transmits the generated feedback to the terminal.

[1993] Step 18:

[1994] The device displays feedback to the user.

[1995] Real-time feedback

[1996] Step 19:

[1997] The user improves their answer based on the feedback and re-enters it.

[1998] Step 20:

[1999] The device sends the improved answer to the server.

[2000] Step 21:

[2001] The server then passes the answer back to the generative AI model for analysis.

[2002] Step 22:

[2003] The server generates new feedback and sends it back to the device.

[2004] Step 23:

[2005] The device displays new feedback to the user.

[2006] Progress Tracking and Analysis

[2007] Step 24:

[2008] The server stores the user's answer history, feedback history, and progress data in a database after each session.

[2009] Step 25:

[2010] The server analyzes the user's progress based on the stored data and generates a visual report.

[2011] Step 26:

[2012] The server sends the generated progress report to the terminal.

[2013] Step 27:

[2014] The terminal displays a progress report to the user.

[2015] As described above, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills.

[2016] Example 1

[2017] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2018] Conventional communication skill improvement systems have difficulty providing real-time feedback to users' responses, preventing them from immediately improving their communication skills. Furthermore, they lack the ability to store and analyze progress data, preventing them from effectively supporting users' continuous improvement.

[2019] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2020] In this invention, the server includes user registration means, means for displaying multiple pieces of scenario information, generation means for generating a dialogue scenario based on scenario information selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, transmission means for the user's terminal to transmit input response data to the server, transmission means for the server to transmit the generated feedback to the user's terminal, and display means for the terminal to receive the data and display it to the user. This allows the user to receive instant feedback and continuously improve their communication skills.

[2021] The "user registration means" is a means by which a user inputs information for new registration in the system and stores that information in the database.

[2022] "Scenario information" is information relating to a number of dialogue scenarios that the user can select from, and role-playing is carried out based on this information.

[2023] The "generation means" is a means for generating a dialogue scenario and a prompt sentence based on scenario information selected by the user.

[2024] The "acquisition means" is a means by which the system acquires response data from the user.

[2025] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time based on the analysis results.

[2026] The "means for providing" is a means for transmitting the generated feedback to the user's terminal and displaying it.

[2027] "Progress Data" refers to data including a user's response history, feedback history, progress status, etc., and is used to evaluate a user's growth and improvement.

[2028] The "transmission means" is a means by which the user's terminal transmits response data to the server, and the server transmits the generated feedback to the user's terminal.

[2029] The "display means" is a means for visually displaying to the user the feedback and scenario information received by the user's terminal.

[2030] A "generative AI model" is an artificial intelligence model used to analyze user response data and generate appropriate feedback or the next prompt.

[2031] A "prompt sentence" is dialogue text that includes a question or instruction for the user to follow next.

[2032] This invention is a system that utilizes generative AI models to improve users' communication skills. The system allows users to register, select from a variety of scenarios, role-play based on the scenarios, and receive real-time feedback. The system also supports continuous improvement by saving and analyzing the user's progress.

[2033] This system is implemented primarily using the following hardware and software:

[2034] Hardware: Servers, user devices (PCs, smartphones, tablets)

[2035] Software: Database, generative AI model (e.g., GPT-3), front-end framework (e.g., React), back-end framework (e.g., Node.js)

[2036] When the system starts up, the server initializes the database and creates the necessary tables, such as the user table, scenario table, and log table. This allows users to register with the system and prepare it for use. When a user accesses the new registration screen and enters information such as their username, email address, and password, the terminal sends this information to the server. The server receives the data and stores it in the database.

[2037] The server then retrieves available scenario information from the database and sends it to the user's device, allowing the user to select the scenario they are interested in from the displayed list of scenarios. Once the user selects a scenario, the selection information is sent to the server, which then prepares a generative AI model based on the selected scenario.

[2038] When role-playing begins, the server generates questions for the first phase of the selected scenario and sends them to the user. For example, in a technical interview scenario, a question such as "Tell me about a project you recently worked on" is generated. The user enters answers to the questions, and the answer data is sent to the server via the terminal.

[2039] The server passes the received answers to the generative AI model, which analyzes them in real time. The generated feedback is then sent back to the user's device and displayed to the user. Depending on the feedback, the user can refine their answer and send it back to the server. By repeating this process, the user's communication skills can be improved.

[2040] After the session ends, the server saves the user's answer history, feedback history, and progress data. Based on this data, the server analyzes the user's progress and generates a visual report. The report is sent to the user's device, allowing the user to visually check their progress.

[2041] For example, if user "Sato" uses the technical interview scenario,

[2042] 1. Initial setup and user registration:

[2043] Mr. Sato accesses the new registration screen and enters his username, email address, and password.

[2044] The server receives this information and stores it in a database.

[2045] 2. Scenario Selection:

[2046] Sato accesses the dashboard and selects the technical interview scenario.

[2047] The server loads the data for the selected scenario and prepares the generative AI model.

[2048] 3. Role-playing:

[2049] The server generates a question such as "Tell me about a project you've worked on recently" and sends it to Sato.

[2050] Mr. Sato enters the answer and sends it to the server.

[2051] The server analyzes the answers using a generative AI model, generating feedback such as, "Please explain the specific technology stack and role."

[2052] 4. Real-time feedback:

[2053] Sato refines the answer based on the feedback and submits it to the server again.

[2054] The server analyzes the refined answer and provides more specific feedback.

[2055] 5. Progress Tracking and Analysis:

[2056] After the session ends, the server stores Mr. Sato's answer history, feedback history, and progress data in a database.

[2057] A progress report is generated based on the saved data and sent to Sato.

[2058] Sato checks the report and visually confirms his own growth.

[2059] Examples of prompts include:

[2060] "Tell me about a project you've worked on recently."

[2061] "Please explain your specific role in the project and the technology you used."

[2062] "Tell us about a challenge you faced in the project and how you solved it."

[2063] Thus, the present invention provides an effective means for users to interactively improve their communication skills, and aims to improve communication abilities in remote environments.

[2064] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2065] Program flow:

[2066] Step 1: Initialize the database and set up the structure

[2067] The server initializes the database when the system starts up and creates the necessary tables (user table, scenario table, log table, etc.).

[2068] Input: System startup event

[2069] Data manipulation: Creating tables using SQL queries

[2070] Output: Initialized database

[2071] Specific behavior:

[2072] Generate each table with the CREATE TABLE statement

[2073] Insert initial data into the database using INSERT statements as needed

[2074] Step 2: Receiving and storing user registration information

[2075] The user accesses the new registration screen, enters their username, email address, and password, and presses the registration button. The device then sends this data to the server.

[2076] Input: User registration information (user name, email address, password)

[2077] Data processing: data validation and encoding

[2078] Output: User information stored in the database

[2079] Specific behavior:

[2080] The front-end collects form data and sends it to the server via API

[2081] The backend performs input validation (e.g., email address format check)

[2082] The password is hashed and saved in the database using an INSERT statement.

[2083] Step 3: Get and display the scenario list

[2084] The server retrieves available scenario information from the database and sends it to the user's terminal, which then displays the received scenario information.

[2085] Input: Request to get a scenario list

[2086] Data processing: Retrieving scenario information from the database

[2087] Output: Scenario information sent to the user's device

[2088] Specific behavior:

[2089] Retrieve a list of scenarios from the database using a SELECT statement

[2090] The acquired data is encoded in JSON format and sent to the terminal via API.

[2091] The front end displays the scenario list in list format.

[2092] Step 4: Select and submit a scenario

[2093] The user selects the scenario of interest and sends the selection to the server, which then prepares a generative AI model based on the selected scenario.

[2094] Input: ID of the selected scenario

[2095] Data processing: Loading AI models based on scenario ID

[2096] Output: A prepared generative AI model

[2097] Specific behavior:

[2098] The front end gets the ID of the selected scenario and sends it to the server

[2099] The server loads the generative AI model based on the scenario ID and prepares the next prompt.

[2100] Step 5: Generate and submit your question

[2101] The server generates the questions for the first phase of the selected scenario and sends them to the user.

[2102] Input: Data for the selected scenario

[2103] Data processing: prompt sentence generation

[2104] Output: The question sent to the user's device

[2105] Specific behavior:

[2106] Executes dialog generation logic based on scenarios

[2107] Generate an initial question, encode it in JSON format, and send it to the terminal.

[2108] The front end displays the question to the user

[2109] Step 6: User answers and submits

[2110] The user inputs an answer to the question, and the answer data is sent to the server via the terminal.

[2111] Input: User response data

[2112] Data processing: Validation and encoding of response data

[2113] Output: Response data sent to the server

[2114] Specific behavior:

[2115] The front end collects the answers entered by the user

[2116] Response data is sent to the server via API

[2117] Step 7: Analyze responses and generate feedback

[2118] The server passes the received answers to a generative AI model for real-time analysis, and the generated feedback is sent to the user's device.

[2119] Input: User response data

[2120] Data processing: Analysis with generative AI models

[2121] Output: Generated feedback

[2122] Specific behavior:

[2123] Provide answer data as input to the generative AI model

[2124] Obtain analysis results and format them as feedback

[2125] Feedback is encoded in JSON format and sent to the user's device

[2126] The front end displays feedback to the user

[2127] Step 8: Re-enter and submit your improved answers

[2128] The user refines the answer based on the feedback and submits it to the server again.

[2129] Input: Improved response data

[2130] Data processing: Revalidation and encoding

[2131] Output: Improved answer sent to the server

[2132] Specific behavior:

[2133] The user refines the answer and enters it again

[2134] Response data is sent to the server again

[2135] Step 9: Save your data

[2136] After the session ends, the server stores the user's answer history, feedback history, and progress data.

[2137] Input: User session data (answers, feedback)

[2138] Data processing: structuring data

[2139] Output: Session data stored in the database

[2140] Specific behavior:

[2141] Save answer history and feedback history to the log table using INSERT statements

[2142] Step 10: Analyze data and generate reports

[2143] The server analyzes the user's progress based on the stored data and creates a visual report, which is then sent to the user's device for display.

[2144] Input: Saved session data

[2145] Data processing: data analysis and visualization

[2146] Output: Progress report sent to the user's terminal

[2147] Specific behavior:

[2148] Analyzes stored data and generates progress reports

[2149] Use a data visualization library (e.g., D3.js) to generate graphs, etc.

[2150] Encode the report in JSON format and send it to the user's device

[2151] The front end displays the report to the user

[2152] In this way, the system provides a series of processes for users to interactively improve their communication skills.

[2153] (Application example 1)

[2154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2155] In today's virtual stores, salespeople's communication skills have a significant impact on the quality of the customer experience. However, traditional training methods make it difficult to provide real-time feedback and encourage continuous skill improvement. Furthermore, due to a lack of mechanisms for accurately tracking and analyzing progress, it is difficult to efficiently support salespeople in improving their skills.

[2156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2157] In this invention, the server includes user registration means, means for displaying multiple scenario data, generation means for generating a conversation scenario based on scenario data selected by the user, acquisition means for acquiring response data from the user, analysis means for analyzing the acquired response data and generating feedback in real time, provision means for providing the generated feedback to the user, means for saving and analyzing user progress data, and means for evaluating the sales skills of salespeople in the virtual store and providing scenarios for training them. This allows salespeople to receive feedback in real time while undergoing training, enabling continuous skill improvement.

[2158] The "user registration means" is a function for registering a new user in the system, and is usually a means for inputting information such as a user name, email address, and password, and storing the information in a database.

[2159] "Means for displaying multiple scenario data" is a function that displays a list of various scenarios that the user can select on the screen, and is a means that allows the user to select which specific scenario to use.

[2160] The "means for generating a conversation scenario" is a function for automatically generating a conversation in accordance with a scenario selected by a user, based on the scenario data.

[2161] The "means for acquiring answer data from the user" is a function for the system to receive answers input by the user to the conversation scenario and store them as data.

[2162] The "analysis means for generating feedback in real time" is a function for analyzing user response data and automatically generating appropriate feedback immediately.

[2163] The "means for providing the generated feedback to the user" is a function for transmitting the generated feedback to the user's terminal and displaying it on the screen.

[2164] The "means for saving and analyzing user progress data" is a function for evaluating the improvement of a user's skills by saving the user's past response data and feedback content and analyzing them as appropriate.

[2165] "Means for providing scenarios for evaluating and training salespeople's response skills in a virtual store" refers to a function for providing scenarios designed to improve salespeople's response skills in a virtual store environment and for conducting training through those scenarios.

[2166] The present invention is a system for improving a user's communication ability, and is particularly designed to evaluate and train salespeople's communication skills in a virtual store. The components and processes of this system are as follows:

[2167] System Components

[2168] 1. User registration method

[2169] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the Register button. The server receives this information and saves it in the database.

[2170] 2. A way to display multiple scenario data

[2171] The server retrieves a list of available scenarios from the database and sends it to the user's device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[2172] 3. Conversation scenario generation method

[2173] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[2174] 4. How to obtain response data from users

[2175] The server receives the answer data entered by the user when answering the scenario and stores it in a database.

[2176] 5. Analytics that generate real-time feedback

[2177] The server analyzes each answer using a generative AI model and generates feedback, which is immediately provided to the user, allowing them to refine their answer and submit it again.

[2178] 6. Means for providing generated feedback to users

[2179] The generated feedback is immediately sent to the user's device and displayed to the user, allowing the user to receive feedback in real time.

[2180] 7. A means of storing and analyzing user progress data

[2181] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report that is provided to the user. This report allows the user to visually confirm their own progress.

[2182] 8. A means of providing scenarios to assess and train sales associate skills in a virtual store

[2183] The server provides scenarios in a virtual store environment to improve sales staff skills, including new product introductions, complaint handling, and cross-selling, allowing for training that is tailored to actual sales situations.

[2184] Hardware and software used

[2185] This system uses the following hardware and software:

[2186] Hardware: Smartphones, smart glasses, head-mounted displays (HMDs)

[2187] Software: TensorFlow (library underlying generative AI models), MySQL (database management system), Node.js (server-side programming)

[2188] Specific examples

[2189] Scenario where a user is introducing a new product in a virtual store:

[2190] 1. User Registration

[2191] The user accesses the new registration screen and enters the required information (user name, email address, password). The server receives this information and stores it in the database.

[2192] 2. Scenario Selection

[2193] The user accesses the dashboard and selects the "New Product Introduction Scenario." The server loads the data for the selected scenario and prepares the generative AI model.

[2194] 3. Role-playing

[2195] The server generates the first question, "What are the main features of this product?" and sends it to the user. The user answers, "This product has a high-performance battery and fast charging capabilities." The server then runs this answer through a generative AI model, which generates feedback such as, "Please be specific about battery life and charging time."

[2196] Example prompt sentence:

[2197] "If a customer asks about the features of a new product, how would you explain it?"

[2198] The system allows salespeople to receive real-time feedback and effectively improve their skills in the virtual store.

[2199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2200] Step 1:

[2201] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, etc.). This ensures that data for user registration and scenario selection is correctly saved and managed.

[2202] Input: System boot

[2203] Output: Initialize the database and create tables.

[2204] What happens next: The server connects to the MySQL database and creates the necessary tables using the CREATE statement.

[2205] Step 2:

[2206] The user accesses the new registration screen, enters information such as a user name, email address, and password, and presses the registration button.

[2207] Input: Username, Email Address, Password

[2208] Output: Save registration information to database

[2209] Specific operation: The information entered by the user is sent to the server, and the server saves the information in the database using the INSERT statement.

[2210] Step 3:

[2211] The server retrieves a list of available scenarios from the database and sends it to the user's terminal.

[2212] Input: Request scenario list

[2213] Output: Send scenario list

[2214] Specific operation: The server uses a SELECT statement to retrieve scenario data from the database and sends the retrieved data to the user's device in JSON format.

[2215] Step 4:

[2216] The user selects a scenario of interest from the displayed list of scenarios and transmits the selected information to the server.

[2217] Input: Scenario selection information

[2218] Output: Preparation for selected scenarios

[2219] Specific operation: After receiving the user's selection information, the server retrieves detailed data on the selected scenario from the database and prepares the generative AI model.

[2220] Step 5:

[2221] The server generates questions for the first phase of the selected scenario and sends them to the user, who then inputs the answers, which the device then sends to the server.

[2222] Input: Scenario question generation request, user answer

[2223] Output: Send question, get answer

[2224] Specific operation: The server uses the generative AI model to generate questions based on the scenario and sends them to the user's device. When the user enters an answer, the answer is sent to the server.

[2225] Step 6:

[2226] The server passes the answer to the generative AI model, which analyzes it in real time and generates feedback, which is then sent back to the user's device and displayed to them.

[2227] Input: User response data

[2228] Output: Generated feedback

[2229] Specific operation: The server inputs the answer data into the generative AI model, obtains feedback as the model's analysis result, and sends the feedback to the user's device and displays it to the user.

[2230] Step 7:

[2231] After the session ends, the server stores the user's answer history, feedback history, and progress data in a database.

[2232] Input: Session data

[2233] Output: Saved progress data

[2234] Specific behavior: The server stores the user's answers and feedback data in a database using INSERT or UPDATE statements, for future reference in sessions and for progress analysis.

[2235] Step 8:

[2236] The server analyzes the user's progress based on the stored data, creates a visualized report, and provides it to the user.

[2237] Input: Saved progress data

[2238] Output: Generate and send a progress report

[2239] Specific operation: The server analyzes past response data and feedback, generates a report visualizing the user's growth and areas for improvement, and sends it to the user's device.

[2240] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2241] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[2242] Program processing overview

[2243] Initial Setup and User Registration

[2244] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The server receives this information and saves it in the database.

[2245] Scenario Selection

[2246] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares a generative AI model based on the selected scenario.

[2247] Role-playing

[2248] The server generates questions for the first phase of the selected scenario and sends them to the user. The user inputs the answers, and the device sends the input to the server. The server passes the answers to the generative AI model, analyzes them in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[2249] Use of emotion engine

[2250] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to an analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[2251] Real-time feedback

[2252] For each answer, the server performs analysis and emotion recognition and generates feedback that is instantly sent to the user, allowing them to improve their answer and submit it again.

[2253] Progress Tracking and Analysis

[2254] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress and creates a visualized report to provide to the user.

[2255] Specific examples

[2256] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[2257] 1. Initial setup and user registration:

[2258] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[2259] The server receives this information and stores it in a database.

[2260] 2. Scenario Selection:

[2261] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[2262] The server loads the data for the selected scenario and prepares the generative AI model.

[2263] 3. Role-playing:

[2264] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[2265] Yamada replies, "I've recently been working on developing a web application."

[2266] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[2267] 4. Use of Emotion Engine:

[2268] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[2269] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[2270] 5. Real-time feedback:

[2271] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[2272] The server will analyze again and provide more specific feedback.

[2273] 6. Progress Tracking and Analysis:

[2274] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[2275] The server generates a progress report based on the stored data and sends it to Yamada.

[2276] Yamada checks the report and visually confirms his own growth.

[2277] In this way, by combining emotion engines, it becomes possible to provide more accurate feedback according to the user's emotions, and to more effectively support the improvement of communication skills.

[2278] The processing flow will be explained below.

[2279] Specific explanation of program processing (including emotion engine)

[2280] Initial Setup and User Registration

[2281] Step 1:

[2282] When the system starts up, the server establishes a database connection and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.).

[2283] Step 2:

[2284] The terminal displays a new registration screen to the user, and the user enters the required information (user name, email address, password).

[2285] Step 3:

[2286] After the user has completed the input, he clicks the Register button.

[2287] Step 4:

[2288] The terminal transmits the input information to the server.

[2289] Step 5:

[2290] The server receives the information, checks it for format and duplication, and stores it in a database.

[2291] Scenario Selection

[2292] Step 6:

[2293] The server retrieves a list of available scenarios from the database and sends it to the terminal.

[2294] Step 7:

[2295] The terminal displays a list of scenarios to the user.

[2296] Step 8:

[2297] The user selects the scenario of interest and clicks the Select button.

[2298] Step 9:

[2299] The terminal transmits the scenario information selected by the user to the server.

[2300] Step 10:

[2301] The server prepares the generative AI model based on the selected scenario information and loads the relevant data.

[2302] Role-playing

[2303] Step 11:

[2304] The server generates questions for the first phase of the selected scenario and sends them to the terminal.

[2305] Step 12:

[2306] The terminal displays the first question to the user.

[2307] Step 13:

[2308] The user answers the questions by typing or speaking.

[2309] Step 14:

[2310] The terminal transmits the inputted answer to the server.

[2311] Step 15:

[2312] The server passes the answer to a generative AI model, which analyzes it in real time.

[2313] Use of emotion engine

[2314] Step 16:

[2315] The server extracts emotion information from the acquired user response data using an emotion engine.

[2316] Step 17:

[2317] The emotion engine uses speech and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's responses.

[2318] Step 18:

[2319] The emotion engine provides the recognized emotion information to the server.

[2320] Real-time feedback

[2321] Step 19:

[2322] The server generates appropriate feedback based on the analysis results, including emotional information.

[2323] Step 20:

[2324] The server transmits the generated feedback to the terminal.

[2325] Step 21:

[2326] The device displays feedback to the user.

[2327] Step 22:

[2328] The user improves their answer based on the feedback and re-enters it.

[2329] Step 23:

[2330] The device sends the improved answer to the server.

[2331] Step 24:

[2332] The server then passes the answer back to the generative AI model for analysis.

[2333] Step 25:

[2334] The server generates new feedback and sends it back to the device.

[2335] Step 26:

[2336] The device displays new feedback to the user.

[2337] Progress Tracking and Analysis

[2338] Step 27:

[2339] After each session, the server stores the user's answer history, feedback history, emotion data, and progress data in a database.

[2340] Step 28:

[2341] The server analyzes the user's progress based on the stored data and generates a visual report.

[2342] Step 29:

[2343] The server sends the generated progress report to the terminal.

[2344] Step 30:

[2345] The terminal displays a progress report to the user.

[2346] Through these steps, the system allows users to receive real-time feedback through interactive role-playing, enabling them to efficiently improve their communication skills. By utilizing an emotion engine, the system provides feedback based on the user's emotions, providing a more effective learning experience.

[2347] Example 2

[2348] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2349] In conventional communication improvement systems, feedback to users' responses was uniform and could not be adapted to the user's emotions. As a result, the feedback users received was not optimized for individual situations, making it difficult to improve their communication skills efficiently.

[2350] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2351] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring answer data from the user, an analysis means for analyzing the acquired answer data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, a means for performing emotion analysis on the acquired answer data, and a means for reflecting the results of the emotion analysis in the feedback. This makes it possible to provide individualized feedback according to the user's emotions and more effectively improve communication skills.

[2352] "User registration means" refers to the means by which a user accesses the system, inputs his / her own information, and performs registration.

[2353] The "means for displaying a plurality of scenario data" is a means for visually displaying a plurality of scenarios provided by the system to the user.

[2354] The "generation means" is a means for generating a conversation scenario and questions based on scenario data selected by the user.

[2355] The "acquisition means" is a means for collecting response data from users and transmitting it to the system.

[2356] The "analysis means" is a means for analyzing the acquired response data and generating feedback in real time.

[2357] The "means for providing" is a means for providing and displaying the generated feedback to the user.

[2358] The "means for saving and analyzing progress data" refers to a means for saving a user's response history and feedback history in a database and analyzing progress based on that history.

[2359] The "means for performing emotion analysis" is a means for analyzing the emotional information contained in the acquired response data and recognizing specific emotions (joy, anger, sadness, etc.).

[2360] The "means for reflecting the results of emotion analysis in the feedback" refers to a means for adjusting the feedback content based on the results of emotion analysis and providing optimal feedback to the user.

[2361] This invention is a system that improves a user's communication skills by combining a generative AI model with an emotion engine. The system allows users to register, select various scenarios, and engage in scenario-based role-playing, providing real-time feedback along the way. Furthermore, the system analyzes the user's emotional information and reflects it in the feedback, helping to improve the user's communication skills more effectively.

[2362] Initial Setup and User Registration

[2363] When the system starts up, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the entered information to the server, which then stores the received information in the database. When registration is complete, the server sends a success message to the terminal, which displays it to the user.

[2364] Scenario Selection

[2365] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario and prepares for the next phase.

[2366] Role-playing

[2367] The server generates questions for the first phase of the selected scenario and sends them to the user. The device displays the questions to the user, who then enters answers. The device then sends the entered answers to the server, which passes them to the generative AI model for real-time analysis. The server generates feedback based on the analysis results and sends it to the device. The device then displays the generated feedback to the user.

[2368] Use of emotion engine

[2369] The server uses an emotion engine to extract emotional information from the acquired user response data. The emotion engine uses voice analysis and text analysis to recognize emotions (e.g., joy, anger, sadness, etc.) from the user's response. The recognized emotional information is provided to the analysis means to improve the accuracy of the feedback. For example, if the user's response indicates a negative emotion, the feedback can use an affiliative expression that corresponds to that emotion.

[2370] Real-time feedback

[2371] For each answer, the server analyzes and recognizes emotions, and generates feedback. The feedback is immediately sent to the user's device, allowing the user to improve their answer and submit it again. This allows users to receive continuous feedback and improve their communication skills.

[2372] Progress Tracking and Analysis

[2373] After the session ends, the server stores the user's answer history, feedback history, emotional data, and progress data in a database. The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually check their own progress.

[2374] Specific examples

[2375] Here, a specific example will be given in which a user "Yamada" uses the system to carry out a scenario of a technical interview.

[2376] 1. Initial setup and user registration:

[2377] Yamada accesses the new registration screen and enters the required information (user name, email address, password).

[2378] The server receives this information and stores it in a database.

[2379] 2. Scenario Selection:

[2380] Yamada accesses the dashboard and selects "Technical Interview Scenario."

[2381] The server loads the data for the selected scenario and prepares the generative AI model.

[2382] 3. Role-playing:

[2383] The server generates the first question, "Tell me about a project you've worked on recently," and sends it to Yamada.

[2384] Yamada replies, "I've recently been working on developing a web application."

[2385] The server then uses a generative AI model to analyze this response and generate feedback such as, "Please explain your specific technology stack and role."

[2386] 4. Use of Emotion Engine:

[2387] The server analyzes Yamada's answers using an emotion engine and extracts emotional information.

[2388] If the response indicates a positive sentiment, generate feedback like, "That's a good initiative. Can you tell us more about your achievements?"

[2389] 5. Real-time feedback:

[2390] Based on the feedback, Yamada answers again, "I used React as the front end and Node.js as the back end."

[2391] The server will analyze again and provide more specific feedback.

[2392] 6. Progress Tracking and Analysis:

[2393] After the session ends, the server stores Yamada's answer history, feedback history, and progress data.

[2394] The server generates a progress report based on the stored data and sends it to Yamada.

[2395] Yamada checks the report and visually confirms his own growth.

[2396] Prompt Sentence Examples

[2397] Here are some example prompts to input to a generative AI model:

[2398] For the user "Yamada's" most recent answer: "I recently worked on developing a web app," generate feedback that explains the technology stack and specific role. Also, analyze this answer using the emotion engine to derive emotions (e.g., joy, anger, sadness), and adjust the feedback accordingly.

[2399] By making full use of such a complex approach and providing highly accurate feedback that reflects the user's emotions, it is possible to efficiently support the improvement of communication skills.

[2400] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2401] Step 1:

[2402] When the system starts, the server initializes the database and creates the necessary tables (user table, scenario table, log table, emotion data table, etc.). In this step, the server sets the schema of each table and inserts the initial data. The input is the initial system information, and the output is an initialized database.

[2403] Step 2:

[2404] The user accesses the new registration screen, enters information such as username, email address, and password, and presses the register button. The terminal sends the user's input information to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The input is the user's registration information, and the output is a success message.

[2405] Step 3:

[2406] The server retrieves a list of available scenarios from the database and sends it to the terminal. The terminal displays the received scenario list to the user. The input is a request for a scenario list, and the output is the scenario list.

[2407] Step 4:

[2408] The user selects the scenario of interest from the displayed list of scenarios and sends the selection information to the server. The server prepares the generative AI model based on the selected scenario. The input is the scenario selection information, and the output is a notification that the scenario data has been loaded.

[2409] Step 5:

[2410] The server generates questions for the first phase of the selected scenario and sends them to the user. The terminal displays the questions to the user. The user inputs the answers and sends them to the terminal. The input is the user's answers, and the output is the obtained answer data.

[2411] Step 6:

[2412] The server passes the received answers to the generative AI model, which analyzes them in real time. The input is the user's answer data, and the output is the analysis result.

[2413] Step 7:

[2414] The server generates feedback based on the analysis results and sends it to the terminal, which then displays the generated feedback to the user. The input is the analysis results and the output is the feedback message.

[2415] Step 8:

[2416] The server extracts emotional information from the acquired user response data using an emotion engine. The emotion engine uses voice analysis and text analysis to recognize emotions from the user's response. The input is the user's response data, and the output is emotional information.

[2417] Step 9:

[2418] The server adjusts the feedback content based on the results of emotion analysis and generates optimal feedback for the user. The input is emotion information and initial feedback, and the output is the adjusted feedback.

[2419] Step 10:

[2420] The server then sends the generated feedback back to the device, which displays it to the user. The user then inputs a new answer based on the feedback and sends it back to the device. The input is the adjusted feedback, and the output is the new answer.

[2421] Step 11:

[2422] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. The input is the session data, and the output is the stored data.

[2423] Step 12:

[2424] The server analyzes the user's progress based on the stored data, creates a visualized report, and sends it to the user's device. The device displays the report to the user, allowing the user to visually confirm their own growth. The input is the progress data, and the output is the visualized report.

[2425] (Application example 2)

[2426] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2427] Conventional customer service training systems do not provide sufficient feedback to improve users' communication skills, especially when it comes to providing feedback that takes into account the user's emotional information. As a result, the effectiveness of training is limited, and it is difficult to improve skills that are suited to real customer service situations.

[2428] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2429] In this invention, the server includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on analyzed emotion information. This makes it possible to provide more accurate feedback that reflects the user's emotion information, thereby enabling the improvement of effective communication skills suited to real customer service situations.

[2430] (User registration means) is a function for registering new users in the system and saving their information in the database.

[2431] (Scenario data) is a collection of data that defines virtual conversation situations for users to use in training.

[2432] (Generation means) is a function that automatically generates specific conversation scenarios and questions based on scenario data selected by the user.

[2433] (Acquisition means) is a function for collecting response data entered by users and processing the data within the system.

[2434] (Analysis means) is a function that analyzes the acquired response data and generates feedback in real time.

[2435] (Providing means) is a function that presents the generated feedback to the user and enables the user to make a new response based on the feedback.

[2436] "Progress Data" is a collection of information that indicates the user's response history, feedback history, and other relevant data obtained throughout the training.

[2437] (Emotion analysis means) is a function for extracting emotional information from the user's response data and reflecting that information in the feedback.

[2438] A system that realizes this invention is a system that includes a user registration means, a means for displaying multiple scenario data, a generation means for generating a conversation scenario based on scenario data selected by the user, an acquisition means for acquiring response data from the user, an analysis means for analyzing the acquired response data and generating feedback in real time, a provision means for providing the generated feedback to the user, a means for saving and analyzing user progress data, and an emotion analysis means for adjusting the feedback based on the analyzed emotion information.

[2439] Hardware and software used

[2440] Hardware:

[2441] head-mounted display

[2442] Smartphone

[2443] software:

[2444] Python

[2445] TextBlob (emotional analysis)

[2446] TfidfVectorizer (text analysis)

[2447] Logistic Regression (generative AI model)

[2448] Processing Description

[2449] When the system starts up, the server initializes the database and creates the necessary tables such as the user table, scenario table, log table, emotion data table, etc. When a user accesses the new registration screen, enters information such as a username, email address, and password, and presses the register button, the server receives this information and saves it in the database.

[2450] The server retrieves a list of available scenarios from the database and sends it to the device. The user selects the scenario of interest from the displayed list and sends the selection information to the server, which then prepares a generative AI model based on the selected scenario.

[2451] The server generates the first question for the selected scenario and sends it to the user. The user enters the answer, and the device sends the input to the server. The server passes the answer to the generative AI model, analyzes it in real time, and generates feedback. The generated feedback is then sent back to the user's device and displayed to the user.

[2452] The server uses an emotion engine to extract emotion information from the acquired user response data. The emotion engine uses TextBlob to recognize emotions from the user's responses. The recognized emotion information is provided to the analysis means and used to improve the accuracy of feedback. For example, if the user's response indicates a negative emotion, the feedback can use a more friendly expression that corresponds to that emotion.

[2453] After the session ends, the server stores the user's answer history, feedback history, emotion data, and progress data in a database. Based on this data, the server analyzes the user's progress, creates a visualized report, and provides it to the user.

[2454] Specific examples

[2455] Here is a concrete example of how a new customer service representative at a brick-and-mortar store uses the training system.

[2456] 1. Initial setup and user registration:

[2457] The customer service representative accesses the new registration screen and enters the required information (user name, email address, password).

[2458] The server receives this information and stores it in a database.

[2459] 2. Scenario Selection:

[2460] A customer service representative accesses the dashboard and selects the "First Time Customer Scenario."

[2461] The server lo...

Claims

1. A user registration means; a means for displaying a plurality of scenario data; a generating means for generating a conversation scenario based on scenario data selected by a user; An acquisition means for acquiring response data from a user; an analysis means for analyzing the acquired response data and generating feedback in real time; providing means for providing the generated feedback to the user; a means for storing and analyzing user progress data; A system including:

2. 2. The system according to claim 1, wherein the generating means generates the next question based on scenario data selected by the user.

3. The system according to claim 1 , wherein the analysis means passes the user's response data to a generative AI model for analysis and generates feedback.

Images