system

Abstract

Provide a system. A means for receiving registration information from a user and registering information of a new user in a database; means for transmitting scenario data based on the selection of a training scenario; means for receiving a user's selection of an event in a scenario and performing an evaluation based on the selection; a means for transmitting role-playing data and storing a record of user behavior; means for generating and displaying evaluation results and feedback to the user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional new employee training has the problem of simulating a real-world work environment and making it difficult to properly evaluate each trainee's ability to adapt and their skills. Furthermore, there was insufficient feedback on the training content, making it difficult to specifically grasp individual areas for improvement and progress. Furthermore, there was no established method for effectively recording and evaluating role-plays and simulations, which resulted in low training efficiency and effectiveness. [Means for solving the problem]

[0005] The present invention provides a system including: means for receiving registration information from a user and registering new user information in a database; means for transmitting scenario data based on a training scenario selection; means for receiving a user's selection for an event in the scenario and performing an evaluation based on the selection; means for transmitting data for role-playing and saving a record of the user's behavior; and means for generating and displaying evaluation results and feedback to the user. This system makes it possible to simulate a trainee's real-time judgments and responses during new employee training, appropriately evaluate the results, and provide specific feedback. Furthermore, the system records the response status of each trainee, allowing for reevaluation and identifying areas for continuous improvement, thereby significantly improving the effectiveness and efficiency of training content.

[0006] A "user" is a person who uses the system to conduct training or simulations.

[0007] "Registration information" refers to personal information and authentication information provided by a user to the system, which is stored in a database.

[0008] A "database" is a collection of data that stores user information, scenario data, etc. and can be searched or referenced as needed.

[0009] A "training scenario" defines the events and conditions that make up a series of simulations or role-plays that users experience.

[0010] "Scenario data" refers to data that includes specific information and setting values ​​related to a training scenario, and is used for simulations and role-plays.

[0011] An "event" is a specific situation or occurrence that occurs within the simulation and requires a user response.

[0012] "Selection" refers to the action of a user choosing one of the options presented within a simulation or role-play.

[0013] "Evaluation" is the act of determining the appropriateness and effectiveness of a user's choices and actions and calculating the results.

[0014] "Role-playing" is a training technique in which a user plays a specific role and behaves and interacts in accordance with that role.

[0015] "Behavior record" is data that records a series of actions and conversations that a user performs during a role-play or simulation.

[0016] "Feedback" is information that provides improvements and advice based on the evaluation results of the user's actions and responses.

[0017] A "system" is a set of devices and programs for carrying out a series of user registrations, training scenarios, simulations, role-plays, and evaluations and feedback. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] System Overview

[0040] This invention is a system for supporting new employee training, allowing trainees to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and user terminals, and has the function of evaluating trainees' response capabilities through simulations and role-playing, and providing feedback.

[0041] System Details

[0042] The system is realized using a server and a terminal including the following means:

[0043] 1. User registration method

[0044] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[0045] The terminal sends the information to the server, which then registers the information as a new user in the database.

[0046] The server sends a registration completion message to the terminal, which displays it.

[0047] 2. Training scenario selection method

[0048] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[0049] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[0050] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[0051] The device displays the scenario start screen.

[0052] 3. Simulation execution method

[0053] The terminal displays events in the scenario on the screen, and the user selects a response.

[0054] The terminal transmits the user's selection to the server, which performs an evaluation based on the selection.

[0055] The server transmits the evaluation results to the terminal, which displays them to the user.

[0056] 4. Role-playing tools

[0057] The user selects a role-playing session and clicks the "Start" button.

[0058] The terminal transmits information to the server, and the server transmits data for role-playing to the terminal.

[0059] The device assigns a role to the user, and the user initiates an action or interaction.

[0060] The device sends the activity record to the server, which stores it.

[0061] 5. Evaluation and feedback generation methods

[0062] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[0063] The server sends the feedback to the terminal, which displays it to the user.

[0064] Specific examples

[0065] 1. Example of user registration

[0066] Tanaka, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[0067] The terminal sends this information to the server, and the server registers Tanaka's information in the database.

[0068] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Tanaka.

[0069] 2. Specific examples of training scenario selection

[0070] After completing registration, Tanaka logs in and a list of training scenarios is displayed. He selects "Customer Response Simulation."

[0071] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[0072] The device displays the scenario start screen, and Tanaka begins the simulation.

[0073] 3. Specific examples of simulation execution

[0074] During the simulation, the "Complaint handling" event is displayed on Tanaka's device.

[0075] Tanaka selects "Apologize" and the device sends the selection to the server.

[0076] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[0077] The device displays the evaluation results to Tanaka.

[0078] 4. Example of a role-play session

[0079] Tanaka begins the "sales presentation" role-play.

[0080] The device assigns Tanaka the role of presenter, and Tanaka actually gives the presentation.

[0081] The device records the content of Tanaka's presentation and sends it to the server.

[0082] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[0083] The device displays the evaluation results and feedback to Tanaka.

[0084] Throughout the system, users' actions are assessed and specific feedback is provided to help trainees improve their skills.

[0085] The processing flow will be explained below.

[0086] User Registration

[0087] Step 1:

[0088] The terminal presents the user with a login screen.

[0089] Step 2:

[0090] The user enters their email address and password on the login screen and clicks the "Register" button.

[0091] Step 3:

[0092] The terminal transmits the input information to the server.

[0093] Step 4:

[0094] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[0095] Step 5:

[0096] The server sends a registration completion message to the terminal.

[0097] Step 6:

[0098] The terminal displays a completion message to the user.

[0099] Training scenario selection

[0100] Step 1:

[0101] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[0102] Step 2:

[0103] The terminal displays a list of scenarios to the user.

[0104] Step 3:

[0105] The user selects the desired training scenario and clicks the "Start" button.

[0106] Step 4:

[0107] The terminal transmits scenario selection information to the server.

[0108] Step 5:

[0109] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[0110] Step 6:

[0111] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[0112] Simulation execution

[0113] Step 1:

[0114] The device displays the events in the scenario on the screen.

[0115] Step 2:

[0116] The user selects a response to the event and decides on the option.

[0117] Step 3:

[0118] The terminal transmits the user's selection to the server.

[0119] Step 4:

[0120] The server performs an evaluation based on the user's selection and generates an evaluation result.

[0121] Step 5:

[0122] The server transmits the evaluation results to the terminal.

[0123] Step 6:

[0124] The terminal displays the evaluation results to the user.

[0125] Role-playing session

[0126] Step 1:

[0127] The user selects a role-playing session and clicks the "Start" button.

[0128] Step 2:

[0129] The terminal transmits role-play selection information to the server.

[0130] Step 3:

[0131] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[0132] Step 4:

[0133] The device assigns a role to the user and displays role-based instructions and goals.

[0134] Step 5:

[0135] Users act and interact based on their assigned roles.

[0136] Step 6:

[0137] The device sends a record of the user's actions and interactions to the server.

[0138] Step 7:

[0139] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[0140] Step 8:

[0141] The server generates ratings and feedback and sends them to the device.

[0142] Step 9:

[0143] The device displays the rating and feedback to the user.

[0144] Evaluation and feedback generation

[0145] Step 1:

[0146] The server receives and analyzes all simulation and role-play recording data.

[0147] Step 2:

[0148] The server generates a user rating based on the analysis results.

[0149] Step 3:

[0150] The server generates feedback along with the evaluation results and sends it to the terminal.

[0151] Step 4:

[0152] The device displays the rating and feedback to the user.

[0153] Example 1

[0154] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] Conventional training systems have difficulty effectively evaluating trainees' skills and providing feedback under conditions equivalent to the actual work environment. Furthermore, it is difficult to ensure real-time and high-quality feedback during the evaluation and feedback generation process, making it difficult to provide specific advice necessary for improving trainees' skills.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0157] In this invention, the server includes means for receiving registration information from a user and registering new user information in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and storing a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, and means for transmitting prompt sentences to a generative AI model to generate the evaluation results and feedback and receiving the generated feedback, thereby enabling high-quality feedback in real time to the trainee's behavior and selection.

[0158] "Registration information" refers to personal information, such as a user's email address and password, that is required to access and use the system.

[0159] The "database" is a system for centrally managing and storing data such as registration information, training scenarios, and evaluation results.

[0160] A "training scenario" is a storyline or content of a simulation or role-play for a specific task or situation that is selected and executed by a user.

[0161] "Scenario data" refers to detailed information and settings for constructing a training scenario, and is data that enables a scenario to be executed by a user.

[0162] An "event" is a specific situation or challenge that occurs within a training scenario, and is the subject of a user's response selection.

[0163] "Behavior record" is historical data of the actions and interactions selected by the user during a simulation or role-play.

[0164] The "evaluation results" are evaluation information of training results and skills generated based on the user's actions and selections.

[0165] "Feedback" refers to information such as specific advice and areas for improvement provided to the user based on the evaluation results.

[0166] A "generative AI model" is an artificial intelligence model used to generate evaluations and feedback on user actions and choices.

[0167] A "prompt" is an instruction given to a generative AI model that contains the information necessary to generate a specific evaluation or feedback.

[0168] This invention is a system for supporting new employee training, allowing users to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and terminals, and has the function of evaluating users' response capabilities through simulations and role-playing, and providing feedback.

[0169] System installation and operation

[0170] User Registration

[0171] When a user accesses the system for the first time, the terminal displays a user registration screen. The user enters registration information such as an email address and password, and the terminal sends this to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The terminal then displays a registration completion message to the user.

[0172] Selecting a training scenario

[0173] When a user logs in, the server retrieves a list of training scenarios from the database and sends it to the terminal. The terminal displays the list of scenarios to the user, and the user selects the desired scenario. The terminal sends this selection information to the server, and the server sends the selected scenario data to the terminal. The terminal displays the scenario start screen.

[0174] Running the simulation

[0175] When a scenario starts, the device displays the events in the scenario on the screen. When the user selects a response, the device sends the selection to the server. The server evaluates the selection and sends the evaluation results to the device. The device displays the evaluation results to the user.

[0176] Role-play execution

[0177] When a user selects a role-playing session and clicks the "Start" button, the device sends that information to the server. The server sends the role-playing data to the device, and the device assigns a role to the user. The user then gives an actual presentation or interacts, and the device sends a record of this behavior to the server. The server saves the behavior record.

[0178] Evaluation and feedback generation

[0179] The server evaluates the simulation and role-playing based on the recorded data and generates comprehensive feedback. The evaluation results and feedback are generated by a generative AI model through prompt sentences. The server sends the generated feedback to the device, which then displays it to the user.

[0180] Hardware and software used

[0181] This system consists of a server and a terminal used by the user. The server uses a database system (e.g., MySQL (registered trademark), PostgreSQL) and an AI model (e.g., GPT-3 (registered trademark), BERT). The terminal uses a web browser to display the screen and input data.

[0182] Specific examples

[0183] For example, consider the case where a user selects "Customer Service Simulation." During the simulation, the "Complaint Handling" event is displayed, and the user selects "Apologize." At this time, the device sends this selection information to the server, and the server sends the following prompt to the AI ​​model:

[0184] "In a complaint handling simulation, the user selected 'Apologize.' Please evaluate the appropriateness of this response and generate feedback."

[0185] Based on this input, the AI ​​model generates feedback such as "This is an appropriate response. It is a choice that is highly effective in easing the customer's anger," and sends it to the device via the server. The device then displays this to the user.

[0186] The system allows users to receive high-quality feedback in real time, effectively improving their skills in a real-world work environment.

[0187] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0188] Step 1:

[0189] A user accesses the system. Specifically, the user opens a browser and enters the system's URL. The input is the URL, and the user registration screen is displayed as the output.

[0190] Step 2:

[0191] The terminal displays the user registration screen. Specifically, it renders the registration form using HTML and CSS. The input is the access information from step 1, and the output is the registration form screen displayed to the user.

[0192] Step 3:

[0193] The user enters registration information. Specifically, the user enters an email address, password, etc., and clicks the "Register" button. Input is the information entered by the user, and output is the data sent to the terminal.

[0194] Step 4:

[0195] The terminal sends the entered information to the server. Specifically, data is sent to the server using JavaScript (registered trademark) or the form's POST method. The input is the user's registration information, and the output is the data sent to the server.

[0196] Step 5:

[0197] The server stores the received information in a database. Specifically, the data received on the server side is inserted into the database using SQL statements. The input is the data received from the terminal, and the output is the record stored in the database.

[0198] Step 6:

[0199] The server sends a registration completion message to the terminal. Specifically, the server returns a success message as a response. The input is the user information stored in the database, and the output is the completion message sent to the terminal.

[0200] Step 7:

[0201] The terminal displays a registration completion message to the user. Specifically, it displays a message in HTML to notify the user. The input is the completion message from the server, and the output is the message displayed to the user.

[0202] Step 8:

[0203] The server retrieves a list of training scenarios from the database and sends them to the terminal. Specifically, it uses SQL statements to select the scenario data and returns it in JSON format. The input is a database query, and the output is the scenario data sent to the terminal.

[0204] Step 9:

[0205] The device displays a list of scenarios to the user. Specifically, it uses JavaScript to draw the scenario list. The input is the scenario data received from the server, and the output is the scenario list displayed to the user.

[0206] Step 10:

[0207] The user selects a scenario. Specifically, the user clicks on a scenario from a dropdown or list. The input is the user's scenario selection, and the output is the selection information sent to the terminal.

[0208] Step 11:

[0209] The device sends the selection information to the server. Specifically, it uses AJAX or the Fetch API to send the selection information to the server. The input is the user's selection information, and the output is the data sent to the server.

[0210] Step 12:

[0211] The server sends the selected scenario data to the device. Specifically, the server returns the corresponding scenario data in JSON format. The input is the user's selection information, and the output is the scenario data sent to the device.

[0212] Step 13:

[0213] The terminal displays the scenario start screen. Specifically, it creates a screen based on the scenario data and displays it to the user. The input is the scenario data received from the server, and the output is the scenario start screen displayed to the user.

[0214] Step 14:

[0215] The device displays the events in the scenario on the screen. Specifically, it uses JavaScript to draw the event content. The input is the scenario data, and the output is the event content that is displayed to the user.

[0216] Step 15:

[0217] The user selects a response to an event. Specifically, the user clicks on an option. The input is the user's selection, and the output is the selection information sent to the terminal.

[0218] Step 16:

[0219] The device sends the user's selection to the server. Specifically, the selection information is sent using AJAX or the Fetch API. The input is the user's selection information, and the output is the data sent to the server.

[0220] Step 17:

[0221] The server evaluates the user's selection. Specifically, the server executes evaluation logic based on the options. The input is the user's selection information, and the output is the evaluation result.

[0222] Step 18:

[0223] The server sends the evaluation results to the terminal. Specifically, the evaluation results are returned in JSON format. The input is the evaluation result, and the output is the data sent to the terminal.

[0224] Step 19:

[0225] The terminal displays the evaluation results to the user. Specifically, the evaluation results are rendered in HTML and notified to the user. The input is the evaluation results received from the server, and the output is the evaluation results displayed to the user.

[0226] Step 20:

[0227] The user selects a role-playing session and clicks the "Start" button. Specifically, the user clicks a button on the screen. The input is the user's selection, and the output is the starting information sent to the terminal.

[0228] Step 21:

[0229] The device sends the information to the server. Specifically, it uses AJAX or the Fetch API to send data. The input is the user's initial information, and the output is the data sent to the server.

[0230] Step 22:

[0231] The server sends role-playing data to the device. Specifically, it retrieves data using SQL statements and returns it in JSON format. The input is the user's starting information, and the output is the role-playing data sent to the device.

[0232] Step 23:

[0233] The device assigns a role to the user. Specifically, it displays the role data on the screen using JavaScript. The input is role-play data, and the output is the role information displayed to the user.

[0234] Step 24:

[0235] The user acts and interacts. Specifically, the user acts according to the simulation. The input is the role-play scenario, and the output is the user's actions.

[0236] Step 25:

[0237] The device sends the behavioral record to the server. Specifically, the behavioral data is sent to the server via AJAX. The input is the user's behavioral data, and the output is the data sent to the server.

[0238] Step 26:

[0239] The server saves the behavioral records. Specifically, it inserts the recorded data into the database. The input is the behavioral data received from the device, and the output is the record saved in the database.

[0240] Step 27:

[0241] The server evaluates all simulation and role-play records. Specifically, the server executes the evaluation logic. The input is the recorded data of the simulation or role-play, and the output is the evaluation result.

[0242] Step 28:

[0243] The server sends a prompt to the generative AI model and generates feedback. Specifically, it sends the evaluation result as a prompt to the generative AI model and generates feedback. The input is the evaluation result data, and the output is the generated feedback.

[0244] Step 29:

[0245] The server sends feedback to the device. Specifically, it sends feedback data in JSON format. The input is the generated feedback, and the output is the data sent to the device.

[0246] Step 30:

[0247] The device displays the feedback to the user. Specifically, the feedback content is rendered in HTML and displayed to the user. The input is the feedback received from the server, and the output is the feedback displayed to the user.

[0248] (Application example 1)

[0249] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0250] There is a need for a system that allows trainees to efficiently acquire practical skills in the operation and maintenance of factory robots without actually using the robots on-site. Another challenge is to introduce a system that allows practical training without the risk of operational errors or safety risks. There is also a need for a flexible training environment that uses easily portable devices such as smartphones and tablets.

[0251] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0252] In this invention, the server includes means for receiving registration information from a user and registering information of a new user in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and saving a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, means for transmitting scenario data including procedures for operating and maintaining a factory robot, and means for running a simulation of a factory environment through an application installed on a smartphone or tablet device. This enables safe and efficient training in the operation and maintenance of a factory robot without using an actual robot.

[0253] The "user registration means" is a function that receives registration information from a user and registers the information of a new user in the database.

[0254] The "training scenario selection means" is a function that transmits scenario data based on the selection of a training scenario.

[0255] The "simulation execution means" is a function that receives a user's selection for an event in a scenario and performs an evaluation based on that selection.

[0256] "Role-playing means" is a function that transmits data for role-playing and stores a record of the user's actions.

[0257] The "evaluation and feedback generating means" is a function that generates evaluation results and feedback and displays them to the user.

[0258] The "factory environment simulation means" is a function that transmits scenario data including the operation and maintenance procedures of factory robots.

[0259] "Applications installed on smartphones and tablet devices" refers to software that is installed on these devices and runs simulations of factory environments.

[0260] This invention is a system that provides training for the operation and maintenance of factory robots, allowing participants to acquire practical skills safely and efficiently. The system operates in conjunction with a server and terminals, and provides functions for user registration, training scenario selection, simulation execution, role-playing, and evaluation and feedback. A factory environment simulation is run through an application installed on a smartphone or tablet device.

[0261] Hardware and Software

[0262] Hardware: Smartphones, tablets, database servers.

[0263] Software: Flask (a Python microframework), PostgreSQL (a database management system), HTML / CSS / JavaScript (for building the user interface).

[0264] System processing details

[0265] 1. User Registration

[0266] The server receives the registration information (e.g., email address, password) from the user and registers it as a new user in the database. When registration is complete, it sends a registration completion message to the terminal and displays the message to the user.

[0267] 2. Training scenario selection

[0268] The server sends a list of various scenarios (e.g., introduction to robot operation, regular maintenance, safety management training), which the terminal displays to the user. When the user selects the desired scenario, the information is sent to the server, and the selected scenario data is sent to the terminal.

[0269] 3. Run the simulation

[0270] The terminal executes a scenario of a factory environment, and the user selects the appropriate action for each event in the scenario. This selection is sent to the server for evaluation. The evaluation results are returned to the terminal as feedback and displayed to the user.

[0271] 4. Role-playing

[0272] The user selects a role-playing session, and the server sends the role-playing data to the device. The user's actions and interactions are recorded and sent to the server. The server evaluates the recording, generates feedback, and sends it to the device.

[0273] 5. Ratings and Feedback

[0274] The server will provide an overall evaluation and generate feedback based on the results of all simulations and role-plays, helping users understand specific areas for improvement and what went well.

[0275] Examples and prompts

[0276] Specific examples

[0277] User registration: A new user registers by entering their email address and password. The server receives this, stores it in the database, and sends a confirmation message.

[0278] Training scenario selection: After logging in, the user selects "Introduction to Robot Operation." The server sends the scenario data.

[0279] Simulation: The user is asked to perform an emergency stop of the robot. The server evaluates the action and sends feedback.

[0280] Role-play: Initiate a role-play of the maintenance procedure, record user actions, and provide evaluation and feedback.

[0281] Prompt Sentence Examples

[0282] Example of prompt text when user selects scenario:

[0283] "Please select one of the following training scenarios: 1. Introduction to Robot Operation 2. Routine Maintenance 3. Safety Management Training"

[0284] This system makes it possible to conduct training in the operation and maintenance of factory robots safely and efficiently.

[0285] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0286] Step 1:

[0287] User Registration

[0288] The terminal receives an email address and password from the user as input. It sends this information to the server. The server saves the new user's information in a database and generates a registration completion message. The server sends this message to the terminal, which displays it to the user. The input is the user's email address and password, and the output is the registration completion message.

[0289] Step 2:

[0290] Training scenario selection

[0291] After the user logs in, the terminal sends a scenario list request to the server. The server retrieves a list of training scenarios from the database and sends it to the terminal. The user selects the desired scenario from the list, and the terminal sends the selection information to the server. The server sends the selected scenario data to the terminal, which displays it to the user. The input is the scenario list request and scenario selection, and the output is the scenario data.

[0292] Step 3:

[0293] Simulation execution

[0294] When a user starts a scenario, the device displays the events in the scenario. The user selects an action for the event, and the device sends that selection to the server. The server receives the selection, runs an evaluation algorithm, and generates an evaluation result. The result is sent to the device, which displays the evaluation result to the user. The input is the user's selection, and the output is the evaluation result.

[0295] Step 4:

[0296] role play

[0297] When a user starts role-playing, the device requests role-playing data from the server. The server sends the data, and the device assigns a role to the user. The device records the user's actions and interactions and sends them to the server. The server evaluates the records, generates feedback results, and sends them to the device. The device displays the results to the user. The input is the user's action record, and the output is the feedback results.

[0298] Step 5:

[0299] Ratings and Feedback

[0300] The server performs a comprehensive evaluation based on the result data of all simulations and role-plays. It analyzes the user's operation records and behavioral data and evaluates them using an evaluation algorithm. The server generates comprehensive feedback and sends it to the device. The device displays the feedback to the user. The input is the operation records and behavioral data, and the output is comprehensive feedback.

[0301] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0302] System Overview

[0303] This invention is a new employee training system that enables trainees to acquire practical skills and incorporates an emotion engine that recognizes the user's emotions. This system operates in conjunction with a server and terminals, evaluating the trainee's ability to respond through simulations and role-playing and providing feedback. The emotion engine recognizes the user's emotions during training and reflects them in the progression of the scenario and feedback accordingly, thereby achieving effective, individually customized training.

[0304] System Details

[0305] The system is implemented using the following means:

[0306] 1. User registration method

[0307] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[0308] The terminal sends the information to the server, which then registers the information as a new user in the database.

[0309] The server sends a registration completion message to the terminal, which displays it.

[0310] 2. Training scenario selection method

[0311] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[0312] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[0313] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[0314] The device displays the scenario start screen.

[0315] 3. Simulation execution method

[0316] The terminal displays events in the scenario on the screen, and the user selects a response.

[0317] The terminal transmits the user's selection to the server.

[0318] The server performs an evaluation based on the user's selection and generates an evaluation result.

[0319] The server transmits the evaluation results to the terminal, which displays them to the user.

[0320] 4. Role-playing tools

[0321] The user selects a role-playing session and clicks the "Start" button.

[0322] The terminal transmits the selection information to the server, and the server transmits the role-play data to the terminal.

[0323] The device assigns a role to the user, and the user initiates an action or interaction.

[0324] The device sends the activity record to the server, which stores it.

[0325] 5. Evaluation and feedback generation methods

[0326] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[0327] The server sends the feedback to the terminal, which displays it to the user.

[0328] 6. Emotional Engine Means

[0329] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[0330] The device sends the collected emotion data to a server, which stores it.

[0331] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[0332] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[0333] Specific examples

[0334] 1. Example of user registration

[0335] Mr. A, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[0336] The terminal sends this information to the server, and the server registers Mr. A's information in the database.

[0337] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Mr. A.

[0338] 2. Specific examples of training scenario selection

[0339] After completing registration, Mr. A logs in and is presented with a list of training scenarios. He selects "Customer Service Simulation."

[0340] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[0341] The device displays the scenario start screen, and Person A begins the simulation.

[0342] 3. Specific examples of simulation execution

[0343] During the simulation, the "Complaint handling" event is displayed on Mr. A's device.

[0344] Person A selects "Apologize" and the device sends that selection to the server.

[0345] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[0346] The device displays the evaluation results to Mr. A.

[0347] 4. Example of a role-play session

[0348] Person A begins the "sales presentation" role-play.

[0349] The device assigns the role of presentation to Person A, who then gives the actual presentation.

[0350] The device records the content of Mr. A's presentation and sends it to the server.

[0351] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[0352] The device displays the evaluation results and feedback to Person A.

[0353] 5. Examples of using the Emotion Engine

[0354] During simulations and role-playing, the device recognizes emotions from Person A's facial expressions and voice and collects emotional data.

[0355] If the emotion indicates "tension" or "stress," the device sends it to the server.

[0356] The server analyzes the emotional data and adjusts the progress of the scenario accordingly.

[0357] For example, the server could lower the difficulty of the scenario based on emotional data and send appropriate feedback to Person A, such as "Take a break and relax here."

[0358] The device displays these instructions to Person A, prompting him to take appropriate action.

[0359] This system provides individually customized training content that takes into account the user's emotions, and allows for real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[0360] The processing flow will be explained below.

[0361] User Registration

[0362] Step 1:

[0363] The terminal presents the user with a login screen.

[0364] Step 2:

[0365] The user enters their email address and password on the login screen and clicks the "Register" button.

[0366] Step 3:

[0367] The terminal transmits the input information to the server.

[0368] Step 4:

[0369] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[0370] Step 5:

[0371] The server sends a registration completion message to the terminal.

[0372] Step 6:

[0373] The terminal displays a completion message to the user.

[0374] Training scenario selection

[0375] Step 1:

[0376] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[0377] Step 2:

[0378] The terminal displays a list of scenarios to the user.

[0379] Step 3:

[0380] The user selects the desired training scenario and clicks the "Start" button.

[0381] Step 4:

[0382] The terminal transmits scenario selection information to the server.

[0383] Step 5:

[0384] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[0385] Step 6:

[0386] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[0387] Simulation execution

[0388] Step 1:

[0389] The device displays the events in the scenario on the screen.

[0390] Step 2:

[0391] The user selects a response to the event and decides on the option.

[0392] Step 3:

[0393] The terminal transmits the user's selection to the server.

[0394] Step 4:

[0395] The server performs an evaluation based on the user's selection and generates an evaluation result.

[0396] Step 5:

[0397] The server transmits the evaluation results to the terminal.

[0398] Step 6:

[0399] The terminal displays the evaluation results to the user.

[0400] Role-playing session

[0401] Step 1:

[0402] The user selects a role-playing session and clicks the "Start" button.

[0403] Step 2:

[0404] The terminal transmits role-play selection information to the server.

[0405] Step 3:

[0406] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[0407] Step 4:

[0408] The device assigns a role to the user and displays role-based instructions and goals.

[0409] Step 5:

[0410] Users act and interact based on their assigned roles.

[0411] Step 6:

[0412] The device sends a record of the user's actions and interactions to the server.

[0413] Step 7:

[0414] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[0415] Step 8:

[0416] The server generates ratings and feedback and sends them to the device.

[0417] Step 9:

[0418] The device displays the rating and feedback to the user.

[0419] Evaluation and feedback generation

[0420] Step 1:

[0421] The server receives and analyzes all simulation and role-play recording data.

[0422] Step 2:

[0423] The server generates a user rating based on the analysis results.

[0424] Step 3:

[0425] The server generates feedback along with the evaluation results and sends it to the terminal.

[0426] Step 4:

[0427] The device displays the rating and feedback to the user.

[0428] Using the Emotion Engine

[0429] Step 1:

[0430] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[0431] Step 2:

[0432] The device sends the collected emotion data to a server, which stores it.

[0433] Step 3:

[0434] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[0435] Step 4:

[0436] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[0437] Step 5:

[0438] The device displays the difficulty of the scenario and feedback to the user.

[0439] Specific examples

[0440] 1. Example of user registration

[0441] Step 1:

[0442] The terminal displays a login screen to Mr. A, who is undergoing new employee training.

[0443] Step 2:

[0444] User A enters his email address and password on the login screen and clicks the Register button.

[0445] Step 3:

[0446] The terminal transmits the input information to the server.

[0447] Step 4:

[0448] Based on the information received by the server, a new user, Mr. A, is registered in the database. The uniqueness of the email address and the strength of the password are validated.

[0449] Step 5:

[0450] The server sends a registration completion message to the terminal.

[0451] Step 6:

[0452] The device displays a completion message to Person A.

[0453] 2. Specific examples of training scenario selection

[0454] Step 1:

[0455] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[0456] Step 2:

[0457] The device displays a list of scenarios to Person A.

[0458] Step 3:

[0459] User A selects the desired training scenario, "Customer Service Simulation," and clicks the start button.

[0460] Step 4:

[0461] The terminal transmits scenario selection information to the server.

[0462] Step 5:

[0463] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[0464] Step 6:

[0465] The device displays the scenario start screen to Person A and the simulation begins.

[0466] 3. Specific examples of simulation execution

[0467] Step 1:

[0468] The device displays the "Complaint handling" event in the scenario on the screen.

[0469] Step 2:

[0470] User A selects "Apologize" and confirms the option.

[0471] Step 3:

[0472] The device sends Mr. A's selection to the server.

[0473] Step 4:

[0474] The server evaluates based on Mr. A's selection and generates an evaluation result as an "appropriate response."

[0475] Step 5:

[0476] The server transmits the evaluation results to the terminal.

[0477] Step 6:

[0478] The device displays the evaluation results to Mr. A.

[0479] 4. Example of a role-play session

[0480] Step 1:

[0481] User A selects the "Sales Presentation" role-play and clicks the start button.

[0482] Step 2:

[0483] The terminal transmits role-play selection information to the server.

[0484] Step 3:

[0485] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[0486] Step 4:

[0487] The device assigns Person A the role of presentation, and Person A begins to take action and interact.

[0488] Step 5:

[0489] User A acts and interacts based on the role assigned to him.

[0490] Step 6:

[0491] The device sends records of A's actions and conversations to the server.

[0492] Step 7:

[0493] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[0494] Step 8:

[0495] The server generates ratings and feedback and sends them to the device.

[0496] Step 9:

[0497] The device displays the rating and feedback to Person A.

[0498] 5. Examples of using the Emotion Engine

[0499] Step 1:

[0500] The device recognizes emotions from Mr. A's facial expressions and voice during training and collects emotional data.

[0501] Step 2:

[0502] The device transmits the collected emotion data to a server.

[0503] Step 3:

[0504] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[0505] Step 4:

[0506] The server adjusts the difficulty of the scenario based on the emotional data and sends that information to the device.

[0507] Step 5:

[0508] The device displays the adjusted difficulty of the scenario and feedback to Person A. For example, if the emotion indicates "tension" or "stress," the server can lower the difficulty of the scenario and display instructions to Person A such as "Take a break here and relax."

[0509] This system provides individually customized training content that takes into account the user's emotions, enabling real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[0510] Example 2

[0511] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0512] Conventional training systems provide uniform scenarios and feedback without considering the user's feelings, making it difficult to provide optimal training for each individual user. Furthermore, they are unable to generate and display appropriate feedback in real time, which can limit the effectiveness of training.

[0513] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving registration information from a user and registering information of a new user in a database; means for transmitting scenario data based on a selection of a training scenario; means for receiving a user's selection for an event in the scenario and performing an evaluation based on the selection; means for transmitting data for role-playing and saving a record of the user's behavior; means for generating and displaying evaluation results and feedback to the user; and means for incorporating an emotion engine that recognizes the user's emotions, collecting emotion data, and reflecting it in the progression of the scenario and feedback. This enables individually customized training that incorporates the user's emotions, and allows appropriate feedback and scenario adjustments to be made in real time.

[0514] "Means for receiving registration information from users and registering new user information in the database" refers to a function that receives personal information and authentication information provided by users to the system, stores that information in the database, and registers new users.

[0515] The "means for transmitting scenario data based on the selection of a training scenario" is a function for transmitting data corresponding to the training scenario selected by the user from the server to the terminal.

[0516] The "means for receiving a user's selection for an event in a scenario and performing an evaluation based on that selection" is a function for receiving a user's selection for an event or action in a training scenario and evaluating the content of that selection.

[0517] "Means for transmitting data for role-playing and storing user behavior records" is a function for transmitting data for role-playing scenarios to the user's terminal and collecting and storing records of the user's behavior and interactions.

[0518] "Means for generating evaluation results and feedback and displaying them to the user" refers to a function that performs evaluation based on the user's actions and choices, generates the results and areas for improvement as feedback, and displays them on the user's device.

[0519] "A built-in emotion engine that recognizes the user's emotions, collects emotional data, and has the means to reflect it in the progression of the scenario and feedback" refers to a function that recognizes emotions in real time from the user's facial expressions and voice, collects and analyzes that emotional data, and reflects it in the progression of the training scenario and the content of the feedback.

[0520] MODE FOR CARRYING OUT THE INVENTION

[0521] System Overview

[0522] The training system of the present invention consists of multiple components, including a server, a terminal, and an emotion engine. The system aims to provide users with practical skills through simulations and role-playing, and to enhance the effectiveness of training by recognizing users' emotions in real time.

[0523] User registration method

[0524] The terminal displays a new registration screen and the user enters the required information (email address, password, etc.). The terminal then sends that information to the server, which registers the information as a new user in the database. Finally, the server sends a registration completion message to the terminal, which then displays it to the user. As a concrete example, when new trainee A enters his or her email address and password and presses the registration button, a message is displayed indicating that registration is complete.

[0525] Training scenario selection method

[0526] The server retrieves a list of available training scenarios from the database and sends them to the terminal for display. The user selects the desired training scenario, and the terminal sends the selection information to the server. The server then sends the scenario data to the terminal, and the terminal displays the scenario start screen. As a concrete example, when A selects "Customer Response Simulation" and clicks the start button, the scenario is displayed on the terminal.

[0527] Simulation execution method

[0528] When a training scenario starts, the terminal displays the events and tasks in the scenario to the user, and the user selects a response. The user's selection is sent from the terminal to the server, and the server performs an evaluation based on that selection. The evaluation results are sent from the server to the terminal, and the terminal displays them to the user. As a specific example, if a scenario about "handling a complaint" is displayed on A's terminal and A selects "apologize," the server evaluates and displays the results.

[0529] Role-playing tools

[0530] The user selects a role-playing session and clicks the start button. The device retrieves the role-playing data from the server and displays it to the user. The user acts and interacts based on the assigned role. The device sends a record of this to the server, which then saves the record. As a concrete example, A starts a "sales presentation" role-playing, and the content of the presentation is recorded and sent to the server for evaluation.

[0531] Assessment and feedback generation methods

[0532] The server evaluates the records of all simulations and role-plays and generates feedback. The generated feedback is sent from the server to the terminal and displayed to the user. For example, after handling a complaint or giving a sales presentation, A receives feedback such as "The presentation was well-structured."

[0533] Emotion Engine Means

[0534] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data. This data is sent from the device to a server, where it is stored and analyzed. The analysis results are reflected in the progression of the scenario and feedback. As a concrete example, if user A is feeling "tension" or "stress" during the simulation, the system will detect this and adjust the difficulty of the scenario, as well as displaying instructions such as "Take a break and relax here."

[0535] Hardware / Software used

[0536] This system consists of a device with a web browser (PC, tablet, smartphone, etc.) and a server that processes and stores data. The emotion engine uses facial expression recognition software and voice recognition software. Specific examples of emotion recognition software include Microsoft® Azure® Emotion API and Google® Cloud® Cloud Speech-to-Text.

[0537] Prompt Sentence Examples

[0538] In this system, examples of prompt sentences to be input to the generative AI model are as follows:

[0539] Please register a new user.

[0540] "Please start the customer interaction simulation."

[0541] "Generate appropriate feedback for complaint handling."

[0542] "Evaluate your sales presentation recording."

[0543] This system allows users to improve their skills by receiving real-time feedback that reflects their emotions.

[0544] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0545] Specific processing steps of the program

[0546] User Registration

[0547] Step 1:

[0548] The terminal displays a new registration screen, and the user enters the required information such as an email address and password.

[0549] Input: User's email address and password

[0550] Output: The device receives the input data

[0551] Step 2:

[0552] The terminal transmits the acquired registration information to the server.

[0553] Input: Registration information acquired by the device

[0554] Output: Registration information sent to the server

[0555] Step 3:

[0556] The server registers the new user in the database based on the received registration information.

[0557] Input: Registration information received by the server

[0558] Action: Add new user information to the database

[0559] Output: Registration complete message

[0560] Step 4:

[0561] The server sends a registration completion message to the terminal.

[0562] Input: Registration completion message

[0563] Output: Messages are displayed on the terminal

[0564] Training scenario selection

[0565] Step 1:

[0566] The server retrieves a list of available training scenarios from a database and transmits it to the terminal.

[0567] Input: Request to get scenario list

[0568] Process: Get the scenario list from the database

[0569] Output: Send the scenario list to the device.

[0570] Step 2:

[0571] The terminal displays the acquired scenario list to the user.

[0572] Input: Scenario list from the server

[0573] Output: The scenario list is displayed on the terminal.

[0574] Step 3:

[0575] The user selects the desired scenario.

[0576] Input: User scenario selection

[0577] Output: The device gets the selection information.

[0578] Step 4:

[0579] The terminal transmits the user's scenario selection information to the server.

[0580] Input: User scenario selection information

[0581] Output: Selection information is sent to the server

[0582] Step 5:

[0583] The server retrieves the selected scenario data from the database and transmits it to the terminal.

[0584] Input: Scenario selection information

[0585] Process: Retrieve scenario data from the database

[0586] Output: Sends scenario data to the device

[0587] Step 6:

[0588] The device displays the scenario start screen.

[0589] Input: Scenario data from the server

[0590] Output: The scenario start screen is displayed on the terminal.

[0591] Simulation execution

[0592] Step 1:

[0593] The terminal displays events in the scenario to the user, and the user operates to start the simulation.

[0594] Input: Scenario data received from the server

[0595] Output: Scenario events are displayed on the terminal

[0596] Step 2:

[0597] The user selects a response to an event in the scenario.

[0598] Input: Response selection for scenario events

[0599] Output: Selection data is retrieved to the terminal

[0600] Step 3:

[0601] The terminal transmits the user's selection data to the server.

[0602] Input: User selected data

[0603] Output: The selected data is sent to the server.

[0604] Step 4:

[0605] The server performs an evaluation based on the selected data and generates an evaluation result.

[0606] Input: User selected data

[0607] Processing: Evaluation and feedback generation based on evaluation algorithms

[0608] Output: Evaluation result

[0609] Step 5:

[0610] The server transmits the evaluation results to the terminal, which displays them to the user.

[0611] Input: Evaluation result from the server

[0612] Output: The evaluation result is displayed on the terminal.

[0613] role play

[0614] Step 1:

[0615] The user selects a role-playing session and clicks the "Start" button.

[0616] Input: User roleplay selection

[0617] Output: The device gets the selection information.

[0618] Step 2:

[0619] The terminal transmits the selection information to the server, and the server transmits the role-play data to the terminal.

[0620] Input: User's roleplay selection information

[0621] Output: Roleplay data is sent from the server to the device.

[0622] Step 3:

[0623] The device assigns a role to the user, and the user initiates an action or interaction.

[0624] Input: Roleplay data

[0625] Output: The roleplay screen is displayed on the device.

[0626] Step 4:

[0627] The terminal transmits the user's behavior record to the server.

[0628] Input: User activity log

[0629] Output: The activity record is sent to the server.

[0630] Evaluation and feedback generation

[0631] Step 1:

[0632] The server generates a comprehensive evaluation and feedback based on the simulation and role-play recording.

[0633] Input: Simulation and role-play recording

[0634] Processing: Evaluation by feedback generation algorithm

[0635] Output: Feedback

[0636] Step 2:

[0637] The server sends the generated feedback to the terminal, which displays it to the user.

[0638] Input: Generated feedback

[0639] Output: Feedback is displayed on the terminal

[0640] Emotion Engine

[0641] Step 1:

[0642] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[0643] Input: Data such as the user's facial expressions and voice

[0644] Output: Emotion data

[0645] Step 2:

[0646] The device transmits the collected emotion data to a server.

[0647] Input: Emotion data

[0648] Output: Emotion data is sent to the server

[0649] Step 3:

[0650] The server analyzes the emotional data and adjusts the progression of the scenario and feedback based on that.

[0651] Input: Emotion data

[0652] Processing: Analysis by sentiment analysis algorithm

[0653] Output: Scenario difficulty adjustment, feedback adjustment

[0654] Step 4:

[0655] The server transmits feedback and scenario information adjusted based on the emotion data to the terminal, which then displays it to the user.

[0656] Input: Tailored feedback and scenario information

[0657] Output: The adjusted information is displayed on the terminal.

[0658] (Application example 2)

[0659] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0660] Conventional new employee training systems provide uniform scenarios without considering the user's emotional state, making it difficult to provide detailed support and feedback tailored to each individual user's emotional state. Furthermore, even if a user is feeling stressed or tense, there is no mechanism for grasping that state in real time and adjusting the scenario appropriately, limiting the effectiveness of the training. This creates a need for real-time feedback tailored to the user's emotions, particularly for drivers of autonomous vehicles, who require practical skills.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0662] In this invention, the server includes means for receiving registration information from users and registering new user information in a database, means for transmitting scenario data based on training scenario selections, means for receiving user selections for events in the scenario and evaluating based on the selections, means for transmitting data for role-playing and saving a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, and means for recognizing emotions from the user's facial expressions and voice in real time, sending and saving the emotion data to the server, and reflecting it in adjusting the difficulty level of the scenario. This makes it possible to grasp the user's emotional state in real time and appropriately adjust the training scenario and feedback according to the state of each individual user.

[0663] The "user registration means" is a means for receiving registration information from a user and registering the information of a new user in the database.

[0664] The "scenario selection means" is a means for transmitting scenario data based on the selection of a training scenario.

[0665] The "evaluation means" is a means for receiving a user's selection for an event in a scenario and making an evaluation based on the selection.

[0666] The "behavior recording means" is a means for transmitting data for role-playing and storing a record of the user's behavior.

[0667] The "feedback generating means" is a means for generating evaluation results and feedback and displaying them to the user.

[0668] The "emotion recognition means" is a means for recognizing emotions from the user's facial expressions and voice in real time, transmitting and storing the emotion data on a server, and reflecting it in adjusting the difficulty level of the scenario.

[0669] A system for implementing this invention provides adaptive new employee training by recognizing a user's emotions in real time and appropriately adjusting training scenarios and feedback based on the emotions. This system operates by integrating multiple hardware and software components.

[0670] System configuration

[0671] 1. User registration method

[0672] The terminal receives the registration information from the user and sends it to the server, which registers the information as a new user in its database.

[0673] 2. Scenario Selection Method

[0674] The server retrieves training scenario data from the database and sends it to the terminal, which displays a list of scenarios for the user to select.

[0675] 3. Evaluation methods

[0676] The terminal receives the user's selection for an event in a scenario and transmits it to the server, which performs an evaluation based on the selection and generates an evaluation result.

[0677] 4. Behavioral Recording Method

[0678] The device transmits role-playing data and records the user's actions, and sends these records to the server, which stores them.

[0679] 5. Feedback Generation Methods

[0680] The server generates feedback based on the evaluation results, sends it to the terminal, and displays it to the user.

[0681] 6. Emotion recognition means

[0682] The device analyzes the user's facial expressions and voice data in real time to recognize their emotions. This emotional data is then sent to a server, which stores and analyzes it and adjusts the difficulty of the scenario.

[0683] Hardware and software used

[0684] Hardware: Camera (face recognition and facial expression analysis), microphone (voice analysis), CPU / GPU (computational processing)

[0685] Software: OpenCV (video capture and face recognition), Keras / TENSORFLOW (registered trademark) (emotion recognition model)

[0686] The main process of this system begins with the user entering their registration information and registering. Next, the user selects a training scenario from a list of scenarios displayed. Once the scenario begins, an evaluation is conducted based on the events selected by the user, and subsequent actions and interactions during role-playing are recorded. Based on these records and the evaluation results, the server generates feedback and displays it to the user. Furthermore, the device analyzes the user's facial and voice data, recognizes and collects emotions, and sends them to the server. The server adjusts the difficulty of the scenario based on the emotional data, providing a more adaptive training experience.

[0687] Specific examples

[0688] For example, in driver training for autonomous vehicles, the system analyzes the driver's facial expressions and voice in real time. If the driver shows signs of stress or tension, the system will lower the difficulty of the scenario and display a message to the driver to calm down. If the driver is relaxed, the system will provide instructions to continue the scenario.

[0689] Example prompt sentence:

[0690] Design a system to analyze the driver's emotions in real time for autonomous vehicles and improve driving safety based on those emotions. The system captures camera footage and analyzes the driver's emotions using facial recognition and emotion recognition models. Depending on the emotions, provide appropriate feedback to the driver in real time.

[0691] This system enables effective training that takes into account the user's emotional state, providing a safe and adaptive training experience, especially for drivers of autonomous vehicles.

[0692] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0693] Step 1:

[0694] The user operates the terminal and enters new user registration information (email address, password).

[0695] Input: The email address and password entered by the user

[0696] What happens: The device sends this information to the server.

[0697] Output: The server saves the registration information in the database and sends a registration completion message to the terminal.

[0698] Step 2:

[0699] The user operates the terminal to select a training scenario.

[0700] Input: User-selected training scenario

[0701] Operation: The device sends the selection information to the server, which retrieves the corresponding scenario data from the database and sends it to the device.

[0702] Output: The terminal displays the contents of the training scenario to the user.

[0703] Step 3:

[0704] The user selects options for events in the scenario.

[0705] Input: The event response selected by the user.

[0706] Operation: The device sends the selection information to the server, which evaluates the selection and generates an evaluation result.

[0707] Output: The server sends the generated evaluation results to the terminal, which displays them to the user.

[0708] Step 4:

[0709] Users perform role-plays and record their actions.

[0710] Input: User actions and interactions during roleplay

[0711] How it works: The device records actions and interactions and sends them to the server, which stores the recordings.

[0712] Output: The server evaluates the user based on the behavioral records stored and generates feedback.

[0713] Step 5:

[0714] The server generates feedback based on the evaluation results and sends it to the terminal.

[0715] Input: Evaluation result performed by the server

[0716] Operation: The server creates feedback based on the evaluation results and sends it to the device, which then displays the feedback to the user.

[0717] Output: Feedback displayed on the terminal

[0718] Step 6:

[0719] The device analyzes the user's facial expressions and voice to collect emotional data.

[0720] Input: User facial expression images and voice audio data

[0721] How it works: The device captures data using the camera and microphone, analyzes emotions in real time using a generative AI model (e.g., Keras / TensorFlow), and sends the emotion data to a server.

[0722] Output: Emotion data sent to the server

[0723] Step 7:

[0724] The server analyzes the user's emotional data and adjusts the difficulty of the scenario.

[0725] Input: Emotion data sent from the device

[0726] Operation: The server analyzes the emotional data and resets the difficulty level of the scenario to match the current state. It then sends the difficulty adjustment information to the device.

[0727] Output: The adjusted scenario displayed on the device

[0728] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0729] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0730] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0731] [Second embodiment]

[0732] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0733] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0734] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0735] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0736] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0737] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0738] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0739] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0740] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0741] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0742] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0743] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0744] System Overview

[0745] This invention is a system for supporting new employee training, allowing trainees to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and user terminals, and has the function of evaluating trainees' response capabilities through simulations and role-playing, and providing feedback.

[0746] System Details

[0747] The system is realized using a server and a terminal including the following means:

[0748] 1. User registration method

[0749] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[0750] The terminal sends the information to the server, which then registers the information as a new user in the database.

[0751] The server sends a registration completion message to the terminal, which displays it.

[0752] 2. Training scenario selection method

[0753] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[0754] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[0755] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[0756] The device displays the scenario start screen.

[0757] 3. Simulation execution method

[0758] The terminal displays events in the scenario on the screen, and the user selects a response.

[0759] The terminal transmits the user's selection to the server, which performs an evaluation based on the selection.

[0760] The server transmits the evaluation results to the terminal, which displays them to the user.

[0761] 4. Role-playing tools

[0762] The user selects a role-playing session and clicks the "Start" button.

[0763] The terminal transmits information to the server, and the server transmits data for role-playing to the terminal.

[0764] The device assigns a role to the user, and the user initiates an action or interaction.

[0765] The device sends the activity record to the server, which stores it.

[0766] 5. Evaluation and feedback generation methods

[0767] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[0768] The server sends the feedback to the terminal, which displays it to the user.

[0769] Specific examples

[0770] 1. Example of user registration

[0771] Tanaka, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[0772] The terminal sends this information to the server, and the server registers Tanaka's information in the database.

[0773] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Tanaka.

[0774] 2. Specific examples of training scenario selection

[0775] After completing registration, Tanaka logs in and a list of training scenarios is displayed. He selects "Customer Response Simulation."

[0776] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[0777] The device displays the scenario start screen, and Tanaka begins the simulation.

[0778] 3. Specific examples of simulation execution

[0779] During the simulation, the "Complaint handling" event is displayed on Tanaka's device.

[0780] Tanaka selects "Apologize" and the device sends the selection to the server.

[0781] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[0782] The device displays the evaluation results to Tanaka.

[0783] 4. Example of a role-play session

[0784] Tanaka begins the "sales presentation" role-play.

[0785] The device assigns Tanaka the role of presenter, and Tanaka actually gives the presentation.

[0786] The device records the content of Tanaka's presentation and sends it to the server.

[0787] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[0788] The device displays the evaluation results and feedback to Tanaka.

[0789] Throughout the system, users' actions are assessed and specific feedback is provided to help trainees improve their skills.

[0790] The processing flow will be explained below.

[0791] User Registration

[0792] Step 1:

[0793] The terminal presents the user with a login screen.

[0794] Step 2:

[0795] The user enters their email address and password on the login screen and clicks the "Register" button.

[0796] Step 3:

[0797] The terminal transmits the input information to the server.

[0798] Step 4:

[0799] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[0800] Step 5:

[0801] The server sends a registration completion message to the terminal.

[0802] Step 6:

[0803] The terminal displays a completion message to the user.

[0804] Training scenario selection

[0805] Step 1:

[0806] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[0807] Step 2:

[0808] The terminal displays a list of scenarios to the user.

[0809] Step 3:

[0810] The user selects the desired training scenario and clicks the "Start" button.

[0811] Step 4:

[0812] The terminal transmits scenario selection information to the server.

[0813] Step 5:

[0814] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[0815] Step 6:

[0816] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[0817] Simulation execution

[0818] Step 1:

[0819] The device displays the events in the scenario on the screen.

[0820] Step 2:

[0821] The user selects a response to the event and decides on the option.

[0822] Step 3:

[0823] The terminal transmits the user's selection to the server.

[0824] Step 4:

[0825] The server performs an evaluation based on the user's selection and generates an evaluation result.

[0826] Step 5:

[0827] The server transmits the evaluation results to the terminal.

[0828] Step 6:

[0829] The terminal displays the evaluation results to the user.

[0830] Role-playing session

[0831] Step 1:

[0832] The user selects a role-playing session and clicks the "Start" button.

[0833] Step 2:

[0834] The terminal transmits role-play selection information to the server.

[0835] Step 3:

[0836] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[0837] Step 4:

[0838] The device assigns a role to the user and displays role-based instructions and goals.

[0839] Step 5:

[0840] Users act and interact based on their assigned roles.

[0841] Step 6:

[0842] The device sends a record of the user's actions and interactions to the server.

[0843] Step 7:

[0844] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[0845] Step 8:

[0846] The server generates ratings and feedback and sends them to the device.

[0847] Step 9:

[0848] The device displays the rating and feedback to the user.

[0849] Evaluation and feedback generation

[0850] Step 1:

[0851] The server receives and analyzes all simulation and role-play recording data.

[0852] Step 2:

[0853] The server generates a user rating based on the analysis results.

[0854] Step 3:

[0855] The server generates feedback along with the evaluation results and sends it to the terminal.

[0856] Step 4:

[0857] The device displays the rating and feedback to the user.

[0858] Example 1

[0859] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0860] Conventional training systems have difficulty effectively evaluating trainees' skills and providing feedback under conditions equivalent to the actual work environment. Furthermore, it is difficult to ensure real-time and high-quality feedback during the evaluation and feedback generation process, making it difficult to provide specific advice necessary for improving trainees' skills.

[0861] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0862] In this invention, the server includes means for receiving registration information from a user and registering new user information in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and storing a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, and means for transmitting prompt sentences to a generative AI model to generate the evaluation results and feedback and receiving the generated feedback, thereby enabling high-quality feedback in real time to the trainee's behavior and selection.

[0863] "Registration information" refers to personal information, such as a user's email address and password, that is required to access and use the system.

[0864] The "database" is a system for centrally managing and storing data such as registration information, training scenarios, and evaluation results.

[0865] A "training scenario" is a storyline or content of a simulation or role-play for a specific task or situation that is selected and executed by a user.

[0866] "Scenario data" refers to detailed information and settings for constructing a training scenario, and is data that enables a scenario to be executed by a user.

[0867] An "event" is a specific situation or challenge that occurs within a training scenario, and is the subject of a user's response selection.

[0868] "Behavior record" is historical data of the actions and interactions selected by the user during a simulation or role-play.

[0869] The "evaluation results" are evaluation information of training results and skills generated based on the user's actions and selections.

[0870] "Feedback" refers to information such as specific advice and areas for improvement provided to the user based on the evaluation results.

[0871] A "generative AI model" is an artificial intelligence model used to generate evaluations and feedback on user actions and choices.

[0872] A "prompt" is an instruction given to a generative AI model that contains the information necessary to generate a specific evaluation or feedback.

[0873] This invention is a system for supporting new employee training, allowing users to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and terminals, and has the function of evaluating users' response capabilities through simulations and role-playing, and providing feedback.

[0874] System installation and operation

[0875] User Registration

[0876] When a user accesses the system for the first time, the terminal displays a user registration screen. The user enters registration information such as an email address and password, and the terminal sends this to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The terminal then displays a registration completion message to the user.

[0877] Selecting a training scenario

[0878] When a user logs in, the server retrieves a list of training scenarios from the database and sends it to the terminal. The terminal displays the list of scenarios to the user, and the user selects the desired scenario. The terminal sends this selection information to the server, and the server sends the selected scenario data to the terminal. The terminal displays the scenario start screen.

[0879] Running the simulation

[0880] When a scenario starts, the device displays the events in the scenario on the screen. When the user selects a response, the device sends the selection to the server. The server evaluates the selection and sends the evaluation results to the device. The device displays the evaluation results to the user.

[0881] Role-play execution

[0882] When a user selects a role-playing session and clicks the "Start" button, the device sends that information to the server. The server sends the role-playing data to the device, and the device assigns a role to the user. The user then gives an actual presentation or interacts, and the device sends a record of this behavior to the server. The server saves the behavior record.

[0883] Evaluation and feedback generation

[0884] The server evaluates the simulation and role-playing based on the recorded data and generates comprehensive feedback. The evaluation results and feedback are generated by a generative AI model through prompt sentences. The server sends the generated feedback to the device, which then displays it to the user.

[0885] Hardware and software used

[0886] This system consists of a server and a terminal used by the user. The server uses a database system (e.g., MySQL, PostgreSQL) and an AI model (e.g., GPT-3, BERT). The terminal uses a web browser to display the screen and input data.

[0887] Specific examples

[0888] For example, consider the case where a user selects "Customer Service Simulation." During the simulation, the "Complaint Handling" event is displayed, and the user selects "Apologize." At this time, the device sends this selection information to the server, and the server sends the following prompt to the AI ​​model:

[0889] "In a complaint handling simulation, the user selected 'Apologize.' Please evaluate the appropriateness of this response and generate feedback."

[0890] Based on this input, the AI ​​model generates feedback such as "This is an appropriate response. It is a choice that is highly effective in easing the customer's anger," and sends it to the device via the server. The device then displays this to the user.

[0891] The system allows users to receive high-quality feedback in real time, effectively improving their skills in a real-world work environment.

[0892] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0893] Step 1:

[0894] A user accesses the system. Specifically, the user opens a browser and enters the system's URL. The input is the URL, and the user registration screen is displayed as the output.

[0895] Step 2:

[0896] The terminal displays the user registration screen. Specifically, it renders the registration form using HTML and CSS. The input is the access information from step 1, and the output is the registration form screen displayed to the user.

[0897] Step 3:

[0898] The user enters registration information. Specifically, the user enters an email address, password, etc., and clicks the "Register" button. Input is the information entered by the user, and output is the data sent to the terminal.

[0899] Step 4:

[0900] The terminal sends the entered information to the server. Specifically, it sends data to the server using JavaScript or the form's POST method. The input is the user's registration information, and the output is the data sent to the server.

[0901] Step 5:

[0902] The server stores the received information in a database. Specifically, the data received on the server side is inserted into the database using SQL statements. The input is the data received from the terminal, and the output is the record stored in the database.

[0903] Step 6:

[0904] The server sends a registration completion message to the terminal. Specifically, the server returns a success message as a response. The input is the user information stored in the database, and the output is the completion message sent to the terminal.

[0905] Step 7:

[0906] The terminal displays a registration completion message to the user. Specifically, it displays a message in HTML to notify the user. The input is the completion message from the server, and the output is the message displayed to the user.

[0907] Step 8:

[0908] The server retrieves a list of training scenarios from the database and sends them to the terminal. Specifically, it uses SQL statements to select the scenario data and returns it in JSON format. The input is a database query, and the output is the scenario data sent to the terminal.

[0909] Step 9:

[0910] The device displays a list of scenarios to the user. Specifically, it uses JavaScript to draw the scenario list. The input is the scenario data received from the server, and the output is the scenario list displayed to the user.

[0911] Step 10:

[0912] The user selects a scenario. Specifically, the user clicks on a scenario from a dropdown or list. The input is the user's scenario selection, and the output is the selection information sent to the terminal.

[0913] Step 11:

[0914] The device sends the selection information to the server. Specifically, it uses AJAX or the Fetch API to send the selection information to the server. The input is the user's selection information, and the output is the data sent to the server.

[0915] Step 12:

[0916] The server sends the selected scenario data to the device. Specifically, the server returns the corresponding scenario data in JSON format. The input is the user's selection information, and the output is the scenario data sent to the device.

[0917] Step 13:

[0918] The terminal displays the scenario start screen. Specifically, it creates a screen based on the scenario data and displays it to the user. The input is the scenario data received from the server, and the output is the scenario start screen displayed to the user.

[0919] Step 14:

[0920] The device displays the events in the scenario on the screen. Specifically, it uses JavaScript to draw the event content. The input is the scenario data, and the output is the event content that is displayed to the user.

[0921] Step 15:

[0922] The user selects a response to an event. Specifically, the user clicks on an option. The input is the user's selection, and the output is the selection information sent to the terminal.

[0923] Step 16:

[0924] The device sends the user's selection to the server. Specifically, the selection information is sent using AJAX or the Fetch API. The input is the user's selection information, and the output is the data sent to the server.

[0925] Step 17:

[0926] The server evaluates the user's selection. Specifically, the server executes evaluation logic based on the options. The input is the user's selection information, and the output is the evaluation result.

[0927] Step 18:

[0928] The server sends the evaluation results to the terminal. Specifically, the evaluation results are returned in JSON format. The input is the evaluation result, and the output is the data sent to the terminal.

[0929] Step 19:

[0930] The terminal displays the evaluation results to the user. Specifically, the evaluation results are rendered in HTML and notified to the user. The input is the evaluation results received from the server, and the output is the evaluation results displayed to the user.

[0931] Step 20:

[0932] The user selects a role-playing session and clicks the "Start" button. Specifically, the user clicks a button on the screen. The input is the user's selection, and the output is the starting information sent to the terminal.

[0933] Step 21:

[0934] The device sends the information to the server. Specifically, it uses AJAX or the Fetch API to send data. The input is the user's initial information, and the output is the data sent to the server.

[0935] Step 22:

[0936] The server sends role-playing data to the device. Specifically, it retrieves data using SQL statements and returns it in JSON format. The input is the user's starting information, and the output is the role-playing data sent to the device.

[0937] Step 23:

[0938] The device assigns a role to the user. Specifically, it displays the role data on the screen using JavaScript. The input is role-play data, and the output is the role information displayed to the user.

[0939] Step 24:

[0940] The user acts and interacts. Specifically, the user acts according to the simulation. The input is the role-play scenario, and the output is the user's actions.

[0941] Step 25:

[0942] The device sends the behavioral record to the server. Specifically, the behavioral data is sent to the server via AJAX. The input is the user's behavioral data, and the output is the data sent to the server.

[0943] Step 26:

[0944] The server saves the behavioral records. Specifically, it inserts the recorded data into the database. The input is the behavioral data received from the device, and the output is the record saved in the database.

[0945] Step 27:

[0946] The server evaluates all simulation and role-play records. Specifically, the server executes the evaluation logic. The input is the recorded data of the simulation or role-play, and the output is the evaluation result.

[0947] Step 28:

[0948] The server sends a prompt to the generative AI model and generates feedback. Specifically, it sends the evaluation result as a prompt to the generative AI model and generates feedback. The input is the evaluation result data, and the output is the generated feedback.

[0949] Step 29:

[0950] The server sends feedback to the device. Specifically, it sends feedback data in JSON format. The input is the generated feedback, and the output is the data sent to the device.

[0951] Step 30:

[0952] The device displays the feedback to the user. Specifically, the feedback content is rendered in HTML and displayed to the user. The input is the feedback received from the server, and the output is the feedback displayed to the user.

[0953] (Application example 1)

[0954] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0955] There is a need for a system that allows trainees to efficiently acquire practical skills in the operation and maintenance of factory robots without actually using the robots on-site. Another challenge is to introduce a system that allows practical training without the risk of operational errors or safety risks. There is also a need for a flexible training environment that uses easily portable devices such as smartphones and tablets.

[0956] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0957] In this invention, the server includes means for receiving registration information from a user and registering information of a new user in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and saving a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, means for transmitting scenario data including procedures for operating and maintaining a factory robot, and means for running a simulation of a factory environment through an application installed on a smartphone or tablet device. This enables safe and efficient training in the operation and maintenance of a factory robot without using an actual robot.

[0958] The "user registration means" is a function that receives registration information from a user and registers the information of a new user in the database.

[0959] The "training scenario selection means" is a function that transmits scenario data based on the selection of a training scenario.

[0960] The "simulation execution means" is a function that receives a user's selection for an event in a scenario and performs an evaluation based on that selection.

[0961] "Role-playing means" is a function that transmits data for role-playing and stores a record of the user's actions.

[0962] The "evaluation and feedback generating means" is a function that generates evaluation results and feedback and displays them to the user.

[0963] The "factory environment simulation means" is a function that transmits scenario data including the operation and maintenance procedures of factory robots.

[0964] "Applications installed on smartphones and tablet devices" refers to software that is installed on these devices and runs simulations of factory environments.

[0965] This invention is a system that provides training for the operation and maintenance of factory robots, allowing participants to acquire practical skills safely and efficiently. The system operates in conjunction with a server and terminals, and provides functions for user registration, training scenario selection, simulation execution, role-playing, and evaluation and feedback. A factory environment simulation is run through an application installed on a smartphone or tablet device.

[0966] Hardware and Software

[0967] Hardware: Smartphones, tablets, database servers.

[0968] Software: Flask (a Python microframework), PostgreSQL (a database management system), HTML / CSS / JavaScript (for building the user interface).

[0969] System processing details

[0970] 1. User Registration

[0971] The server receives the registration information (e.g., email address, password) from the user and registers it as a new user in the database. When registration is complete, it sends a registration completion message to the terminal and displays the message to the user.

[0972] 2. Training scenario selection

[0973] The server sends a list of various scenarios (e.g., introduction to robot operation, regular maintenance, safety management training), which the terminal displays to the user. When the user selects the desired scenario, the information is sent to the server, and the selected scenario data is sent to the terminal.

[0974] 3. Run the simulation

[0975] The terminal executes a scenario of a factory environment, and the user selects the appropriate action for each event in the scenario. This selection is sent to the server for evaluation. The evaluation results are returned to the terminal as feedback and displayed to the user.

[0976] 4. Role-playing

[0977] The user selects a role-playing session, and the server sends the role-playing data to the device. The user's actions and interactions are recorded and sent to the server. The server evaluates the recording, generates feedback, and sends it to the device.

[0978] 5. Ratings and Feedback

[0979] The server will provide an overall evaluation and generate feedback based on the results of all simulations and role-plays, helping users understand specific areas for improvement and what went well.

[0980] Examples and prompts

[0981] Specific examples

[0982] User registration: A new user registers by entering their email address and password. The server receives this, stores it in the database, and sends a confirmation message.

[0983] Training scenario selection: After logging in, the user selects "Introduction to Robot Operation." The server sends the scenario data.

[0984] Simulation: The user is asked to perform an emergency stop of the robot. The server evaluates the action and sends feedback.

[0985] Role-play: Initiate a role-play of the maintenance procedure, record user actions, and provide evaluation and feedback.

[0986] Prompt Sentence Examples

[0987] Example of prompt text when user selects scenario:

[0988] "Please select one of the following training scenarios: 1. Introduction to Robot Operation 2. Routine Maintenance 3. Safety Management Training"

[0989] This system makes it possible to conduct training in the operation and maintenance of factory robots safely and efficiently.

[0990] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0991] Step 1:

[0992] User Registration

[0993] The terminal receives an email address and password from the user as input. It sends this information to the server. The server saves the new user's information in a database and generates a registration completion message. The server sends this message to the terminal, which displays it to the user. The input is the user's email address and password, and the output is the registration completion message.

[0994] Step 2:

[0995] Training scenario selection

[0996] After the user logs in, the terminal sends a scenario list request to the server. The server retrieves a list of training scenarios from the database and sends it to the terminal. The user selects the desired scenario from the list, and the terminal sends the selection information to the server. The server sends the selected scenario data to the terminal, which displays it to the user. The input is the scenario list request and scenario selection, and the output is the scenario data.

[0997] Step 3:

[0998] Simulation execution

[0999] When a user starts a scenario, the device displays the events in the scenario. The user selects an action for the event, and the device sends that selection to the server. The server receives the selection, runs an evaluation algorithm, and generates an evaluation result. The result is sent to the device, which displays the evaluation result to the user. The input is the user's selection, and the output is the evaluation result.

[1000] Step 4:

[1001] role play

[1002] When a user starts role-playing, the device requests role-playing data from the server. The server sends the data, and the device assigns a role to the user. The device records the user's actions and interactions and sends them to the server. The server evaluates the records, generates feedback results, and sends them to the device. The device displays the results to the user. The input is the user's action record, and the output is the feedback results.

[1003] Step 5:

[1004] Ratings and Feedback

[1005] The server performs a comprehensive evaluation based on the result data of all simulations and role-plays. It analyzes the user's operation records and behavioral data and evaluates them using an evaluation algorithm. The server generates comprehensive feedback and sends it to the device. The device displays the feedback to the user. The input is the operation records and behavioral data, and the output is comprehensive feedback.

[1006] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1007] System Overview

[1008] This invention is a new employee training system that enables trainees to acquire practical skills and incorporates an emotion engine that recognizes the user's emotions. This system operates in conjunction with a server and terminals, evaluating the trainee's ability to respond through simulations and role-playing and providing feedback. The emotion engine recognizes the user's emotions during training and reflects them in the progression of the scenario and feedback accordingly, thereby achieving effective, individually customized training.

[1009] System Details

[1010] The system is implemented using the following means:

[1011] 1. User registration method

[1012] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[1013] The terminal sends the information to the server, which then registers the information as a new user in the database.

[1014] The server sends a registration completion message to the terminal, which displays it.

[1015] 2. Training scenario selection method

[1016] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[1017] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[1018] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[1019] The device displays the scenario start screen.

[1020] 3. Simulation execution method

[1021] The terminal displays events in the scenario on the screen, and the user selects a response.

[1022] The terminal transmits the user's selection to the server.

[1023] The server performs an evaluation based on the user's selection and generates an evaluation result.

[1024] The server transmits the evaluation results to the terminal, which displays them to the user.

[1025] 4. Role-playing tools

[1026] The user selects a role-playing session and clicks the "Start" button.

[1027] The terminal transmits the selection information to the server, and the server transmits the role-play data to the terminal.

[1028] The device assigns a role to the user, and the user initiates an action or interaction.

[1029] The device sends the activity record to the server, which stores it.

[1030] 5. Evaluation and feedback generation methods

[1031] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[1032] The server sends the feedback to the terminal, which displays it to the user.

[1033] 6. Emotional Engine Means

[1034] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[1035] The device sends the collected emotion data to a server, which stores it.

[1036] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[1037] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[1038] Specific examples

[1039] 1. Example of user registration

[1040] Mr. A, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[1041] The terminal sends this information to the server, and the server registers Mr. A's information in the database.

[1042] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Mr. A.

[1043] 2. Specific examples of training scenario selection

[1044] After completing registration, Mr. A logs in and is presented with a list of training scenarios. He selects "Customer Service Simulation."

[1045] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[1046] The device displays the scenario start screen, and Person A begins the simulation.

[1047] 3. Specific examples of simulation execution

[1048] During the simulation, the "Complaint handling" event is displayed on Mr. A's device.

[1049] Person A selects "Apologize" and the device sends that selection to the server.

[1050] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[1051] The device displays the evaluation results to Mr. A.

[1052] 4. Example of a role-play session

[1053] Person A begins the "sales presentation" role-play.

[1054] The device assigns the role of presentation to Person A, who then gives the actual presentation.

[1055] The device records the content of Mr. A's presentation and sends it to the server.

[1056] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[1057] The device displays the evaluation results and feedback to Person A.

[1058] 5. Examples of using the Emotion Engine

[1059] During simulations and role-playing, the device recognizes emotions from Person A's facial expressions and voice and collects emotional data.

[1060] If the emotion indicates "tension" or "stress," the device sends it to the server.

[1061] The server analyzes the emotional data and adjusts the progress of the scenario accordingly.

[1062] For example, the server could lower the difficulty of the scenario based on emotional data and send appropriate feedback to Person A, such as "Take a break and relax here."

[1063] The device displays these instructions to Person A, prompting him to take appropriate action.

[1064] This system provides individually customized training content that takes into account the user's emotions, and allows for real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[1065] The processing flow will be explained below.

[1066] User Registration

[1067] Step 1:

[1068] The terminal presents the user with a login screen.

[1069] Step 2:

[1070] The user enters their email address and password on the login screen and clicks the "Register" button.

[1071] Step 3:

[1072] The terminal transmits the input information to the server.

[1073] Step 4:

[1074] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[1075] Step 5:

[1076] The server sends a registration completion message to the terminal.

[1077] Step 6:

[1078] The terminal displays a completion message to the user.

[1079] Training scenario selection

[1080] Step 1:

[1081] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[1082] Step 2:

[1083] The terminal displays a list of scenarios to the user.

[1084] Step 3:

[1085] The user selects the desired training scenario and clicks the "Start" button.

[1086] Step 4:

[1087] The terminal transmits scenario selection information to the server.

[1088] Step 5:

[1089] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[1090] Step 6:

[1091] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[1092] Simulation execution

[1093] Step 1:

[1094] The device displays the events in the scenario on the screen.

[1095] Step 2:

[1096] The user selects a response to the event and decides on the option.

[1097] Step 3:

[1098] The terminal transmits the user's selection to the server.

[1099] Step 4:

[1100] The server performs an evaluation based on the user's selection and generates an evaluation result.

[1101] Step 5:

[1102] The server transmits the evaluation results to the terminal.

[1103] Step 6:

[1104] The terminal displays the evaluation results to the user.

[1105] Role-playing session

[1106] Step 1:

[1107] The user selects a role-playing session and clicks the "Start" button.

[1108] Step 2:

[1109] The terminal transmits role-play selection information to the server.

[1110] Step 3:

[1111] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[1112] Step 4:

[1113] The device assigns a role to the user and displays role-based instructions and goals.

[1114] Step 5:

[1115] Users act and interact based on their assigned roles.

[1116] Step 6:

[1117] The device sends a record of the user's actions and interactions to the server.

[1118] Step 7:

[1119] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[1120] Step 8:

[1121] The server generates ratings and feedback and sends them to the device.

[1122] Step 9:

[1123] The device displays the rating and feedback to the user.

[1124] Evaluation and feedback generation

[1125] Step 1:

[1126] The server receives and analyzes all simulation and role-play recording data.

[1127] Step 2:

[1128] The server generates a user rating based on the analysis results.

[1129] Step 3:

[1130] The server generates feedback along with the evaluation results and sends it to the terminal.

[1131] Step 4:

[1132] The device displays the rating and feedback to the user.

[1133] Using the Emotion Engine

[1134] Step 1:

[1135] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[1136] Step 2:

[1137] The device sends the collected emotion data to a server, which stores it.

[1138] Step 3:

[1139] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[1140] Step 4:

[1141] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[1142] Step 5:

[1143] The device displays the difficulty of the scenario and feedback to the user.

[1144] Specific examples

[1145] 1. Example of user registration

[1146] Step 1:

[1147] The terminal displays a login screen to Mr. A, who is undergoing new employee training.

[1148] Step 2:

[1149] User A enters his email address and password on the login screen and clicks the Register button.

[1150] Step 3:

[1151] The terminal transmits the input information to the server.

[1152] Step 4:

[1153] Based on the information received by the server, a new user, Mr. A, is registered in the database. The uniqueness of the email address and the strength of the password are validated.

[1154] Step 5:

[1155] The server sends a registration completion message to the terminal.

[1156] Step 6:

[1157] The device displays a completion message to Person A.

[1158] 2. Specific examples of training scenario selection

[1159] Step 1:

[1160] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[1161] Step 2:

[1162] The device displays a list of scenarios to Person A.

[1163] Step 3:

[1164] User A selects the desired training scenario, "Customer Service Simulation," and clicks the start button.

[1165] Step 4:

[1166] The terminal transmits scenario selection information to the server.

[1167] Step 5:

[1168] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[1169] Step 6:

[1170] The device displays the scenario start screen to Person A and the simulation begins.

[1171] 3. Specific examples of simulation execution

[1172] Step 1:

[1173] The device displays the "Complaint handling" event in the scenario on the screen.

[1174] Step 2:

[1175] User A selects "Apologize" and confirms the option.

[1176] Step 3:

[1177] The device sends Mr. A's selection to the server.

[1178] Step 4:

[1179] The server evaluates based on Mr. A's selection and generates an evaluation result as an "appropriate response."

[1180] Step 5:

[1181] The server transmits the evaluation results to the terminal.

[1182] Step 6:

[1183] The device displays the evaluation results to Mr. A.

[1184] 4. Example of a role-play session

[1185] Step 1:

[1186] User A selects the "Sales Presentation" role-play and clicks the start button.

[1187] Step 2:

[1188] The terminal transmits role-play selection information to the server.

[1189] Step 3:

[1190] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[1191] Step 4:

[1192] The device assigns Person A the role of presentation, and Person A begins to take action and interact.

[1193] Step 5:

[1194] User A acts and interacts based on the role assigned to him.

[1195] Step 6:

[1196] The device sends records of A's actions and conversations to the server.

[1197] Step 7:

[1198] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[1199] Step 8:

[1200] The server generates ratings and feedback and sends them to the device.

[1201] Step 9:

[1202] The device displays the rating and feedback to Person A.

[1203] 5. Examples of using the Emotion Engine

[1204] Step 1:

[1205] The device recognizes emotions from Mr. A's facial expressions and voice during training and collects emotional data.

[1206] Step 2:

[1207] The device transmits the collected emotion data to a server.

[1208] Step 3:

[1209] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[1210] Step 4:

[1211] The server adjusts the difficulty of the scenario based on the emotional data and sends that information to the device.

[1212] Step 5:

[1213] The device displays the adjusted difficulty of the scenario and feedback to Person A. For example, if the emotion indicates "tension" or "stress," the server can lower the difficulty of the scenario and display instructions to Person A such as "Take a break here and relax."

[1214] This system provides individually customized training content that takes into account the user's emotions, enabling real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[1215] Example 2

[1216] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1217] Conventional training systems provide uniform scenarios and feedback without considering the user's feelings, making it difficult to provide optimal training for each individual user. Furthermore, they are unable to generate and display appropriate feedback in real time, which can limit the effectiveness of training.

[1218] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving registration information from a user and registering information of a new user in a database; means for transmitting scenario data based on a selection of a training scenario; means for receiving a user's selection for an event in the scenario and performing an evaluation based on the selection; means for transmitting data for role-playing and saving a record of the user's behavior; means for generating and displaying evaluation results and feedback to the user; and means for incorporating an emotion engine that recognizes the user's emotions, collecting emotion data, and reflecting it in the progression of the scenario and feedback. This enables individually customized training that incorporates the user's emotions, and allows appropriate feedback and scenario adjustments to be made in real time.

[1219] "Means for receiving registration information from users and registering new user information in the database" refers to a function that receives personal information and authentication information provided by users to the system, stores that information in the database, and registers new users.

[1220] The "means for transmitting scenario data based on the selection of a training scenario" is a function for transmitting data corresponding to the training scenario selected by the user from the server to the terminal.

[1221] The "means for receiving a user's selection for an event in a scenario and performing an evaluation based on that selection" is a function for receiving a user's selection for an event or action in a training scenario and evaluating the content of that selection.

[1222] "Means for transmitting data for role-playing and storing user behavior records" is a function for transmitting data for role-playing scenarios to the user's terminal and collecting and storing records of the user's behavior and interactions.

[1223] "Means for generating evaluation results and feedback and displaying them to the user" refers to a function that performs evaluation based on the user's actions and choices, generates the results and areas for improvement as feedback, and displays them on the user's device.

[1224] "A built-in emotion engine that recognizes the user's emotions, collects emotional data, and has the means to reflect it in the progression of the scenario and feedback" refers to a function that recognizes emotions in real time from the user's facial expressions and voice, collects and analyzes that emotional data, and reflects it in the progression of the training scenario and the content of the feedback.

[1225] MODE FOR CARRYING OUT THE INVENTION

[1226] System Overview

[1227] The training system of the present invention consists of multiple components, including a server, a terminal, and an emotion engine. The system aims to provide users with practical skills through simulations and role-playing, and to enhance the effectiveness of training by recognizing users' emotions in real time.

[1228] User registration method

[1229] The terminal displays a new registration screen and the user enters the required information (email address, password, etc.). The terminal then sends that information to the server, which registers the information as a new user in the database. Finally, the server sends a registration completion message to the terminal, which then displays it to the user. As a concrete example, when new trainee A enters his or her email address and password and presses the registration button, a message is displayed indicating that registration is complete.

[1230] Training scenario selection method

[1231] The server retrieves a list of available training scenarios from the database and sends them to the terminal for display. The user selects the desired training scenario, and the terminal sends the selection information to the server. The server then sends the scenario data to the terminal, and the terminal displays the scenario start screen. As a concrete example, when A selects "Customer Response Simulation" and clicks the start button, the scenario is displayed on the terminal.

[1232] Simulation execution method

[1233] When a training scenario starts, the terminal displays the events and tasks in the scenario to the user, and the user selects a response. The user's selection is sent from the terminal to the server, and the server performs an evaluation based on that selection. The evaluation results are sent from the server to the terminal, and the terminal displays them to the user. As a specific example, if a scenario about "handling a complaint" is displayed on A's terminal and A selects "apologize," the server evaluates and displays the results.

[1234] Role-playing tools

[1235] The user selects a role-playing session and clicks the start button. The device retrieves the role-playing data from the server and displays it to the user. The user acts and interacts based on the assigned role. The device sends a record of this to the server, which then saves the record. As a concrete example, A starts a "sales presentation" role-playing, and the content of the presentation is recorded and sent to the server for evaluation.

[1236] Assessment and feedback generation methods

[1237] The server evaluates the records of all simulations and role-plays and generates feedback. The generated feedback is sent from the server to the terminal and displayed to the user. For example, after handling a complaint or giving a sales presentation, A receives feedback such as "The presentation was well-structured."

[1238] Emotion Engine Means

[1239] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data. This data is sent from the device to a server, where it is stored and analyzed. The analysis results are reflected in the progression of the scenario and feedback. As a concrete example, if user A is feeling "tension" or "stress" during the simulation, the system will detect this and adjust the difficulty of the scenario, as well as displaying instructions such as "Take a break and relax here."

[1240] Hardware / Software used

[1241] This system consists of a device with a web browser (PC, tablet, smartphone, etc.) and a server that processes and stores data. The emotion engine uses facial expression recognition software and voice recognition software. Specifically, the emotion recognition software could use Microsoft Azure's Emotion API or Google Cloud's Cloud Speech-to-Text.

[1242] Prompt Sentence Examples

[1243] In this system, examples of prompt sentences to be input to the generative AI model are as follows:

[1244] Please register a new user.

[1245] "Please start the customer interaction simulation."

[1246] "Generate appropriate feedback for complaint handling."

[1247] "Evaluate your sales presentation recording."

[1248] This system allows users to improve their skills by receiving real-time feedback that reflects their emotions.

[1249] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1250] Specific processing steps of the program

[1251] User Registration

[1252] Step 1:

[1253] The terminal displays a new registration screen, and the user enters the required information such as an email address and password.

[1254] Input: User's email address and password

[1255] Output: The device receives the input data

[1256] Step 2:

[1257] The terminal transmits the acquired registration information to the server.

[1258] Input: Registration information acquired by the device

[1259] Output: Registration information sent to the server

[1260] Step 3:

[1261] The server registers the new user in the database based on the received registration information.

[1262] Input: Registration information received by the server

[1263] Action: Add new user information to the database

[1264] Output: Registration complete message

[1265] Step 4:

[1266] The server sends a registration completion message to the terminal.

[1267] Input: Registration completion message

[1268] Output: Messages are displayed on the terminal

[1269] Training scenario selection

[1270] Step 1:

[1271] The server retrieves a list of available training scenarios from a database and transmits it to the terminal.

[1272] Input: Request to get scenario list

[1273] Process: Get the scenario list from the database

[1274] Output: Send the scenario list to the device.

[1275] Step 2:

[1276] The terminal displays the acquired scenario list to the user.

[1277] Input: Scenario list from the server

[1278] Output: The scenario list is displayed on the terminal.

[1279] Step 3:

[1280] The user selects the desired scenario.

[1281] Input: User scenario selection

[1282] Output: The device gets the selection information.

[1283] Step 4:

[1284] The terminal transmits the user's scenario selection information to the server.

[1285] Input: User scenario selection information

[1286] Output: Selection information is sent to the server

[1287] Step 5:

[1288] The server retrieves the selected scenario data from the database and transmits it to the terminal.

[1289] Input: Scenario selection information

[1290] Process: Retrieve scenario data from the database

[1291] Output: Sends scenario data to the device

[1292] Step 6:

[1293] The device displays the scenario start screen.

[1294] Input: Scenario data from the server

[1295] Output: The scenario start screen is displayed on the terminal.

[1296] Simulation execution

[1297] Step 1:

[1298] The terminal displays events in the scenario to the user, and the user operates to start the simulation.

[1299] Input: Scenario data received from the server

[1300] Output: Scenario events are displayed on the terminal

[1301] Step 2:

[1302] The user selects a response to an event in the scenario.

[1303] Input: Response selection for scenario events

[1304] Output: Selection data is retrieved to the terminal

[1305] Step 3:

[1306] The terminal transmits the user's selection data to the server.

[1307] Input: User selected data

[1308] Output: The selected data is sent to the server.

[1309] Step 4:

[1310] The server performs an evaluation based on the selected data and generates an evaluation result.

[1311] Input: User selected data

[1312] Processing: Evaluation and feedback generation based on evaluation algorithms

[1313] Output: Evaluation result

[1314] Step 5:

[1315] The server transmits the evaluation results to the terminal, which displays them to the user.

[1316] Input: Evaluation result from the server

[1317] Output: The evaluation result is displayed on the terminal.

[1318] role play

[1319] Step 1:

[1320] The user selects a role-playing session and clicks the "Start" button.

[1321] Input: User roleplay selection

[1322] Output: The device gets the selection information.

[1323] Step 2:

[1324] The terminal transmits the selection information to the server, and the server transmits the role-play data to the terminal.

[1325] Input: User's roleplay selection information

[1326] Output: Roleplay data is sent from the server to the device.

[1327] Step 3:

[1328] The device assigns a role to the user, and the user initiates an action or interaction.

[1329] Input: Roleplay data

[1330] Output: The roleplay screen is displayed on the device.

[1331] Step 4:

[1332] The terminal transmits the user's behavior record to the server.

[1333] Input: User activity log

[1334] Output: The activity record is sent to the server.

[1335] Evaluation and feedback generation

[1336] Step 1:

[1337] The server generates a comprehensive evaluation and feedback based on the simulation and role-play recording.

[1338] Input: Simulation and role-play recording

[1339] Processing: Evaluation by feedback generation algorithm

[1340] Output: Feedback

[1341] Step 2:

[1342] The server sends the generated feedback to the terminal, which displays it to the user.

[1343] Input: Generated feedback

[1344] Output: Feedback is displayed on the terminal

[1345] Emotion Engine

[1346] Step 1:

[1347] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[1348] Input: Data such as the user's facial expressions and voice

[1349] Output: Emotion data

[1350] Step 2:

[1351] The device transmits the collected emotion data to a server.

[1352] Input: Emotion data

[1353] Output: Emotion data is sent to the server

[1354] Step 3:

[1355] The server analyzes the emotional data and adjusts the progression of the scenario and feedback based on that.

[1356] Input: Emotion data

[1357] Processing: Analysis by sentiment analysis algorithm

[1358] Output: Scenario difficulty adjustment, feedback adjustment

[1359] Step 4:

[1360] The server transmits feedback and scenario information adjusted based on the emotion data to the terminal, which then displays it to the user.

[1361] Input: Tailored feedback and scenario information

[1362] Output: The adjusted information is displayed on the terminal.

[1363] (Application example 2)

[1364] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1365] Conventional new employee training systems provide uniform scenarios without considering the user's emotional state, making it difficult to provide detailed support and feedback tailored to each individual user's emotional state. Furthermore, even if a user is feeling stressed or tense, there is no mechanism for grasping that state in real time and adjusting the scenario appropriately, limiting the effectiveness of the training. This creates a need for real-time feedback tailored to the user's emotions, particularly for drivers of autonomous vehicles, who require practical skills.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1367] In this invention, the server includes means for receiving registration information from users and registering new user information in a database, means for transmitting scenario data based on training scenario selections, means for receiving user selections for events in the scenario and evaluating based on the selections, means for transmitting data for role-playing and saving a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, and means for recognizing emotions from the user's facial expressions and voice in real time, sending and saving the emotion data to the server, and reflecting it in adjusting the difficulty level of the scenario. This makes it possible to grasp the user's emotional state in real time and appropriately adjust the training scenario and feedback according to the state of each individual user.

[1368] The "user registration means" is a means for receiving registration information from a user and registering the information of a new user in the database.

[1369] The "scenario selection means" is a means for transmitting scenario data based on the selection of a training scenario.

[1370] The "evaluation means" is a means for receiving a user's selection for an event in a scenario and making an evaluation based on the selection.

[1371] The "behavior recording means" is a means for transmitting data for role-playing and storing a record of the user's behavior.

[1372] The "feedback generating means" is a means for generating evaluation results and feedback and displaying them to the user.

[1373] The "emotion recognition means" is a means for recognizing emotions from the user's facial expressions and voice in real time, transmitting and storing the emotion data on a server, and reflecting it in adjusting the difficulty level of the scenario.

[1374] A system for implementing this invention provides adaptive new employee training by recognizing a user's emotions in real time and appropriately adjusting training scenarios and feedback based on the emotions. This system operates by integrating multiple hardware and software components.

[1375] System configuration

[1376] 1. User registration method

[1377] The terminal receives the registration information from the user and sends it to the server, which registers the information as a new user in its database.

[1378] 2. Scenario Selection Method

[1379] The server retrieves training scenario data from the database and sends it to the terminal, which displays a list of scenarios for the user to select.

[1380] 3. Evaluation methods

[1381] The terminal receives the user's selection for an event in a scenario and transmits it to the server, which performs an evaluation based on the selection and generates an evaluation result.

[1382] 4. Behavioral Recording Method

[1383] The device transmits role-playing data and records the user's actions, and sends these records to the server, which stores them.

[1384] 5. Feedback Generation Methods

[1385] The server generates feedback based on the evaluation results, sends it to the terminal, and displays it to the user.

[1386] 6. Emotion recognition means

[1387] The device analyzes the user's facial expressions and voice data in real time to recognize their emotions. This emotional data is then sent to a server, which stores and analyzes it and adjusts the difficulty of the scenario.

[1388] Hardware and software used

[1389] Hardware: Camera (face recognition and facial expression analysis), microphone (voice analysis), CPU / GPU (computational processing)

[1390] Software: OpenCV (video capture and face recognition), Keras / TensorFlow (emotion recognition model)

[1391] The main process of this system begins with the user entering their registration information and registering. Next, the user selects a training scenario from a list of scenarios displayed. Once the scenario begins, an evaluation is conducted based on the events selected by the user, and subsequent actions and interactions during role-playing are recorded. Based on these records and the evaluation results, the server generates feedback and displays it to the user. Furthermore, the device analyzes the user's facial and voice data, recognizes and collects emotions, and sends them to the server. The server adjusts the difficulty of the scenario based on the emotional data, providing a more adaptive training experience.

[1392] Specific examples

[1393] For example, in driver training for autonomous vehicles, the system analyzes the driver's facial expressions and voice in real time. If the driver shows signs of stress or tension, the system will lower the difficulty of the scenario and display a message to the driver to calm down. If the driver is relaxed, the system will provide instructions to continue the scenario.

[1394] Example prompt sentence:

[1395] Design a system to analyze the driver's emotions in real time for autonomous vehicles and improve driving safety based on those emotions. The system captures camera footage and analyzes the driver's emotions using facial recognition and emotion recognition models. Depending on the emotions, provide appropriate feedback to the driver in real time.

[1396] This system enables effective training that takes into account the user's emotional state, providing a safe and adaptive training experience, especially for drivers of autonomous vehicles.

[1397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1398] Step 1:

[1399] The user operates the terminal and enters new user registration information (email address, password).

[1400] Input: The email address and password entered by the user

[1401] What happens: The device sends this information to the server.

[1402] Output: The server saves the registration information in the database and sends a registration completion message to the terminal.

[1403] Step 2:

[1404] The user operates the terminal to select a training scenario.

[1405] Input: User-selected training scenario

[1406] Operation: The device sends the selection information to the server, which retrieves the corresponding scenario data from the database and sends it to the device.

[1407] Output: The terminal displays the contents of the training scenario to the user.

[1408] Step 3:

[1409] The user selects options for events in the scenario.

[1410] Input: The event response selected by the user.

[1411] Operation: The device sends the selection information to the server, which evaluates the selection and generates an evaluation result.

[1412] Output: The server sends the generated evaluation results to the terminal, which displays them to the user.

[1413] Step 4:

[1414] Users perform role-plays and record their actions.

[1415] Input: User actions and interactions during roleplay

[1416] How it works: The device records actions and interactions and sends them to the server, which stores the recordings.

[1417] Output: The server evaluates the user based on the behavioral records stored and generates feedback.

[1418] Step 5:

[1419] The server generates feedback based on the evaluation results and sends it to the terminal.

[1420] Input: Evaluation result performed by the server

[1421] Operation: The server creates feedback based on the evaluation results and sends it to the device, which then displays the feedback to the user.

[1422] Output: Feedback displayed on the terminal

[1423] Step 6:

[1424] The device analyzes the user's facial expressions and voice to collect emotional data.

[1425] Input: User facial expression images and voice audio data

[1426] How it works: The device captures data using the camera and microphone, analyzes emotions in real time using a generative AI model (e.g., Keras / TensorFlow), and sends the emotion data to a server.

[1427] Output: Emotion data sent to the server

[1428] Step 7:

[1429] The server analyzes the user's emotional data and adjusts the difficulty of the scenario.

[1430] Input: Emotion data sent from the device

[1431] Operation: The server analyzes the emotional data and resets the difficulty level of the scenario to match the current state. It then sends the difficulty adjustment information to the device.

[1432] Output: The adjusted scenario displayed on the device

[1433] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1434] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1435] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1436] [Third embodiment]

[1437] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1438] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1439] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1440] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1441] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1443] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1444] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1445] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1446] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1447] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1448] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1449] System Overview

[1450] This invention is a system for supporting new employee training, allowing trainees to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and user terminals, and has the function of evaluating trainees' response capabilities through simulations and role-playing, and providing feedback.

[1451] System Details

[1452] The system is realized using a server and a terminal including the following means:

[1453] 1. User registration method

[1454] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[1455] The terminal sends the information to the server, which then registers the information as a new user in the database.

[1456] The server sends a registration completion message to the terminal, which displays it.

[1457] 2. Training scenario selection method

[1458] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[1459] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[1460] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[1461] The device displays the scenario start screen.

[1462] 3. Simulation execution method

[1463] The terminal displays events in the scenario on the screen, and the user selects a response.

[1464] The terminal transmits the user's selection to the server, which performs an evaluation based on the selection.

[1465] The server transmits the evaluation results to the terminal, which displays them to the user.

[1466] 4. Role-playing tools

[1467] The user selects a role-playing session and clicks the "Start" button.

[1468] The terminal transmits information to the server, and the server transmits data for role-playing to the terminal.

[1469] The device assigns a role to the user, and the user initiates an action or interaction.

[1470] The device sends the activity record to the server, which stores it.

[1471] 5. Evaluation and feedback generation methods

[1472] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[1473] The server sends the feedback to the terminal, which displays it to the user.

[1474] Specific examples

[1475] 1. Example of user registration

[1476] Tanaka, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[1477] The terminal sends this information to the server, and the server registers Tanaka's information in the database.

[1478] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Tanaka.

[1479] 2. Specific examples of training scenario selection

[1480] After completing registration, Tanaka logs in and a list of training scenarios is displayed. He selects "Customer Response Simulation."

[1481] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[1482] The device displays the scenario start screen, and Tanaka begins the simulation.

[1483] 3. Specific examples of simulation execution

[1484] During the simulation, the "Complaint handling" event is displayed on Tanaka's device.

[1485] Tanaka selects "Apologize" and the device sends the selection to the server.

[1486] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[1487] The device displays the evaluation results to Tanaka.

[1488] 4. Example of a role-play session

[1489] Tanaka begins the "sales presentation" role-play.

[1490] The device assigns Tanaka the role of presenter, and Tanaka actually gives the presentation.

[1491] The device records the content of Tanaka's presentation and sends it to the server.

[1492] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[1493] The device displays the evaluation results and feedback to Tanaka.

[1494] Throughout the system, users' actions are assessed and specific feedback is provided to help trainees improve their skills.

[1495] The processing flow will be explained below.

[1496] User Registration

[1497] Step 1:

[1498] The terminal presents the user with a login screen.

[1499] Step 2:

[1500] The user enters their email address and password on the login screen and clicks the "Register" button.

[1501] Step 3:

[1502] The terminal transmits the input information to the server.

[1503] Step 4:

[1504] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[1505] Step 5:

[1506] The server sends a registration completion message to the terminal.

[1507] Step 6:

[1508] The terminal displays a completion message to the user.

[1509] Training scenario selection

[1510] Step 1:

[1511] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[1512] Step 2:

[1513] The terminal displays a list of scenarios to the user.

[1514] Step 3:

[1515] The user selects the desired training scenario and clicks the "Start" button.

[1516] Step 4:

[1517] The terminal transmits scenario selection information to the server.

[1518] Step 5:

[1519] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[1520] Step 6:

[1521] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[1522] Simulation execution

[1523] Step 1:

[1524] The device displays the events in the scenario on the screen.

[1525] Step 2:

[1526] The user selects a response to the event and decides on the option.

[1527] Step 3:

[1528] The terminal transmits the user's selection to the server.

[1529] Step 4:

[1530] The server performs an evaluation based on the user's selection and generates an evaluation result.

[1531] Step 5:

[1532] The server transmits the evaluation results to the terminal.

[1533] Step 6:

[1534] The terminal displays the evaluation results to the user.

[1535] Role-playing session

[1536] Step 1:

[1537] The user selects a role-playing session and clicks the "Start" button.

[1538] Step 2:

[1539] The terminal transmits role-play selection information to the server.

[1540] Step 3:

[1541] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[1542] Step 4:

[1543] The device assigns a role to the user and displays role-based instructions and goals.

[1544] Step 5:

[1545] Users act and interact based on their assigned roles.

[1546] Step 6:

[1547] The device sends a record of the user's actions and interactions to the server.

[1548] Step 7:

[1549] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[1550] Step 8:

[1551] The server generates ratings and feedback and sends them to the device.

[1552] Step 9:

[1553] The device displays the rating and feedback to the user.

[1554] Evaluation and feedback generation

[1555] Step 1:

[1556] The server receives and analyzes all simulation and role-play recording data.

[1557] Step 2:

[1558] The server generates a user rating based on the analysis results.

[1559] Step 3:

[1560] The server generates feedback along with the evaluation results and sends it to the terminal.

[1561] Step 4:

[1562] The device displays the rating and feedback to the user.

[1563] Example 1

[1564] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1565] Conventional training systems have difficulty effectively evaluating trainees' skills and providing feedback under conditions equivalent to the actual work environment. Furthermore, it is difficult to ensure real-time and high-quality feedback during the evaluation and feedback generation process, making it difficult to provide specific advice necessary for improving trainees' skills.

[1566] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1567] In this invention, the server includes means for receiving registration information from a user and registering new user information in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and storing a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, and means for transmitting prompt sentences to a generative AI model to generate the evaluation results and feedback and receiving the generated feedback, thereby enabling high-quality feedback in real time to the trainee's behavior and selection.

[1568] "Registration information" refers to personal information, such as a user's email address and password, that is required to access and use the system.

[1569] The "database" is a system for centrally managing and storing data such as registration information, training scenarios, and evaluation results.

[1570] A "training scenario" is a storyline or content of a simulation or role-play for a specific task or situation that is selected and executed by a user.

[1571] "Scenario data" refers to detailed information and settings for constructing a training scenario, and is data that enables a scenario to be executed by a user.

[1572] An "event" is a specific situation or challenge that occurs within a training scenario, and is the subject of a user's response selection.

[1573] "Behavior record" is historical data of the actions and interactions selected by the user during a simulation or role-play.

[1574] The "evaluation results" are evaluation information of training results and skills generated based on the user's actions and selections.

[1575] "Feedback" refers to information such as specific advice and areas for improvement provided to the user based on the evaluation results.

[1576] A "generative AI model" is an artificial intelligence model used to generate evaluations and feedback on user actions and choices.

[1577] A "prompt" is an instruction given to a generative AI model that contains the information necessary to generate a specific evaluation or feedback.

[1578] This invention is a system for supporting new employee training, allowing users to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and terminals, and has the function of evaluating users' response capabilities through simulations and role-playing, and providing feedback.

[1579] System installation and operation

[1580] User Registration

[1581] When a user accesses the system for the first time, the terminal displays a user registration screen. The user enters registration information such as an email address and password, and the terminal sends this to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The terminal then displays a registration completion message to the user.

[1582] Selecting a training scenario

[1583] When a user logs in, the server retrieves a list of training scenarios from the database and sends it to the terminal. The terminal displays the list of scenarios to the user, and the user selects the desired scenario. The terminal sends this selection information to the server, and the server sends the selected scenario data to the terminal. The terminal displays the scenario start screen.

[1584] Running the simulation

[1585] When a scenario starts, the device displays the events in the scenario on the screen. When the user selects a response, the device sends the selection to the server. The server evaluates the selection and sends the evaluation results to the device. The device displays the evaluation results to the user.

[1586] Role-play execution

[1587] When a user selects a role-playing session and clicks the "Start" button, the device sends that information to the server. The server sends the role-playing data to the device, and the device assigns a role to the user. The user then gives an actual presentation or interacts, and the device sends a record of this behavior to the server. The server saves the behavior record.

[1588] Evaluation and feedback generation

[1589] The server evaluates the simulation and role-playing based on the recorded data and generates comprehensive feedback. The evaluation results and feedback are generated by a generative AI model through prompt sentences. The server sends the generated feedback to the device, which then displays it to the user.

[1590] Hardware and software used

[1591] This system consists of a server and a terminal used by the user. The server uses a database system (e.g., MySQL, PostgreSQL) and an AI model (e.g., GPT-3, BERT). The terminal uses a web browser to display the screen and input data.

[1592] Specific examples

[1593] For example, consider the case where a user selects "Customer Service Simulation." During the simulation, the "Complaint Handling" event is displayed, and the user selects "Apologize." At this time, the device sends this selection information to the server, and the server sends the following prompt to the AI ​​model:

[1594] "In a complaint handling simulation, the user selected 'Apologize.' Please evaluate the appropriateness of this response and generate feedback."

[1595] Based on this input, the AI ​​model generates feedback such as "This is an appropriate response. It is a choice that is highly effective in easing the customer's anger," and sends it to the device via the server. The device then displays this to the user.

[1596] The system allows users to receive high-quality feedback in real time, effectively improving their skills in a real-world work environment.

[1597] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1598] Step 1:

[1599] A user accesses the system. Specifically, the user opens a browser and enters the system's URL. The input is the URL, and the user registration screen is displayed as the output.

[1600] Step 2:

[1601] The terminal displays the user registration screen. Specifically, it renders the registration form using HTML and CSS. The input is the access information from step 1, and the output is the registration form screen displayed to the user.

[1602] Step 3:

[1603] The user enters registration information. Specifically, the user enters an email address, password, etc., and clicks the "Register" button. Input is the information entered by the user, and output is the data sent to the terminal.

[1604] Step 4:

[1605] The terminal sends the entered information to the server. Specifically, it sends data to the server using JavaScript or the form's POST method. The input is the user's registration information, and the output is the data sent to the server.

[1606] Step 5:

[1607] The server stores the received information in a database. Specifically, the data received on the server side is inserted into the database using SQL statements. The input is the data received from the terminal, and the output is the record stored in the database.

[1608] Step 6:

[1609] The server sends a registration completion message to the terminal. Specifically, the server returns a success message as a response. The input is the user information stored in the database, and the output is the completion message sent to the terminal.

[1610] Step 7:

[1611] The terminal displays a registration completion message to the user. Specifically, it displays a message in HTML to notify the user. The input is the completion message from the server, and the output is the message displayed to the user.

[1612] Step 8:

[1613] The server retrieves a list of training scenarios from the database and sends them to the terminal. Specifically, it uses SQL statements to select the scenario data and returns it in JSON format. The input is a database query, and the output is the scenario data sent to the terminal.

[1614] Step 9:

[1615] The device displays a list of scenarios to the user. Specifically, it uses JavaScript to draw the scenario list. The input is the scenario data received from the server, and the output is the scenario list displayed to the user.

[1616] Step 10:

[1617] The user selects a scenario. Specifically, the user clicks on a scenario from a dropdown or list. The input is the user's scenario selection, and the output is the selection information sent to the terminal.

[1618] Step 11:

[1619] The device sends the selection information to the server. Specifically, it uses AJAX or the Fetch API to send the selection information to the server. The input is the user's selection information, and the output is the data sent to the server.

[1620] Step 12:

[1621] The server sends the selected scenario data to the device. Specifically, the server returns the corresponding scenario data in JSON format. The input is the user's selection information, and the output is the scenario data sent to the device.

[1622] Step 13:

[1623] The terminal displays the scenario start screen. Specifically, it creates a screen based on the scenario data and displays it to the user. The input is the scenario data received from the server, and the output is the scenario start screen displayed to the user.

[1624] Step 14:

[1625] The device displays the events in the scenario on the screen. Specifically, it uses JavaScript to draw the event content. The input is the scenario data, and the output is the event content that is displayed to the user.

[1626] Step 15:

[1627] The user selects a response to an event. Specifically, the user clicks on an option. The input is the user's selection, and the output is the selection information sent to the terminal.

[1628] Step 16:

[1629] The device sends the user's selection to the server. Specifically, the selection information is sent using AJAX or the Fetch API. The input is the user's selection information, and the output is the data sent to the server.

[1630] Step 17:

[1631] The server evaluates the user's selection. Specifically, the server executes evaluation logic based on the options. The input is the user's selection information, and the output is the evaluation result.

[1632] Step 18:

[1633] The server sends the evaluation results to the terminal. Specifically, the evaluation results are returned in JSON format. The input is the evaluation result, and the output is the data sent to the terminal.

[1634] Step 19:

[1635] The terminal displays the evaluation results to the user. Specifically, the evaluation results are rendered in HTML and notified to the user. The input is the evaluation results received from the server, and the output is the evaluation results displayed to the user.

[1636] Step 20:

[1637] The user selects a role-playing session and clicks the "Start" button. Specifically, the user clicks a button on the screen. The input is the user's selection, and the output is the starting information sent to the terminal.

[1638] Step 21:

[1639] The device sends the information to the server. Specifically, it uses AJAX or the Fetch API to send data. The input is the user's initial information, and the output is the data sent to the server.

[1640] Step 22:

[1641] The server sends role-playing data to the device. Specifically, it retrieves data using SQL statements and returns it in JSON format. The input is the user's starting information, and the output is the role-playing data sent to the device.

[1642] Step 23:

[1643] The device assigns a role to the user. Specifically, it displays the role data on the screen using JavaScript. The input is role-play data, and the output is the role information displayed to the user.

[1644] Step 24:

[1645] The user acts and interacts. Specifically, the user acts according to the simulation. The input is the role-play scenario, and the output is the user's actions.

[1646] Step 25:

[1647] The device sends the behavioral record to the server. Specifically, the behavioral data is sent to the server via AJAX. The input is the user's behavioral data, and the output is the data sent to the server.

[1648] Step 26:

[1649] The server saves the behavioral records. Specifically, it inserts the recorded data into the database. The input is the behavioral data received from the device, and the output is the record saved in the database.

[1650] Step 27:

[1651] The server evaluates all simulation and role-play records. Specifically, the server executes the evaluation logic. The input is the recorded data of the simulation or role-play, and the output is the evaluation result.

[1652] Step 28:

[1653] The server sends a prompt to the generative AI model and generates feedback. Specifically, it sends the evaluation result as a prompt to the generative AI model and generates feedback. The input is the evaluation result data, and the output is the generated feedback.

[1654] Step 29:

[1655] The server sends feedback to the device. Specifically, it sends feedback data in JSON format. The input is the generated feedback, and the output is the data sent to the device.

[1656] Step 30:

[1657] The device displays the feedback to the user. Specifically, the feedback content is rendered in HTML and displayed to the user. The input is the feedback received from the server, and the output is the feedback displayed to the user.

[1658] (Application example 1)

[1659] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1660] There is a need for a system that allows trainees to efficiently acquire practical skills in the operation and maintenance of factory robots without actually using the robots on-site. Another challenge is to introduce a system that allows practical training without the risk of operational errors or safety risks. There is also a need for a flexible training environment that uses easily portable devices such as smartphones and tablets.

[1661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1662] In this invention, the server includes means for receiving registration information from a user and registering information of a new user in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and saving a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, means for transmitting scenario data including procedures for operating and maintaining a factory robot, and means for running a simulation of a factory environment through an application installed on a smartphone or tablet device. This enables safe and efficient training in the operation and maintenance of a factory robot without using an actual robot.

[1663] The "user registration means" is a function that receives registration information from a user and registers the information of a new user in the database.

[1664] The "training scenario selection means" is a function that transmits scenario data based on the selection of a training scenario.

[1665] The "simulation execution means" is a function that receives a user's selection for an event in a scenario and performs an evaluation based on that selection.

[1666] "Role-playing means" is a function that transmits data for role-playing and stores a record of the user's actions.

[1667] The "evaluation and feedback generating means" is a function that generates evaluation results and feedback and displays them to the user.

[1668] The "factory environment simulation means" is a function that transmits scenario data including the operation and maintenance procedures of factory robots.

[1669] "Applications installed on smartphones and tablet devices" refers to software that is installed on these devices and runs simulations of factory environments.

[1670] This invention is a system that provides training for the operation and maintenance of factory robots, allowing participants to acquire practical skills safely and efficiently. The system operates in conjunction with a server and terminals, and provides functions for user registration, training scenario selection, simulation execution, role-playing, and evaluation and feedback. A factory environment simulation is run through an application installed on a smartphone or tablet device.

[1671] Hardware and Software

[1672] Hardware: Smartphones, tablets, database servers.

[1673] Software: Flask (a Python microframework), PostgreSQL (a database management system), HTML / CSS / JavaScript (for building the user interface).

[1674] System processing details

[1675] 1. User Registration

[1676] The server receives the registration information (e.g., email address, password) from the user and registers it as a new user in the database. When registration is complete, it sends a registration completion message to the terminal and displays the message to the user.

[1677] 2. Training scenario selection

[1678] The server sends a list of various scenarios (e.g., introduction to robot operation, regular maintenance, safety management training), which the terminal displays to the user. When the user selects the desired scenario, the information is sent to the server, and the selected scenario data is sent to the terminal.

[1679] 3. Run the simulation

[1680] The terminal executes a scenario of a factory environment, and the user selects the appropriate action for each event in the scenario. This selection is sent to the server for evaluation. The evaluation results are returned to the terminal as feedback and displayed to the user.

[1681] 4. Role-playing

[1682] The user selects a role-playing session, and the server sends the role-playing data to the device. The user's actions and interactions are recorded and sent to the server. The server evaluates the recording, generates feedback, and sends it to the device.

[1683] 5. Ratings and Feedback

[1684] The server will provide an overall evaluation and generate feedback based on the results of all simulations and role-plays, helping users understand specific areas for improvement and what went well.

[1685] Examples and prompts

[1686] Specific examples

[1687] User registration: A new user registers by entering their email address and password. The server receives this, stores it in the database, and sends a confirmation message.

[1688] Training scenario selection: After logging in, the user selects "Introduction to Robot Operation." The server sends the scenario data.

[1689] Simulation: The user is asked to perform an emergency stop of the robot. The server evaluates the action and sends feedback.

[1690] Role-play: Initiate a role-play of the maintenance procedure, record user actions, and provide evaluation and feedback.

[1691] Prompt Sentence Examples

[1692] Example of prompt text when user selects scenario:

[1693] "Please select one of the following training scenarios: 1. Introduction to Robot Operation 2. Routine Maintenance 3. Safety Management Training"

[1694] This system makes it possible to conduct training in the operation and maintenance of factory robots safely and efficiently.

[1695] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1696] Step 1:

[1697] User Registration

[1698] The terminal receives an email address and password from the user as input. It sends this information to the server. The server saves the new user's information in a database and generates a registration completion message. The server sends this message to the terminal, which displays it to the user. The input is the user's email address and password, and the output is the registration completion message.

[1699] Step 2:

[1700] Training scenario selection

[1701] After the user logs in, the terminal sends a scenario list request to the server. The server retrieves a list of training scenarios from the database and sends it to the terminal. The user selects the desired scenario from the list, and the terminal sends the selection information to the server. The server sends the selected scenario data to the terminal, which displays it to the user. The input is the scenario list request and scenario selection, and the output is the scenario data.

[1702] Step 3:

[1703] Simulation execution

[1704] When a user starts a scenario, the device displays the events in the scenario. The user selects an action for the event, and the device sends that selection to the server. The server receives the selection, runs an evaluation algorithm, and generates an evaluation result. The result is sent to the device, which displays the evaluation result to the user. The input is the user's selection, and the output is the evaluation result.

[1705] Step 4:

[1706] role play

[1707] When a user starts role-playing, the device requests role-playing data from the server. The server sends the data, and the device assigns a role to the user. The device records the user's actions and interactions and sends them to the server. The server evaluates the records, generates feedback results, and sends them to the device. The device displays the results to the user. The input is the user's action record, and the output is the feedback results.

[1708] Step 5:

[1709] Ratings and Feedback

[1710] The server performs a comprehensive evaluation based on the result data of all simulations and role-plays. It analyzes the user's operation records and behavioral data and evaluates them using an evaluation algorithm. The server generates comprehensive feedback and sends it to the device. The device displays the feedback to the user. The input is the operation records and behavioral data, and the output is comprehensive feedback.

[1711] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1712] System Overview

[1713] This invention is a new employee training system that enables trainees to acquire practical skills and incorporates an emotion engine that recognizes the user's emotions. This system operates in conjunction with a server and terminals, evaluating the trainee's ability to respond through simulations and role-playing and providing feedback. The emotion engine recognizes the user's emotions during training and reflects them in the progression of the scenario and feedback accordingly, thereby achieving effective, individually customized training.

[1714] System Details

[1715] The system is implemented using the following means:

[1716] 1. User registration method

[1717] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[1718] The terminal sends the information to the server, which then registers the information as a new user in the database.

[1719] The server sends a registration completion message to the terminal, which displays it.

[1720] 2. Training scenario selection method

[1721] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[1722] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[1723] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[1724] The device displays the scenario start screen.

[1725] 3. Simulation execution method

[1726] The terminal displays events in the scenario on the screen, and the user selects a response.

[1727] The terminal transmits the user's selection to the server.

[1728] The server performs an evaluation based on the user's selection and generates an evaluation result.

[1729] The server transmits the evaluation results to the terminal, which displays them to the user.

[1730] 4. Role-playing tools

[1731] The user selects a role-playing session and clicks the "Start" button.

[1732] The terminal transmits the selection information to the server, and the server transmits the role-play data to the terminal.

[1733] The device assigns a role to the user, and the user initiates an action or interaction.

[1734] The device sends the activity record to the server, which stores it.

[1735] 5. Evaluation and feedback generation methods

[1736] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[1737] The server sends the feedback to the terminal, which displays it to the user.

[1738] 6. Emotional Engine Means

[1739] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[1740] The device sends the collected emotion data to a server, which stores it.

[1741] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[1742] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[1743] Specific examples

[1744] 1. Example of user registration

[1745] Mr. A, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[1746] The terminal sends this information to the server, and the server registers Mr. A's information in the database.

[1747] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Mr. A.

[1748] 2. Specific examples of training scenario selection

[1749] After completing registration, Mr. A logs in and is presented with a list of training scenarios. He selects "Customer Service Simulation."

[1750] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[1751] The device displays the scenario start screen, and Person A begins the simulation.

[1752] 3. Specific examples of simulation execution

[1753] During the simulation, the "Complaint handling" event is displayed on Mr. A's device.

[1754] Person A selects "Apologize" and the device sends that selection to the server.

[1755] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[1756] The device displays the evaluation results to Mr. A.

[1757] 4. Example of a role-play session

[1758] Person A begins the "sales presentation" role-play.

[1759] The device assigns the role of presentation to Person A, who then gives the actual presentation.

[1760] The device records the content of Mr. A's presentation and sends it to the server.

[1761] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[1762] The device displays the evaluation results and feedback to Person A.

[1763] 5. Examples of using the Emotion Engine

[1764] During simulations and role-playing, the device recognizes emotions from Person A's facial expressions and voice and collects emotional data.

[1765] If the emotion indicates "tension" or "stress," the device sends it to the server.

[1766] The server analyzes the emotional data and adjusts the progress of the scenario accordingly.

[1767] For example, the server could lower the difficulty of the scenario based on emotional data and send appropriate feedback to Person A, such as "Take a break and relax here."

[1768] The device displays these instructions to Person A, prompting him to take appropriate action.

[1769] This system provides individually customized training content that takes into account the user's emotions, and allows for real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[1770] The processing flow will be explained below.

[1771] User Registration

[1772] Step 1:

[1773] The terminal presents the user with a login screen.

[1774] Step 2:

[1775] The user enters their email address and password on the login screen and clicks the "Register" button.

[1776] Step 3:

[1777] The terminal transmits the input information to the server.

[1778] Step 4:

[1779] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[1780] Step 5:

[1781] The server sends a registration completion message to the terminal.

[1782] Step 6:

[1783] The terminal displays a completion message to the user.

[1784] Training scenario selection

[1785] Step 1:

[1786] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[1787] Step 2:

[1788] The terminal displays a list of scenarios to the user.

[1789] Step 3:

[1790] The user selects the desired training scenario and clicks the "Start" button.

[1791] Step 4:

[1792] The terminal transmits scenario selection information to the server.

[1793] Step 5:

[1794] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[1795] Step 6:

[1796] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[1797] Simulation execution

[1798] Step 1:

[1799] The device displays the events in the scenario on the screen.

[1800] Step 2:

[1801] The user selects a response to the event and decides on the option.

[1802] Step 3:

[1803] The terminal transmits the user's selection to the server.

[1804] Step 4:

[1805] The server performs an evaluation based on the user's selection and generates an evaluation result.

[1806] Step 5:

[1807] The server transmits the evaluation results to the terminal.

[1808] Step 6:

[1809] The terminal displays the evaluation results to the user.

[1810] Role-playing session

[1811] Step 1:

[1812] The user selects a role-playing session and clicks the "Start" button.

[1813] Step 2:

[1814] The terminal transmits role-play selection information to the server.

[1815] Step 3:

[1816] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[1817] Step 4:

[1818] The device assigns a role to the user and displays role-based instructions and goals.

[1819] Step 5:

[1820] Users act and interact based on their assigned roles.

[1821] Step 6:

[1822] The device sends a record of the user's actions and interactions to the server.

[1823] Step 7:

[1824] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[1825] Step 8:

[1826] The server generates ratings and feedback and sends them to the device.

[1827] Step 9:

[1828] The device displays the rating and feedback to the user.

[1829] Evaluation and feedback generation

[1830] Step 1:

[1831] The server receives and analyzes all simulation and role-play recording data.

[1832] Step 2:

[1833] The server generates a user rating based on the analysis results.

[1834] Step 3:

[1835] The server generates feedback along with the evaluation results and sends it to the terminal.

[1836] Step 4:

[1837] The device displays the rating and feedback to the user.

[1838] Using the Emotion Engine

[1839] Step 1:

[1840] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[1841] Step 2:

[1842] The device sends the collected emotion data to a server, which stores it.

[1843] Step 3:

[1844] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[1845] Step 4:

[1846] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[1847] Step 5:

[1848] The device displays the difficulty of the scenario and feedback to the user.

[1849] Specific examples

[1850] 1. Example of user registration

[1851] Step 1:

[1852] The terminal displays a login screen to Mr. A, who is undergoing new employee training.

[1853] Step 2:

[1854] User A enters his email address and password on the login screen and clicks the Register button.

[1855] Step 3:

[1856] The terminal transmits the input information to the server.

[1857] Step 4:

[1858] Based on the information received by the server, a new user, Mr. A, is registered in the database. The uniqueness of the email address and the strength of the password are validated.

[1859] Step 5:

[1860] The server sends a registration completion message to the terminal.

[1861] Step 6:

[1862] The device displays a completion message to Person A.

[1863] 2. Specific examples of training scenario selection

[1864] Step 1:

[1865] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[1866] Step 2:

[1867] The device displays a list of scenarios to Person A.

[1868] Step 3:

[1869] User A selects the desired training scenario, "Customer Service Simulation," and clicks the start button.

[1870] Step 4:

[1871] The terminal transmits scenario selection information to the server.

[1872] Step 5:

[1873] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[1874] Step 6:

[1875] The device displays the scenario start screen to Person A and the simulation begins.

[1876] 3. Specific examples of simulation execution

[1877] Step 1:

[1878] The device displays the "Complaint handling" event in the scenario on the screen.

[1879] Step 2:

[1880] User A selects "Apologize" and confirms the option.

[1881] Step 3:

[1882] The device sends Mr. A's selection to the server.

[1883] Step 4:

[1884] The server evaluates based on Mr. A's selection and generates an evaluation result as an "appropriate response."

[1885] Step 5:

[1886] The server transmits the evaluation results to the terminal.

[1887] Step 6:

[1888] The device displays the evaluation results to Mr. A.

[1889] 4. Example of a role-play session

[1890] Step 1:

[1891] User A selects the "Sales Presentation" role-play and clicks the start button.

[1892] Step 2:

[1893] The terminal transmits role-play selection information to the server.

[1894] Step 3:

[1895] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[1896] Step 4:

[1897] The device assigns Person A the role of presentation, and Person A begins to take action and interact.

[1898] Step 5:

[1899] User A acts and interacts based on the role assigned to him.

[1900] Step 6:

[1901] The device sends records of A's actions and conversations to the server.

[1902] Step 7:

[1903] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[1904] Step 8:

[1905] The server generates ratings and feedback and sends them to the device.

[1906] Step 9:

[1907] The device displays the rating and feedback to Person A.

[1908] 5. Examples of using the Emotion Engine

[1909] Step 1:

[1910] The device recognizes emotions from Mr. A's facial expressions and voice during training and collects emotional data.

[1911] Step 2:

[1912] The device transmits the collected emotion data to a server.

[1913] Step 3:

[1914] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[1915] Step 4:

[1916] The server adjusts the difficulty of the scenario based on the emotional data and sends that information to the device.

[1917] Step 5:

[1918] The device displays the adjusted difficulty of the scenario and feedback to Person A. For example, if the emotion indicates "tension" or "stress," the server can lower the difficulty of the scenario and display instructions to Person A such as "Take a break here and relax."

[1919] This system provides individually customized training content that takes into account the user's emotions, enabling real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[1920] Example 2

[1921] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1922] Conventional training systems provide uniform scenarios and feedback without considering the user's feelings, making it difficult to provide optimal training for each individual user. Furthermore, they are unable to generate and display appropriate feedback in real time, which can limit the effectiveness of training.

[1923] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving registration information from a user and registering information of a new user in a database; means for transmitting scenario data based on a selection of a training scenario; means for receiving a user's selection for an event in the scenario and performing an evaluation based on the selection; means for transmitting data for role-playing and saving a record of the user's behavior; means for generating and displaying evaluation results and feedback to the user; and means for incorporating an emotion engine that recognizes the user's emotions, collecting emotion data, and reflecting it in the progression of the scenario and feedback. This enables individually customized training that incorporates the user's emotions, and allows appropriate feedback and scenario adjustments to be made in real time.

[1924] "Means for receiving registration information from users and registering new user information in the database" refers to a function that receives personal information and authentication information provided by users to the system, stores that information in the database, and registers new users.

[1925] The "means for transmitting scenario data based on the selection of a training scenario" is a function for transmitting data corresponding to the training scenario selected by the user from the server to the terminal.

[1926] The "means for receiving a user's selection for an event in a scenario and performing an evaluation based on that selection" is a function for receiving a user's selection for an event or action in a training scenario and evaluating the content of that selection.

[1927] "Means for transmitting data for role-playing and storing user behavior records" is a function for transmitting data for role-playing scenarios to the user's terminal and collecting and storing records of the user's behavior and interactions.

[1928] "Means for generating evaluation results and feedback and displaying them to the user" refers to a function that performs evaluation based on the user's actions and choices, generates the results and areas for improvement as feedback, and displays them on the user's device.

[1929] "A built-in emotion engine that recognizes the user's emotions, collects emotional data, and has the means to reflect it in the progression of the scenario and feedback" refers to a function that recognizes emotions in real time from the user's facial expressions and voice, collects and analyzes that emotional data, and reflects it in the progression of the training scenario and the content of the feedback.

[1930] MODE FOR CARRYING OUT THE INVENTION

[1931] System Overview

[1932] The training system of the present invention consists of multiple components, including a server, a terminal, and an emotion engine. The system aims to provide users with practical skills through simulations and role-playing, and to enhance the effectiveness of training by recognizing users' emotions in real time.

[1933] User registration method

[1934] The terminal displays a new registration screen and the user enters the required information (email address, password, etc.). The terminal then sends that information to the server, which registers the information as a new user in the database. Finally, the server sends a registration completion message to the terminal, which then displays it to the user. As a concrete example, when new trainee A enters his or her email address and password and presses the registration button, a message is displayed indicating that registration is complete.

[1935] Training scenario selection method

[1936] The server retrieves a list of available training scenarios from the database and sends them to the terminal for display. The user selects the desired training scenario, and the terminal sends the selection information to the server. The server then sends the scenario data to the terminal, and the terminal displays the scenario start screen. As a concrete example, when A selects "Customer Response Simulation" and clicks the start button, the scenario is displayed on the terminal.

[1937] Simulation execution method

[1938] When a training scenario starts, the terminal displays the events and tasks in the scenario to the user, and the user selects a response. The user's selection is sent from the terminal to the server, and the server performs an evaluation based on that selection. The evaluation results are sent from the server to the terminal, and the terminal displays them to the user. As a specific example, if a scenario about "handling a complaint" is displayed on A's terminal and A selects "apologize," the server evaluates and displays the results.

[1939] Role-playing tools

[1940] The user selects a role-playing session and clicks the start button. The device retrieves the role-playing data from the server and displays it to the user. The user acts and interacts based on the assigned role. The device sends a record of this to the server, which then saves the record. As a concrete example, A starts a "sales presentation" role-playing, and the content of the presentation is recorded and sent to the server for evaluation.

[1941] Assessment and feedback generation methods

[1942] The server evaluates the records of all simulations and role-plays and generates feedback. The generated feedback is sent from the server to the terminal and displayed to the user. For example, after handling a complaint or giving a sales presentation, A receives feedback such as "The presentation was well-structured."

[1943] Emotion Engine Means

[1944] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data. This data is sent from the device to a server, where it is stored and analyzed. The analysis results are reflected in the progression of the scenario and feedback. As a concrete example, if user A is feeling "tension" or "stress" during the simulation, the system will detect this and adjust the difficulty of the scenario, as well as displaying instructions such as "Take a break and relax here."

[1945] Hardware / Software used

[1946] This system consists of a device with a web browser (PC, tablet, smartphone, etc.) and a server that processes and stores data. The emotion engine uses facial expression recognition software and voice recognition software. Specifically, the emotion recognition software could use Microsoft Azure's Emotion API or Google Cloud's Cloud Speech-to-Text.

[1947] Prompt Sentence Examples

[1948] In this system, examples of prompt sentences to be input to the generative AI model are as follows:

[1949] Please register a new user.

[1950] "Please start the customer interaction simulation."

[1951] "Generate appropriate feedback for complaint handling."

[1952] "Evaluate your sales presentation recording."

[1953] This system allows users to improve their skills by receiving real-time feedback that reflects their emotions.

[1954] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1955] Specific processing steps of the program

[1956] User Registration

[1957] Step 1:

[1958] The terminal displays a new registration screen, and the user enters the required information such as an email address and password.

[1959] Input: User's email address and password

[1960] Output: The device receives the input data

[1961] Step 2:

[1962] The terminal transmits the acquired registration information to the server.

[1963] Input: Registration information acquired by the device

[1964] Output: Registration information sent to the server

[1965] Step 3:

[1966] The server registers the new user in the database based on the received registration information.

[1967] Input: Registration information received by the server

[1968] Action: Add new user information to the database

[1969] Output: Registration complete message

[1970] Step 4:

[1971] The server sends a registration completion message to the terminal.

[1972] Input: Registration completion message

[1973] Output: Messages are displayed on the terminal

[1974] Training scenario selection

[1975] Step 1:

[1976] The server retrieves a list of available training scenarios from a database and transmits it to the terminal.

[1977] Input: Request to get scenario list

[1978] Process: Get the scenario list from the database

[1979] Output: Send the scenario list to the device.

[1980] Step 2:

[1981] The terminal displays the acquired scenario list to the user.

[1982] Input: Scenario list from the server

[1983] Output: The scenario list is displayed on the terminal.

[1984] Step 3:

[1985] The user selects the desired scenario.

[1986] Input: User scenario selection

[1987] Output: The device gets the selection information.

[1988] Step 4:

[1989] The terminal transmits the user's scenario selection information to the server.

[1990] Input: User scenario selection information

[1991] Output: Selection information is sent to the server

[1992] Step 5:

[1993] The server retrieves the selected scenario data from the database and transmits it to the terminal.

[1994] Input: Scenario selection information

[1995] Process: Retrieve scenario data from the database

[1996] Output: Sends scenario data to the device

[1997] Step 6:

[1998] The device displays the scenario start screen.

[1999] Input: Scenario data from the server

[2000] Output: The scenario start screen is displayed on the terminal.

[2001] Simulation execution

[2002] Step 1:

[2003] The terminal displays events in the scenario to the user, and the user operates to start the simulation.

[2004] Input: Scenario data received from the server

[2005] Output: Scenario events are displayed on the terminal

[2006] Step 2:

[2007] The user selects a response to an event in the scenario.

[2008] Input: Response selection for scenario events

[2009] Output: Selection data is retrieved to the terminal

[2010] Step 3:

[2011] The terminal transmits the user's selection data to the server.

[2012] Input: User selected data

[2013] Output: The selected data is sent to the server.

[2014] Step 4:

[2015] The server performs an evaluation based on the selected data and generates an evaluation result.

[2016] Input: User selected data

[2017] Processing: Evaluation and feedback generation based on evaluation algorithms

[2018] Output: Evaluation result

[2019] Step 5:

[2020] The server transmits the evaluation results to the terminal, which displays them to the user.

[2021] Input: Evaluation result from the server

[2022] Output: The evaluation result is displayed on the terminal.

[2023] role play

[2024] Step 1:

[2025] The user selects a role-playing session and clicks the "Start" button.

[2026] Input: User roleplay selection

[2027] Output: The device gets the selection information.

[2028] Step 2:

[2029] The terminal transmits the selection information to the server, and the server transmits the role-play data to the terminal.

[2030] Input: User's roleplay selection information

[2031] Output: Roleplay data is sent from the server to the device.

[2032] Step 3:

[2033] The device assigns a role to the user, and the user initiates an action or interaction.

[2034] Input: Roleplay data

[2035] Output: The roleplay screen is displayed on the device.

[2036] Step 4:

[2037] The terminal transmits the user's behavior record to the server.

[2038] Input: User activity log

[2039] Output: The activity record is sent to the server.

[2040] Evaluation and feedback generation

[2041] Step 1:

[2042] The server generates a comprehensive evaluation and feedback based on the simulation and role-play recording.

[2043] Input: Simulation and role-play recording

[2044] Processing: Evaluation by feedback generation algorithm

[2045] Output: Feedback

[2046] Step 2:

[2047] The server sends the generated feedback to the terminal, which displays it to the user.

[2048] Input: Generated feedback

[2049] Output: Feedback is displayed on the terminal

[2050] Emotion Engine

[2051] Step 1:

[2052] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[2053] Input: Data such as the user's facial expressions and voice

[2054] Output: Emotion data

[2055] Step 2:

[2056] The device transmits the collected emotion data to a server.

[2057] Input: Emotion data

[2058] Output: Emotion data is sent to the server

[2059] Step 3:

[2060] The server analyzes the emotional data and adjusts the progression of the scenario and feedback based on that.

[2061] Input: Emotion data

[2062] Processing: Analysis by sentiment analysis algorithm

[2063] Output: Scenario difficulty adjustment, feedback adjustment

[2064] Step 4:

[2065] The server transmits feedback and scenario information adjusted based on the emotion data to the terminal, which then displays it to the user.

[2066] Input: Tailored feedback and scenario information

[2067] Output: The adjusted information is displayed on the terminal.

[2068] (Application example 2)

[2069] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2070] Conventional new employee training systems provide uniform scenarios without considering the user's emotional state, making it difficult to provide detailed support and feedback tailored to each individual user's emotional state. Furthermore, even if a user is feeling stressed or tense, there is no mechanism for grasping that state in real time and adjusting the scenario appropriately, limiting the effectiveness of the training. This creates a need for real-time feedback tailored to the user's emotions, particularly for drivers of autonomous vehicles, who require practical skills.

[2071] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2072] In this invention, the server includes means for receiving registration information from users and registering new user information in a database, means for transmitting scenario data based on training scenario selections, means for receiving user selections for events in the scenario and evaluating based on the selections, means for transmitting data for role-playing and saving a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, and means for recognizing emotions from the user's facial expressions and voice in real time, sending and saving the emotion data to the server, and reflecting it in adjusting the difficulty level of the scenario. This makes it possible to grasp the user's emotional state in real time and appropriately adjust the training scenario and feedback according to the state of each individual user.

[2073] The "user registration means" is a means for receiving registration information from a user and registering the information of a new user in the database.

[2074] The "scenario selection means" is a means for transmitting scenario data based on the selection of a training scenario.

[2075] The "evaluation means" is a means for receiving a user's selection for an event in a scenario and making an evaluation based on the selection.

[2076] The "behavior recording means" is a means for transmitting data for role-playing and storing a record of the user's behavior.

[2077] The "feedback generating means" is a means for generating evaluation results and feedback and displaying them to the user.

[2078] The "emotion recognition means" is a means for recognizing emotions from the user's facial expressions and voice in real time, transmitting and storing the emotion data on a server, and reflecting it in adjusting the difficulty level of the scenario.

[2079] A system for implementing this invention provides adaptive new employee training by recognizing a user's emotions in real time and appropriately adjusting training scenarios and feedback based on the emotions. This system operates by integrating multiple hardware and software components.

[2080] System configuration

[2081] 1. User registration method

[2082] The terminal receives the registration information from the user and sends it to the server, which registers the information as a new user in its database.

[2083] 2. Scenario Selection Method

[2084] The server retrieves training scenario data from the database and sends it to the terminal, which displays a list of scenarios for the user to select.

[2085] 3. Evaluation methods

[2086] The terminal receives the user's selection for an event in a scenario and transmits it to the server, which performs an evaluation based on the selection and generates an evaluation result.

[2087] 4. Behavioral Recording Method

[2088] The device transmits role-playing data and records the user's actions, and sends these records to the server, which stores them.

[2089] 5. Feedback Generation Methods

[2090] The server generates feedback based on the evaluation results, sends it to the terminal, and displays it to the user.

[2091] 6. Emotion recognition means

[2092] The device analyzes the user's facial expressions and voice data in real time to recognize their emotions. This emotional data is then sent to a server, which stores and analyzes it and adjusts the difficulty of the scenario.

[2093] Hardware and software used

[2094] Hardware: Camera (face recognition and facial expression analysis), microphone (voice analysis), CPU / GPU (computational processing)

[2095] Software: OpenCV (video capture and face recognition), Keras / TensorFlow (emotion recognition model)

[2096] The main process of this system begins with the user entering their registration information and registering. Next, the user selects a training scenario from a list of scenarios displayed. Once the scenario begins, an evaluation is conducted based on the events selected by the user, and subsequent actions and interactions during role-playing are recorded. Based on these records and the evaluation results, the server generates feedback and displays it to the user. Furthermore, the device analyzes the user's facial and voice data, recognizes and collects emotions, and sends them to the server. The server adjusts the difficulty of the scenario based on the emotional data, providing a more adaptive training experience.

[2097] Specific examples

[2098] For example, in driver training for autonomous vehicles, the system analyzes the driver's facial expressions and voice in real time. If the driver shows signs of stress or tension, the system will lower the difficulty of the scenario and display a message to the driver to calm down. If the driver is relaxed, the system will provide instructions to continue the scenario.

[2099] Example prompt sentence:

[2100] Design a system to analyze the driver's emotions in real time for autonomous vehicles and improve driving safety based on those emotions. The system captures camera footage and analyzes the driver's emotions using facial recognition and emotion recognition models. Depending on the emotions, provide appropriate feedback to the driver in real time.

[2101] This system enables effective training that takes into account the user's emotional state, providing a safe and adaptive training experience, especially for drivers of autonomous vehicles.

[2102] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2103] Step 1:

[2104] The user operates the terminal and enters new user registration information (email address, password).

[2105] Input: The email address and password entered by the user

[2106] What happens: The device sends this information to the server.

[2107] Output: The server saves the registration information in the database and sends a registration completion message to the terminal.

[2108] Step 2:

[2109] The user operates the terminal to select a training scenario.

[2110] Input: User-selected training scenario

[2111] Operation: The device sends the selection information to the server, which retrieves the corresponding scenario data from the database and sends it to the device.

[2112] Output: The terminal displays the contents of the training scenario to the user.

[2113] Step 3:

[2114] The user selects options for events in the scenario.

[2115] Input: The event response selected by the user.

[2116] Operation: The device sends the selection information to the server, which evaluates the selection and generates an evaluation result.

[2117] Output: The server sends the generated evaluation results to the terminal, which displays them to the user.

[2118] Step 4:

[2119] Users perform role-plays and record their actions.

[2120] Input: User actions and interactions during roleplay

[2121] How it works: The device records actions and interactions and sends them to the server, which stores the recordings.

[2122] Output: The server evaluates the user based on the behavioral records stored and generates feedback.

[2123] Step 5:

[2124] The server generates feedback based on the evaluation results and sends it to the terminal.

[2125] Input: Evaluation result performed by the server

[2126] Operation: The server creates feedback based on the evaluation results and sends it to the device, which then displays the feedback to the user.

[2127] Output: Feedback displayed on the terminal

[2128] Step 6:

[2129] The device analyzes the user's facial expressions and voice to collect emotional data.

[2130] Input: User facial expression images and voice audio data

[2131] How it works: The device captures data using the camera and microphone, analyzes emotions in real time using a generative AI model (e.g., Keras / TensorFlow), and sends the emotion data to a server.

[2132] Output: Emotion data sent to the server

[2133] Step 7:

[2134] The server analyzes the user's emotional data and adjusts the difficulty of the scenario.

[2135] Input: Emotion data sent from the device

[2136] Operation: The server analyzes the emotional data and resets the difficulty level of the scenario to match the current state. It then sends the difficulty adjustment information to the device.

[2137] Output: The adjusted scenario displayed on the device

[2138] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2139] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2140] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2141] [Fourth embodiment]

[2142] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2143] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2144] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2145] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2146] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2147] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2148] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2149] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2150] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2151] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2152] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2153] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2154] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2155] System Overview

[2156] This invention is a system for supporting new employee training, allowing trainees to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and user terminals, and has the function of evaluating trainees' response capabilities through simulations and role-playing, and providing feedback.

[2157] System Details

[2158] The system is realized using a server and a terminal including the following means:

[2159] 1. User registration method

[2160] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[2161] The terminal sends the information to the server, which then registers the information as a new user in the database.

[2162] The server sends a registration completion message to the terminal, which displays it.

[2163] 2. Training scenario selection method

[2164] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[2165] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[2166] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[2167] The device displays the scenario start screen.

[2168] 3. Simulation execution method

[2169] The terminal displays events in the scenario on the screen, and the user selects a response.

[2170] The terminal transmits the user's selection to the server, which performs an evaluation based on the selection.

[2171] The server transmits the evaluation results to the terminal, which displays them to the user.

[2172] 4. Role-playing tools

[2173] The user selects a role-playing session and clicks the "Start" button.

[2174] The terminal transmits information to the server, and the server transmits data for role-playing to the terminal.

[2175] The device assigns a role to the user, and the user initiates an action or interaction.

[2176] The device sends the activity record to the server, which stores it.

[2177] 5. Evaluation and feedback generation methods

[2178] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[2179] The server sends the feedback to the terminal, which displays it to the user.

[2180] Specific examples

[2181] 1. Example of user registration

[2182] Tanaka, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[2183] The terminal sends this information to the server, and the server registers Tanaka's information in the database.

[2184] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Tanaka.

[2185] 2. Specific examples of training scenario selection

[2186] After completing registration, Tanaka logs in and a list of training scenarios is displayed. He selects "Customer Response Simulation."

[2187] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[2188] The device displays the scenario start screen, and Tanaka begins the simulation.

[2189] 3. Specific examples of simulation execution

[2190] During the simulation, the "Complaint handling" event is displayed on Tanaka's device.

[2191] Tanaka selects "Apologize" and the device sends the selection to the server.

[2192] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[2193] The device displays the evaluation results to Tanaka.

[2194] 4. Example of a role-play session

[2195] Tanaka begins the "sales presentation" role-play.

[2196] The device assigns Tanaka the role of presenter, and Tanaka actually gives the presentation.

[2197] The device records the content of Tanaka's presentation and sends it to the server.

[2198] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[2199] The device displays the evaluation results and feedback to Tanaka.

[2200] Throughout the system, users' actions are assessed and specific feedback is provided to help trainees improve their skills.

[2201] The processing flow will be explained below.

[2202] User Registration

[2203] Step 1:

[2204] The terminal presents the user with a login screen.

[2205] Step 2:

[2206] The user enters their email address and password on the login screen and clicks the "Register" button.

[2207] Step 3:

[2208] The terminal transmits the input information to the server.

[2209] Step 4:

[2210] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[2211] Step 5:

[2212] The server sends a registration completion message to the terminal.

[2213] Step 6:

[2214] The terminal displays a completion message to the user.

[2215] Training scenario selection

[2216] Step 1:

[2217] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[2218] Step 2:

[2219] The terminal displays a list of scenarios to the user.

[2220] Step 3:

[2221] The user selects the desired training scenario and clicks the "Start" button.

[2222] Step 4:

[2223] The terminal transmits scenario selection information to the server.

[2224] Step 5:

[2225] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[2226] Step 6:

[2227] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[2228] Simulation execution

[2229] Step 1:

[2230] The device displays the events in the scenario on the screen.

[2231] Step 2:

[2232] The user selects a response to the event and decides on the option.

[2233] Step 3:

[2234] The terminal transmits the user's selection to the server.

[2235] Step 4:

[2236] The server performs an evaluation based on the user's selection and generates an evaluation result.

[2237] Step 5:

[2238] The server transmits the evaluation results to the terminal.

[2239] Step 6:

[2240] The terminal displays the evaluation results to the user.

[2241] Role-playing session

[2242] Step 1:

[2243] The user selects a role-playing session and clicks the "Start" button.

[2244] Step 2:

[2245] The terminal transmits role-play selection information to the server.

[2246] Step 3:

[2247] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[2248] Step 4:

[2249] The device assigns a role to the user and displays role-based instructions and goals.

[2250] Step 5:

[2251] Users act and interact based on their assigned roles.

[2252] Step 6:

[2253] The device sends a record of the user's actions and interactions to the server.

[2254] Step 7:

[2255] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[2256] Step 8:

[2257] The server generates ratings and feedback and sends them to the device.

[2258] Step 9:

[2259] The device displays the rating and feedback to the user.

[2260] Evaluation and feedback generation

[2261] Step 1:

[2262] The server receives and analyzes all simulation and role-play recording data.

[2263] Step 2:

[2264] The server generates a user rating based on the analysis results.

[2265] Step 3:

[2266] The server generates feedback along with the evaluation results and sends it to the terminal.

[2267] Step 4:

[2268] The device displays the rating and feedback to the user.

[2269] Example 1

[2270] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2271] Conventional training systems have difficulty effectively evaluating trainees' skills and providing feedback under conditions equivalent to the actual work environment. Furthermore, it is difficult to ensure real-time and high-quality feedback during the evaluation and feedback generation process, making it difficult to provide specific advice necessary for improving trainees' skills.

[2272] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2273] In this invention, the server includes means for receiving registration information from a user and registering new user information in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and storing a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, and means for transmitting prompt sentences to a generative AI model to generate the evaluation results and feedback and receiving the generated feedback, thereby enabling high-quality feedback in real time to the trainee's behavior and selection.

[2274] "Registration information" refers to personal information, such as a user's email address and password, that is required to access and use the system.

[2275] The "database" is a system for centrally managing and storing data such as registration information, training scenarios, and evaluation results.

[2276] A "training scenario" is a storyline or content of a simulation or role-play for a specific task or situation that is selected and executed by a user.

[2277] "Scenario data" refers to detailed information and settings for constructing a training scenario, and is data that enables a scenario to be executed by a user.

[2278] An "event" is a specific situation or challenge that occurs within a training scenario, and is the subject of a user's response selection.

[2279] "Behavior record" is historical data of the actions and interactions selected by the user during a simulation or role-play.

[2280] The "evaluation results" are evaluation information of training results and skills generated based on the user's actions and selections.

[2281] "Feedback" refers to information such as specific advice and areas for improvement provided to the user based on the evaluation results.

[2282] A "generative AI model" is an artificial intelligence model used to generate evaluations and feedback on user actions and choices.

[2283] A "prompt" is an instruction given to a generative AI model that contains the information necessary to generate a specific evaluation or feedback.

[2284] This invention is a system for supporting new employee training, allowing users to learn practical skills in an environment similar to that of actual work. This system operates in conjunction with a server and terminals, and has the function of evaluating users' response capabilities through simulations and role-playing, and providing feedback.

[2285] System installation and operation

[2286] User Registration

[2287] When a user accesses the system for the first time, the terminal displays a user registration screen. The user enters registration information such as an email address and password, and the terminal sends this to the server. The server stores the received information in a database and sends a registration completion message to the terminal. The terminal then displays a registration completion message to the user.

[2288] Selecting a training scenario

[2289] When a user logs in, the server retrieves a list of training scenarios from the database and sends it to the terminal. The terminal displays the list of scenarios to the user, and the user selects the desired scenario. The terminal sends this selection information to the server, and the server sends the selected scenario data to the terminal. The terminal displays the scenario start screen.

[2290] Running the simulation

[2291] When a scenario starts, the device displays the events in the scenario on the screen. When the user selects a response, the device sends the selection to the server. The server evaluates the selection and sends the evaluation results to the device. The device displays the evaluation results to the user.

[2292] Role-play execution

[2293] When a user selects a role-playing session and clicks the "Start" button, the device sends that information to the server. The server sends the role-playing data to the device, and the device assigns a role to the user. The user then gives an actual presentation or interacts, and the device sends a record of this behavior to the server. The server saves the behavior record.

[2294] Evaluation and feedback generation

[2295] The server evaluates the simulation and role-playing based on the recorded data and generates comprehensive feedback. The evaluation results and feedback are generated by a generative AI model through prompt sentences. The server sends the generated feedback to the device, which then displays it to the user.

[2296] Hardware and software used

[2297] This system consists of a server and a terminal used by the user. The server uses a database system (e.g., MySQL, PostgreSQL) and an AI model (e.g., GPT-3, BERT). The terminal uses a web browser to display the screen and input data.

[2298] Specific examples

[2299] For example, consider the case where a user selects "Customer Service Simulation." During the simulation, the "Complaint Handling" event is displayed, and the user selects "Apologize." At this time, the device sends this selection information to the server, and the server sends the following prompt to the AI ​​model:

[2300] "In a complaint handling simulation, the user selected 'Apologize.' Please evaluate the appropriateness of this response and generate feedback."

[2301] Based on this input, the AI ​​model generates feedback such as "This is an appropriate response. It is a choice that is highly effective in easing the customer's anger," and sends it to the device via the server. The device then displays this to the user.

[2302] The system allows users to receive high-quality feedback in real time, effectively improving their skills in a real-world work environment.

[2303] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2304] Step 1:

[2305] A user accesses the system. Specifically, the user opens a browser and enters the system's URL. The input is the URL, and the user registration screen is displayed as the output.

[2306] Step 2:

[2307] The terminal displays the user registration screen. Specifically, it renders the registration form using HTML and CSS. The input is the access information from step 1, and the output is the registration form screen displayed to the user.

[2308] Step 3:

[2309] The user enters registration information. Specifically, the user enters an email address, password, etc., and clicks the "Register" button. Input is the information entered by the user, and output is the data sent to the terminal.

[2310] Step 4:

[2311] The terminal sends the entered information to the server. Specifically, it sends data to the server using JavaScript or the form's POST method. The input is the user's registration information, and the output is the data sent to the server.

[2312] Step 5:

[2313] The server stores the received information in a database. Specifically, the data received on the server side is inserted into the database using SQL statements. The input is the data received from the terminal, and the output is the record stored in the database.

[2314] Step 6:

[2315] The server sends a registration completion message to the terminal. Specifically, the server returns a success message as a response. The input is the user information stored in the database, and the output is the completion message sent to the terminal.

[2316] Step 7:

[2317] The terminal displays a registration completion message to the user. Specifically, it displays a message in HTML to notify the user. The input is the completion message from the server, and the output is the message displayed to the user.

[2318] Step 8:

[2319] The server retrieves a list of training scenarios from the database and sends them to the terminal. Specifically, it uses SQL statements to select the scenario data and returns it in JSON format. The input is a database query, and the output is the scenario data sent to the terminal.

[2320] Step 9:

[2321] The device displays a list of scenarios to the user. Specifically, it uses JavaScript to draw the scenario list. The input is the scenario data received from the server, and the output is the scenario list displayed to the user.

[2322] Step 10:

[2323] The user selects a scenario. Specifically, the user clicks on a scenario from a dropdown or list. The input is the user's scenario selection, and the output is the selection information sent to the terminal.

[2324] Step 11:

[2325] The device sends the selection information to the server. Specifically, it uses AJAX or the Fetch API to send the selection information to the server. The input is the user's selection information, and the output is the data sent to the server.

[2326] Step 12:

[2327] The server sends the selected scenario data to the device. Specifically, the server returns the corresponding scenario data in JSON format. The input is the user's selection information, and the output is the scenario data sent to the device.

[2328] Step 13:

[2329] The terminal displays the scenario start screen. Specifically, it creates a screen based on the scenario data and displays it to the user. The input is the scenario data received from the server, and the output is the scenario start screen displayed to the user.

[2330] Step 14:

[2331] The device displays the events in the scenario on the screen. Specifically, it uses JavaScript to draw the event content. The input is the scenario data, and the output is the event content that is displayed to the user.

[2332] Step 15:

[2333] The user selects a response to an event. Specifically, the user clicks on an option. The input is the user's selection, and the output is the selection information sent to the terminal.

[2334] Step 16:

[2335] The device sends the user's selection to the server. Specifically, the selection information is sent using AJAX or the Fetch API. The input is the user's selection information, and the output is the data sent to the server.

[2336] Step 17:

[2337] The server evaluates the user's selection. Specifically, the server executes evaluation logic based on the options. The input is the user's selection information, and the output is the evaluation result.

[2338] Step 18:

[2339] The server sends the evaluation results to the terminal. Specifically, the evaluation results are returned in JSON format. The input is the evaluation result, and the output is the data sent to the terminal.

[2340] Step 19:

[2341] The terminal displays the evaluation results to the user. Specifically, the evaluation results are rendered in HTML and notified to the user. The input is the evaluation results received from the server, and the output is the evaluation results displayed to the user.

[2342] Step 20:

[2343] The user selects a role-playing session and clicks the "Start" button. Specifically, the user clicks a button on the screen. The input is the user's selection, and the output is the starting information sent to the terminal.

[2344] Step 21:

[2345] The device sends the information to the server. Specifically, it uses AJAX or the Fetch API to send data. The input is the user's initial information, and the output is the data sent to the server.

[2346] Step 22:

[2347] The server sends role-playing data to the device. Specifically, it retrieves data using SQL statements and returns it in JSON format. The input is the user's starting information, and the output is the role-playing data sent to the device.

[2348] Step 23:

[2349] The device assigns a role to the user. Specifically, it displays the role data on the screen using JavaScript. The input is role-play data, and the output is the role information displayed to the user.

[2350] Step 24:

[2351] The user acts and interacts. Specifically, the user acts according to the simulation. The input is the role-play scenario, and the output is the user's actions.

[2352] Step 25:

[2353] The device sends the behavioral record to the server. Specifically, the behavioral data is sent to the server via AJAX. The input is the user's behavioral data, and the output is the data sent to the server.

[2354] Step 26:

[2355] The server saves the behavioral records. Specifically, it inserts the recorded data into the database. The input is the behavioral data received from the device, and the output is the record saved in the database.

[2356] Step 27:

[2357] The server evaluates all simulation and role-play records. Specifically, the server executes the evaluation logic. The input is the recorded data of the simulation or role-play, and the output is the evaluation result.

[2358] Step 28:

[2359] The server sends a prompt to the generative AI model and generates feedback. Specifically, it sends the evaluation result as a prompt to the generative AI model and generates feedback. The input is the evaluation result data, and the output is the generated feedback.

[2360] Step 29:

[2361] The server sends feedback to the device. Specifically, it sends feedback data in JSON format. The input is the generated feedback, and the output is the data sent to the device.

[2362] Step 30:

[2363] The device displays the feedback to the user. Specifically, the feedback content is rendered in HTML and displayed to the user. The input is the feedback received from the server, and the output is the feedback displayed to the user.

[2364] (Application example 1)

[2365] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2366] There is a need for a system that allows trainees to efficiently acquire practical skills in the operation and maintenance of factory robots without actually using the robots on-site. Another challenge is to introduce a system that allows practical training without the risk of operational errors or safety risks. There is also a need for a flexible training environment that uses easily portable devices such as smartphones and tablets.

[2367] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2368] In this invention, the server includes means for receiving registration information from a user and registering information of a new user in a database, means for transmitting scenario data based on a selection of a training scenario, means for receiving a user's selection of an event in the scenario and performing an evaluation based on the selection, means for transmitting data for role-playing and saving a record of the user's behavior, means for generating and displaying evaluation results and feedback to the user, means for transmitting scenario data including procedures for operating and maintaining a factory robot, and means for running a simulation of a factory environment through an application installed on a smartphone or tablet device. This enables safe and efficient training in the operation and maintenance of a factory robot without using an actual robot.

[2369] The "user registration means" is a function that receives registration information from a user and registers the information of a new user in the database.

[2370] The "training scenario selection means" is a function that transmits scenario data based on the selection of a training scenario.

[2371] The "simulation execution means" is a function that receives a user's selection for an event in a scenario and performs an evaluation based on that selection.

[2372] "Role-playing means" is a function that transmits data for role-playing and stores a record of the user's actions.

[2373] The "evaluation and feedback generating means" is a function that generates evaluation results and feedback and displays them to the user.

[2374] The "factory environment simulation means" is a function that transmits scenario data including the operation and maintenance procedures of factory robots.

[2375] "Applications installed on smartphones and tablet devices" refers to software that is installed on these devices and runs simulations of factory environments.

[2376] This invention is a system that provides training for the operation and maintenance of factory robots, allowing participants to acquire practical skills safely and efficiently. The system operates in conjunction with a server and terminals, and provides functions for user registration, training scenario selection, simulation execution, role-playing, and evaluation and feedback. A factory environment simulation is run through an application installed on a smartphone or tablet device.

[2377] Hardware and Software

[2378] Hardware: Smartphones, tablets, database servers.

[2379] Software: Flask (a Python microframework), PostgreSQL (a database management system), HTML / CSS / JavaScript (for building the user interface).

[2380] System processing details

[2381] 1. User Registration

[2382] The server receives the registration information (e.g., email address, password) from the user and registers it as a new user in the database. When registration is complete, it sends a registration completion message to the terminal and displays the message to the user.

[2383] 2. Training scenario selection

[2384] The server sends a list of various scenarios (e.g., introduction to robot operation, regular maintenance, safety management training), which the terminal displays to the user. When the user selects the desired scenario, the information is sent to the server, and the selected scenario data is sent to the terminal.

[2385] 3. Run the simulation

[2386] The terminal executes a scenario of a factory environment, and the user selects the appropriate action for each event in the scenario. This selection is sent to the server for evaluation. The evaluation results are returned to the terminal as feedback and displayed to the user.

[2387] 4. Role-playing

[2388] The user selects a role-playing session, and the server sends the role-playing data to the device. The user's actions and interactions are recorded and sent to the server. The server evaluates the recording, generates feedback, and sends it to the device.

[2389] 5. Ratings and Feedback

[2390] The server will provide an overall evaluation and generate feedback based on the results of all simulations and role-plays, helping users understand specific areas for improvement and what went well.

[2391] Examples and prompts

[2392] Specific examples

[2393] User registration: A new user registers by entering their email address and password. The server receives this, stores it in the database, and sends a confirmation message.

[2394] Training scenario selection: After logging in, the user selects "Introduction to Robot Operation." The server sends the scenario data.

[2395] Simulation: The user is asked to perform an emergency stop of the robot. The server evaluates the action and sends feedback.

[2396] Role-play: Initiate a role-play of the maintenance procedure, record user actions, and provide evaluation and feedback.

[2397] Prompt Sentence Examples

[2398] Example of prompt text when user selects scenario:

[2399] "Please select one of the following training scenarios: 1. Introduction to Robot Operation 2. Routine Maintenance 3. Safety Management Training"

[2400] This system makes it possible to conduct training in the operation and maintenance of factory robots safely and efficiently.

[2401] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2402] Step 1:

[2403] User Registration

[2404] The terminal receives an email address and password from the user as input. It sends this information to the server. The server saves the new user's information in a database and generates a registration completion message. The server sends this message to the terminal, which displays it to the user. The input is the user's email address and password, and the output is the registration completion message.

[2405] Step 2:

[2406] Training scenario selection

[2407] After the user logs in, the terminal sends a scenario list request to the server. The server retrieves a list of training scenarios from the database and sends it to the terminal. The user selects the desired scenario from the list, and the terminal sends the selection information to the server. The server sends the selected scenario data to the terminal, which displays it to the user. The input is the scenario list request and scenario selection, and the output is the scenario data.

[2408] Step 3:

[2409] Simulation execution

[2410] When a user starts a scenario, the device displays the events in the scenario. The user selects an action for the event, and the device sends that selection to the server. The server receives the selection, runs an evaluation algorithm, and generates an evaluation result. The result is sent to the device, which displays the evaluation result to the user. The input is the user's selection, and the output is the evaluation result.

[2411] Step 4:

[2412] role play

[2413] When a user starts role-playing, the device requests role-playing data from the server. The server sends the data, and the device assigns a role to the user. The device records the user's actions and interactions and sends them to the server. The server evaluates the records, generates feedback results, and sends them to the device. The device displays the results to the user. The input is the user's action record, and the output is the feedback results.

[2414] Step 5:

[2415] Ratings and Feedback

[2416] The server performs a comprehensive evaluation based on the result data of all simulations and role-plays. It analyzes the user's operation records and behavioral data and evaluates them using an evaluation algorithm. The server generates comprehensive feedback and sends it to the device. The device displays the feedback to the user. The input is the operation records and behavioral data, and the output is comprehensive feedback.

[2417] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2418] System Overview

[2419] This invention is a new employee training system that enables trainees to acquire practical skills and incorporates an emotion engine that recognizes the user's emotions. This system operates in conjunction with a server and terminals, evaluating the trainee's ability to respond through simulations and role-playing and providing feedback. The emotion engine recognizes the user's emotions during training and reflects them in the progression of the scenario and feedback accordingly, thereby achieving effective, individually customized training.

[2420] System Details

[2421] The system is implemented using the following means:

[2422] 1. User registration method

[2423] The terminal displays a new user registration screen, and the user enters the required information (e.g., email address and password).

[2424] The terminal sends the information to the server, which then registers the information as a new user in the database.

[2425] The server sends a registration completion message to the terminal, which displays it.

[2426] 2. Training scenario selection method

[2427] The server retrieves a list of training scenarios from the database and sends it to the terminal.

[2428] The terminal displays a list of scenarios to the user, and the user selects the desired scenario.

[2429] The terminal transmits the selection information to the server, and the server transmits the selected scenario data to the terminal.

[2430] The device displays the scenario start screen.

[2431] 3. Simulation execution method

[2432] The terminal displays events in the scenario on the screen, and the user selects a response.

[2433] The terminal transmits the user's selection to the server.

[2434] The server performs an evaluation based on the user's selection and generates an evaluation result.

[2435] The server transmits the evaluation results to the terminal, which displays them to the user.

[2436] 4. Role-playing tools

[2437] The user selects a role-playing session and clicks the "Start" button.

[2438] The terminal transmits the selection information to the server, and the server transmits the role-play data to the terminal.

[2439] The device assigns a role to the user, and the user initiates an action or interaction.

[2440] The device sends the activity record to the server, which stores it.

[2441] 5. Evaluation and feedback generation methods

[2442] The server evaluates all simulation and role-play recordings and generates comprehensive feedback.

[2443] The server sends the feedback to the terminal, which displays it to the user.

[2444] 6. Emotional Engine Means

[2445] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[2446] The device sends the collected emotion data to a server, which stores it.

[2447] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[2448] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[2449] Specific examples

[2450] 1. Example of user registration

[2451] Mr. A, who is undergoing new employee training, accesses the system, enters his email address and password on the login screen, and clicks the "Register" button.

[2452] The terminal sends this information to the server, and the server registers Mr. A's information in the database.

[2453] The server sends a message to the terminal saying "Registration completed," and the terminal displays the message to Mr. A.

[2454] 2. Specific examples of training scenario selection

[2455] After completing registration, Mr. A logs in and is presented with a list of training scenarios. He selects "Customer Service Simulation."

[2456] The terminal transmits the selection information to the server, and the server transmits the "customer interaction scenario" data to the terminal.

[2457] The device displays the scenario start screen, and Person A begins the simulation.

[2458] 3. Specific examples of simulation execution

[2459] During the simulation, the "Complaint handling" event is displayed on Mr. A's device.

[2460] Person A selects "Apologize" and the device sends that selection to the server.

[2461] The server evaluates the selection, generates feedback as an "appropriate response," and sends it to the device.

[2462] The device displays the evaluation results to Mr. A.

[2463] 4. Example of a role-play session

[2464] Person A begins the "sales presentation" role-play.

[2465] The device assigns the role of presentation to Person A, who then gives the actual presentation.

[2466] The device records the content of Mr. A's presentation and sends it to the server.

[2467] The server evaluates the recording and generates feedback such as "The presentation is well structured" and sends it to the device.

[2468] The device displays the evaluation results and feedback to Person A.

[2469] 5. Examples of using the Emotion Engine

[2470] During simulations and role-playing, the device recognizes emotions from Person A's facial expressions and voice and collects emotional data.

[2471] If the emotion indicates "tension" or "stress," the device sends it to the server.

[2472] The server analyzes the emotional data and adjusts the progress of the scenario accordingly.

[2473] For example, the server could lower the difficulty of the scenario based on emotional data and send appropriate feedback to Person A, such as "Take a break and relax here."

[2474] The device displays these instructions to Person A, prompting him to take appropriate action.

[2475] This system provides individually customized training content that takes into account the user's emotions, and allows for real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[2476] The processing flow will be explained below.

[2477] User Registration

[2478] Step 1:

[2479] The terminal presents the user with a login screen.

[2480] Step 2:

[2481] The user enters their email address and password on the login screen and clicks the "Register" button.

[2482] Step 3:

[2483] The terminal transmits the input information to the server.

[2484] Step 4:

[2485] The server registers a new user in the database based on the information received, and performs validation such as email address uniqueness and password strength.

[2486] Step 5:

[2487] The server sends a registration completion message to the terminal.

[2488] Step 6:

[2489] The terminal displays a completion message to the user.

[2490] Training scenario selection

[2491] Step 1:

[2492] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[2493] Step 2:

[2494] The terminal displays a list of scenarios to the user.

[2495] Step 3:

[2496] The user selects the desired training scenario and clicks the "Start" button.

[2497] Step 4:

[2498] The terminal transmits scenario selection information to the server.

[2499] Step 5:

[2500] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[2501] Step 6:

[2502] The terminal displays a scenario start screen to the user, and the user starts the simulation.

[2503] Simulation execution

[2504] Step 1:

[2505] The device displays the events in the scenario on the screen.

[2506] Step 2:

[2507] The user selects a response to the event and decides on the option.

[2508] Step 3:

[2509] The terminal transmits the user's selection to the server.

[2510] Step 4:

[2511] The server performs an evaluation based on the user's selection and generates an evaluation result.

[2512] Step 5:

[2513] The server transmits the evaluation results to the terminal.

[2514] Step 6:

[2515] The terminal displays the evaluation results to the user.

[2516] Role-playing session

[2517] Step 1:

[2518] The user selects a role-playing session and clicks the "Start" button.

[2519] Step 2:

[2520] The terminal transmits role-play selection information to the server.

[2521] Step 3:

[2522] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[2523] Step 4:

[2524] The device assigns a role to the user and displays role-based instructions and goals.

[2525] Step 5:

[2526] Users act and interact based on their assigned roles.

[2527] Step 6:

[2528] The device sends a record of the user's actions and interactions to the server.

[2529] Step 7:

[2530] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[2531] Step 8:

[2532] The server generates ratings and feedback and sends them to the device.

[2533] Step 9:

[2534] The device displays the rating and feedback to the user.

[2535] Evaluation and feedback generation

[2536] Step 1:

[2537] The server receives and analyzes all simulation and role-play recording data.

[2538] Step 2:

[2539] The server generates a user rating based on the analysis results.

[2540] Step 3:

[2541] The server generates feedback along with the evaluation results and sends it to the terminal.

[2542] Step 4:

[2543] The device displays the rating and feedback to the user.

[2544] Using the Emotion Engine

[2545] Step 1:

[2546] The device has a built-in emotion engine that recognizes the user's emotions in real time and collects emotional data.

[2547] Step 2:

[2548] The device sends the collected emotion data to a server, which stores it.

[2549] Step 3:

[2550] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[2551] Step 4:

[2552] The server adjusts the difficulty of the scenario based on the emotional data and transmits the information to the terminal.

[2553] Step 5:

[2554] The device displays the difficulty of the scenario and feedback to the user.

[2555] Specific examples

[2556] 1. Example of user registration

[2557] Step 1:

[2558] The terminal displays a login screen to Mr. A, who is undergoing new employee training.

[2559] Step 2:

[2560] User A enters his email address and password on the login screen and clicks the Register button.

[2561] Step 3:

[2562] The terminal transmits the input information to the server.

[2563] Step 4:

[2564] Based on the information received by the server, a new user, Mr. A, is registered in the database. The uniqueness of the email address and the strength of the password are validated.

[2565] Step 5:

[2566] The server sends a registration completion message to the terminal.

[2567] Step 6:

[2568] The device displays a completion message to Person A.

[2569] 2. Specific examples of training scenario selection

[2570] Step 1:

[2571] The server retrieves a list of training scenarios from the database and sends them to the terminal.

[2572] Step 2:

[2573] The device displays a list of scenarios to Person A.

[2574] Step 3:

[2575] User A selects the desired training scenario, "Customer Service Simulation," and clicks the start button.

[2576] Step 4:

[2577] The terminal transmits scenario selection information to the server.

[2578] Step 5:

[2579] The server retrieves scenario data corresponding to the selected scenario from the database and transmits it to the terminal.

[2580] Step 6:

[2581] The device displays the scenario start screen to Person A and the simulation begins.

[2582] 3. Specific examples of simulation execution

[2583] Step 1:

[2584] The device displays the "Complaint handling" event in the scenario on the screen.

[2585] Step 2:

[2586] User A selects "Apologize" and confirms the option.

[2587] Step 3:

[2588] The device sends Mr. A's selection to the server.

[2589] Step 4:

[2590] The server evaluates based on Mr. A's selection and generates an evaluation result as an "appropriate response."

[2591] Step 5:

[2592] The server transmits the evaluation results to the terminal.

[2593] Step 6:

[2594] The device displays the evaluation results to Mr. A.

[2595] 4. Example of a role-play session

[2596] Step 1:

[2597] User A selects the "Sales Presentation" role-play and clicks the start button.

[2598] Step 2:

[2599] The terminal transmits role-play selection information to the server.

[2600] Step 3:

[2601] The server retrieves the data required for the selected role-play from the database and transmits it to the terminal.

[2602] Step 4:

[2603] The device assigns Person A the role of presentation, and Person A begins to take action and interact.

[2604] Step 5:

[2605] User A acts and interacts based on the role assigned to him.

[2606] Step 6:

[2607] The device sends records of A's actions and conversations to the server.

[2608] Step 7:

[2609] The server stores the behavioral records and analyzes the data to generate ratings and feedback.

[2610] Step 8:

[2611] The server generates ratings and feedback and sends them to the device.

[2612] Step 9:

[2613] The device displays the rating and feedback to Person A.

[2614] 5. Examples of using the Emotion Engine

[2615] Step 1:

[2616] The device recognizes emotions from Mr. A's facial expressions and voice during training and collects emotional data.

[2617] Step 2:

[2618] The device transmits the collected emotion data to a server.

[2619] Step 3:

[2620] The server analyzes the emotional data and reflects it in the progression of the scenario and feedback.

[2621] Step 4:

[2622] The server adjusts the difficulty of the scenario based on the emotional data and sends that information to the device.

[2623] Step 5:

[2624] The device displays the adjusted difficulty of the scenario and feedback to Person A. For example, if the emotion indicates "tension" or "stress," the server can lower the difficulty of the scenario and display instructions to Person A such as "Take a break here and relax."

[2625] This system provides individually customized training content that takes into account the user's emotions, enabling real-time feedback and appropriate scenario adjustments, which is expected to significantly improve the efficiency and effectiveness of training.

[2626] Example 2

[2627] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2628] Conventional training systems provide uniform scenarios and feedback without considering the user's feelings, making it difficult to provide optimal training for each individual user. Furthermore, they are unable to generate and display appropriate feedback in real time, which can limit the effectiveness of training.

[2629] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving registration information from a user and registering information of a new user in a database; means for transmitting scenario data based on a selection of a training scenario; means for receiving a user's selection for an event in the scenario and performing an evaluation based on the selection; means for transmitting data for role-playing and saving a record of the user's behavior; means for generating and displaying evaluation results and feedback to the user; and means for incorporating an emotion engine that recognizes the user's emotions, collecting emotion data, and reflecting it in the progression of the scenario and feedback. This enables individually customized training that incorporates the user's emotions, and allows appropriate feedback and scenario adjustments to be made in real time.

[2630] "Means for receiving registration information from users and registering new user information in the database" refers to a function that receives personal information and authentication information provided by users to the system, stores that information in the database, and registers new users.

[2631] The "means for transmitting scenario data based on the selection of a training scenario" is a function for transmitting data corresponding to the training scenario selected by the user from the server to the terminal.

[2632] The "means for receiving a user's selection for an event in a scenario and performing an evaluation based on that selection" is a function for receiving a user's selection for an event or action in a training scenario and evaluating the content of that selection.

[2633] "Means for transmitting data for role-playing and storing user behavior records" is a function for transmitting data...

Claims

1. means for receiving registration information from users and registering new users' information in a database; means for transmitting scenario data based on the selection of a training scenario; means for receiving a user's selection of an event in a scenario and performing an evaluation based on the selection; means for transmitting role-playing data and storing user activity records; means for generating and displaying evaluation results and feedback to the user; A system including:

2. 2. The system according to claim 1, wherein the user's role-playing actions and interactions are recorded and evaluated based on the recorded actions and interactions.

3. 2. The system of claim 1, wherein feedback is generated based on the results of a simulation or role-play selected by the user, and the feedback is displayed to the user.

Images