System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording a video and a voice by a terminal used by an examinee; means for uploading the recorded video and voice to a central server by the terminal; means for receiving and analyzing the uploaded data by the server; means for evaluating the examinee based on the analysis result by the server; and means for providing the evaluation result to an administrator terminal by the server.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Role-playing tests, such as those for customer service skills and certification exams, are prone to evaluation bias based on human emotion and subjectivity, making fair evaluation difficult. With conventional methods, multiple examiners have different evaluation criteria, and their individual subjective opinions influence the results, making it difficult to achieve consistent and fair evaluations. Therefore, a system that provides fair and objective evaluations is needed. It is also necessary to reduce the workload of examiners. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. Specifically, the system includes a means for recording video and audio using a test-taker's terminal, a means for the test-taker's terminal to upload the recorded video and audio to a central server, a means for the server to receive and analyze the uploaded data, a means for the server to evaluate the test-taker based on the analysis results, and a means for the server to provide the evaluation results to an administrator terminal. This system enables accurate and objective evaluation of test-taker performance, eliminating subjective human bias. The system further includes a means for extracting frames from the recorded video data and performing facial expression and gesture recognition, a means for analyzing the recorded audio data and performing voice tone analysis and speech content analysis, and a means for scoring the analysis results based on the evaluation criteria. In this way, it is possible to provide a fair test environment for all test-taker's test and reduce the workload of examiners.

[0006] "Exam taker terminal" means a device used by an exam taker to record video and audio.

[0007] "Central Server" means a computer system that receives, stores, and analyzes uploaded video and audio data.

[0008] "Video and audio recording" refers to the act of recording a candidate's performance using a candidate's device.

[0009] "Uploading" means sending recorded video and audio data to a central server via the Internet.

[0010] "Data analysis" refers to processing the recorded video and audio data and performing facial expression recognition, gesture recognition, voice tone analysis, and speech content analysis.

[0011] The "evaluation means" is a process for evaluating the examinee's performance based on the analysis results and generating a score.

[0012] The "administrator terminal" is a device for checking the evaluation results and adjusting the evaluation criteria as necessary.

[0013] "Evaluation criteria" are objective standards that are set in advance to evaluate the test taker's performance.

[0014] "Scoring" refers to performing a numerical evaluation based on the analysis results.

[0015] "Facial expression recognition" is a technology that analyzes the facial expressions of test takers from video data.

[0016] "Gesture recognition" is a technology that analyzes the test taker's hand and body movements (gestures) from video data.

[0017] "Voice tone analysis" is a technology that analyzes the tone and emotions of a test-taker's voice from audio data.

[0018] "Speech content analysis" is a technology that analyzes the linguistic content of voice data and evaluates whether appropriate utterances and responses are being made. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Specific embodiments of this system are described below.

[0041] Subsystem 1: Examinee's terminal

[0042] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0043] Subsystem 2: Central Server

[0044] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the recorded audio data for tone and speech content. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0045] Subsystem 3: Administrator terminal

[0046] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0047] Specific examples

[0048] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content from the audio data. Based on this analysis, a score is generated according to the evaluation criteria. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[0049] In this way, the present invention provides a system for fair and objective evaluation of test-taker performance, eliminating human subjective bias, reducing examiner workload and enabling consistent evaluation.

[0050] The processing flow will be explained below.

[0051] Step 1:

[0052] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[0053] Step 2:

[0054] The device's camera begins recording the user's actions as video.

[0055] Step 3:

[0056] The device's microphone begins recording the user's voice.

[0057] Step 4:

[0058] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[0059] Step 5:

[0060] The device stores the recorded video and audio data in local storage.

[0061] Step 6:

[0062] The device uploads the stored video and audio data to a central server via the Internet.

[0063] Step 7:

[0064] The server receives the uploaded video and audio data and stores it in storage.

[0065] Step 8:

[0066] The server analyzes the video data and extracts frames, which are then used to perform facial expression and gesture recognition.

[0067] Step 9:

[0068] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[0069] Step 10:

[0070] The server generates a score based on the analysis results and in accordance with preset evaluation criteria.

[0071] Step 11:

[0072] The server stores the generated scores and analysis details in a database.

[0073] Step 12:

[0074] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[0075] Step 13:

[0076] The administrator uses the administrator terminal to check each test taker's score, analysis details, and related information.

[0077] Step 14:

[0078] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] In conventional role-playing tests, evaluations are easily influenced by subjective bias, making it difficult to evaluate test takers' performance fairly and objectively. Furthermore, the burden on examiners is heavy, making it difficult to maintain consistency in evaluations.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes means for recording video and audio using a terminal used by the examinee, means for the terminal to upload the recorded video and audio to a central server, means for the server to analyze the data and perform facial expression recognition and gesture recognition, means for the server to analyze the data and perform voice tone analysis and speech content analysis, means for the server to evaluate the examinee based on the analysis results and generate a score, and means for the server to provide the evaluation results to an administrator terminal. This enables test evaluation to be conducted fairly and objectively, reduces the burden on examiners, and enables consistent evaluation standards to be maintained.

[0084] A "candidate" is a person who takes part in an examination and is assessed on a particular task or skill.

[0085] "Device" means an electronic device used by a candidate for the test, which is equipped with the capability to record video and audio.

[0086] "Recording" refers to the act of using a terminal to record video and audio data during the test.

[0087] "Central Server" means a central data processing device that receives, analyzes, and evaluates recorded video and audio data.

[0088] "Uploading" is the act of sending recorded data from a terminal to a central server.

[0089] "Storage" refers to a storage device for temporarily or permanently storing received data.

[0090] "Analysis" is the process of processing the recorded data and extracting the information necessary for evaluation.

[0091] "Facial expression recognition" is a technology that detects the facial expressions of test takers from video data and analyzes their emotional state.

[0092] "Gesture recognition" is a technology that detects the examinee's hand movements and posture from video data and analyzes their meaning.

[0093] "Voice tone" refers to characteristics of voice data such as pitch, strength, and intonation.

[0094] "Speech content" is text information of words and phrases contained in the voice data.

[0095] "Evaluation criteria" refers to the standards and scales established to fairly and objectively evaluate test takers' performance.

[0096] A "score" is a numerical value or evaluation index generated from the analysis results based on the evaluation criteria.

[0097] An "administrator terminal" is an electronic device used by the test administrator or examiner to check evaluation results and adjust evaluation criteria.

[0098] "Display" refers to the act of showing the evaluation results and analysis details on the screen of the administrator's terminal.

[0099] A "database" is a system that systematically stores and manages data such as analysis results and evaluation criteria.

[0100] "Application" is the act of reflecting the newly adjusted evaluation criteria in the next evaluation.

[0101] This is an evaluation system consisting of multiple subsystems for the purpose of fairly and objectively evaluating examinee performance. The system consists of three main components: examinee terminals, a central server, and an administrator terminal.

[0102] Examinee's device

[0103] The user uses a device with a dedicated recording application installed. This device is equipped with a camera and microphone, which are used to record video and audio during the test. The user starts the recording application and presses the "Start role-playing" button to start recording, and the "Stop recording" button to end recording. Once recording is complete, the device automatically uploads the recorded video and audio data to a central server.

[0104] Central Server

[0105] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs various analyses. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the audio data for tone and speech content. These analyses are performed using dedicated software such as facial expression recognition libraries and audio analysis tools. Based on the analysis results, a score is generated according to pre-set evaluation criteria. The generated score and analysis details are stored in a database.

[0106] Administrator terminal

[0107] The administrator uses the administrator terminal to retrieve the assessment results from the central server. The management screen of the administrator terminal displays each examinee's score, analysis details, and related information. In addition, the administrator can adjust the assessment criteria. When new assessment criteria are entered, they are saved in the database and applied to the next assessment.

[0108] Specific examples

[0109] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content of the speech from the audio data. Based on these analyses, a score is generated according to the evaluation criteria. The test administrator can view this score using the administrator device and adjust the evaluation criteria as necessary.

[0110] Examples of prompt statements

[0111] Example prompt 1: "Analyze the test-taker's video to recognize facial expressions and gestures. Then analyze the audio data to analyze speaking style and content to generate a score based on the assessment criteria."

[0112] Example prompt 2: "Analyze the uploaded test data and generate a score based on the test taker's facial expressions, gestures, tone, and speech content. Then store the generated score along with the analysis results in a database."

[0113] In this way, the system of the present invention allows exam evaluation to be carried out fairly and objectively, reduces the burden on examiners, and maintains consistent evaluation standards.

[0114] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0115] Step 1:

[0116] The user launches a dedicated recording application on their device. As input, a launch command is given, which starts preparation of the device's camera and microphone. As output, the initial screen of the application is displayed, ready for recording.

[0117] Step 2:

[0118] The user presses the "Start Role-Playing" button. The user's operation instruction (pressing the button) is given as input, and the device activates the camera and microphone to start recording video and audio. The output is that the recorded video and audio data is saved in the device's local storage in real time.

[0119] Step 3:

[0120] The user presses the "Stop Recording" button. The user's operation instruction is given as input, and the device stops recording. The output is a state in which the stopped video and audio data has been completely saved to local storage.

[0121] Step 4:

[0122] The device makes an API call to the central server to send the recorded data. The input is the recorded video and audio data that the device uploads to the central server. The output is the video and audio data received by the server.

[0123] Step 5:

[0124] The server receives data sent from the device and stores it in storage. The input is the uploaded video and audio data, which the server stores in temporary storage. The output is the data stored in storage.

[0125] Step 6:

[0126] The server extracts frames from the video data and performs facial expression and gesture recognition. The input is video data stored in storage, which the server passes to the facial expression recognition library for analysis. The output is analyzed facial expression and gesture data.

[0127] Step 7:

[0128] The server analyzes the voice data and performs voice tone analysis and speech content analysis. The input is the voice data stored in the storage, and the server passes it to the voice analysis tool. The output is the analyzed voice tone and speech content data.

[0129] Step 8:

[0130] The server generates a score based on the analysis results according to the set evaluation criteria. The input is the analyzed facial expression, gesture, voice tone, and speech content data, which the server inputs into the evaluation algorithm. The output is the generated score.

[0131] Step 9:

[0132] The server stores the generated score in a database. The input is the generated score, which the server records in the database. The output is the score stored in the database.

[0133] Step 10:

[0134] The administrator uses the administrator terminal to obtain the evaluation results from the central server. The input is a request to obtain the evaluation results, which the administrator terminal sends to the server. The output is to display the evaluation results sent from the server on the screen of the administrator terminal.

[0135] Step 11:

[0136] The administrator can adjust the evaluation criteria using the administrator terminal. The input is the adjusted new evaluation criteria, which the administrator terminal sends to the server. The output is that the new evaluation criteria are stored in the database, ready to be applied at the next evaluation.

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] It is difficult to fairly and objectively evaluate the customer service skills of employees in brick-and-mortar stores. Conventional methods rely on the subjective judgment of the evaluator, which can lead to inconsistency and a lack of fairness. Furthermore, employee training is time-consuming and costly, and providing effective feedback is difficult. The present invention solves these problems and provides a system for efficiently and objectively evaluating and training customer service skills in brick-and-mortar stores.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes: means for recording video and audio using a terminal used by the examinee; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the examinee based on the analysis results; and means for the server to provide the evaluation results to a manager's terminal. In a customer service skill evaluation system used in a brick-and-mortar store, means for recording the actions and speech of employees; means for uploading the recorded data to the central server and analyzing it to generate evaluation results; and means for a manager to check the evaluation results and adjust the evaluation criteria. This makes it possible to evaluate the customer service skills of employees fairly and objectively and provide effective feedback.

[0142] "Examination candidate terminal" refers to a device used by an examination candidate to record video and audio, and includes a camera, microphone, etc.

[0143] The "central server" is a computer system that receives and analyzes data uploaded from test taker terminals.

[0144] An "administrator terminal" is a device that displays the evaluation results and adjusts the evaluation criteria as necessary, and is generally a PC or tablet terminal.

[0145] "Facial expression recognition" is the process of analyzing facial features extracted from frames of video data to identify emotions and expressions.

[0146] "Gesture recognition" is the process of analyzing physical movements extracted from frames of video data and identifying their meaning.

[0147] "Voice tone analysis" is the process of analyzing audio data to evaluate the pitch, intensity, rhythm, etc. of a speaker's voice.

[0148] "Speech analysis" is the process of analyzing the content of audio data to identify the words being spoken and their content.

[0149] The "customer service skills evaluation system" is a system for fairly and objectively evaluating the customer service skills of employees in physical stores based on video and audio data.

[0150] "Recording means" refers to a method or device for recording video and audio using a camera or microphone.

[0151] "Uploading means" refers to a method or device for transmitting recorded video and audio data over a network to a central server.

[0152] "Analysis means" refers to software or algorithms that the central server uses to analyze the uploaded video and audio data.

[0153] An "assessment tool" is a method or algorithm used to evaluate candidates or employees based on the results of the analysis.

[0154] "Evaluation criteria" are the indicators or standards used to evaluate candidates or employees and are used for scoring purposes.

[0155] The present invention provides a system for fairly and objectively evaluating the customer service skills of employees in a brick-and-mortar store. Specific embodiments of this system will be described in detail below.

[0156] 1. Exam-taker's device

[0157] The devices used by employees include smartphones, smart glasses, or head-mounted displays. These devices are equipped with cameras and microphones, allowing them to record video and audio. Recording begins by launching a dedicated application installed on the device and pressing the "Start Role-Playing" button. When the customer service simulation is finished, the user presses the "Stop Recording" button, and the recorded video and audio data is automatically uploaded to the central server.

[0158] 2. Central Server

[0159] The central server receives the data uploaded from the test taker's device and stores it in storage. The server then performs several processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. This is done using the OpenCV library. The recorded audio data is also analyzed for tone and speech content. This is done using the PyAudio library. Based on the results of these analyses, a score is generated according to the evaluation criteria, and the evaluation results are stored in a database.

[0160] 3. Administrator terminal

[0161] The manager terminal is used by the store manager. The manager terminal retrieves the evaluation results from the central server and displays them on a dedicated management screen. On this management screen, each employee's score, analysis details, and related information can be viewed. In addition, the manager can adjust the evaluation criteria through this screen. New evaluation criteria are entered from the manager terminal, and the central server saves them in the database and applies them the next time an employee is evaluated.

[0162] 4. Specific Examples

[0163] For example, consider a case where a newly hired employee at a store is undergoing customer service training. The employee wears smart glasses and performs customer service role-playing with a colleague playing the role of a customer. The recorded video and audio are uploaded to a server, where facial expressions, speech patterns, and gestures are analyzed in detail. A manager can view the results on a manager's device, evaluate the employee's customer service skills, and provide necessary feedback. This system enables consistent and fair evaluations, leading to improved employee skills.

[0164] Generative AI model prompt example

[0165] Analyze the following video and audio data to evaluate an employee's customer service skills. Generate scores for each of the following items: facial expressions, gestures, tone of voice, and content of speech, and output an overall score:

[0166] <Video data URL>

[0167] <Audio data URL>

[0168] In this way, the system of the present invention makes it possible to fairly and objectively evaluate test takers' performance and provide effective feedback, thereby enabling employees to quickly and effectively improve their customer service skills.

[0169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0170] Step 1:

[0171] Employees will use smart devices to conduct customer service simulations.

[0172] Specifically, employees wear a smartphone, smart glasses, or a head-mounted display and perform a customer service role-play. To do so, they launch the recording application and press the "Start role-playing" button to begin recording video and audio.

[0173] Input: Video and audio data collected from the camera and microphone built into your smart device.

[0174] Output: Recorded video and audio data.

[0175] Step 2:

[0176] After the device finishes recording, it uploads the video and audio data to a central server.

[0177] Specifically, when the user presses the "Stop Recording" button, the device automatically transmits the recorded data to a central server via secure communication, ensuring the safety of the data.

[0178] Input: Video and audio data after recording has finished.

[0179] Output: Data is uploaded to a central server.

[0180] Step 3:

[0181] The server receives the uploaded data and stores it in storage.

[0182] Specifically, the server converts the received data into an appropriate format and stores it in a database or file system.

[0183] Input: Uploaded video and audio data.

[0184] Output: Securely stored video and audio data.

[0185] Step 4:

[0186] The server extracts frames from the video data and performs facial expression and gesture recognition.

[0187] Specifically, it uses the OpenCV library to split the video data into frames, analyzes the facial features in each frame, and identifies the employee's facial expressions and gestures.

[0188] Input: Stored video data.

[0189] Output: Analysis results of facial expression and gesture recognition.

[0190] Step 5:

[0191] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[0192] Specifically, it uses the PyAudio library to extract features from audio data, evaluate the pitch, intensity, and rhythm of the speaker's voice, and identify what is being said.

[0193] Input: Stored audio data.

[0194] Output: Voice tone analysis and speech content analysis.

[0195] Step 6:

[0196] The server generates a score based on the analysis results and in accordance with the evaluation criteria.

[0197] Specifically, based on the analysis results, pre-set evaluation criteria are applied to generate a numerical score for the employee's customer service skills.

[0198] Input: Results of facial expression recognition, gesture recognition, voice tone analysis, and speech content analysis.

[0199] Output: Scored evaluation results.

[0200] Step 7:

[0201] The server provides the evaluation results to the administrator terminal.

[0202] Specifically, the generated scores and analysis details will be displayed on a dedicated management screen so that administrators can check them.

[0203] Input: Scored assessment results.

[0204] Output: Evaluation results displayed on the administrator's terminal.

[0205] Step 8:

[0206] The administrator adjusts the evaluation criteria and saves the new criteria on the server.

[0207] Specifically, the administrator inputs the evaluation criteria using the management terminal, and the server stores them in the database and applies them the next time the evaluation is conducted.

[0208] Input: New evaluation criteria from the administrator terminal.

[0209] Output: The new saved criterion.

[0210] The above are the specific processing steps for fairly and objectively evaluating the customer service skills of the examinee.

[0211] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0212] The present invention provides a system for fairly and objectively evaluating test takers in role-playing tests such as customer service skills tests and certification tests. Furthermore, the present invention enables more precise evaluations by incorporating an emotion engine that analyzes the user's emotions. A specific embodiment of this system is described below.

[0213] Subsystem 1: Examinee's terminal

[0214] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0215] Subsystem 2: Central Server

[0216] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0217] Subsystem 3: Administrator terminal

[0218] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0219] Specific examples

[0220] Let's take an example where a candidate is taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes their speaking style and content from the audio data. In addition, an emotion engine analyzes the candidate's emotions using the video and audio data. Based on this analysis, a score is generated according to the evaluation criteria, and additional evaluation criteria are applied based on the emotion analysis results. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[0221] In this way, the present invention provides a system that evaluates test takers' performance fairly and objectively, eliminating human subjective bias. Furthermore, the use of an emotion engine enables advanced evaluation that takes test takers' emotional states into account. The system of the present invention reduces the workload of examiners and ensures consistent evaluation.

[0222] The processing flow will be explained below.

[0223] Step 1:

[0224] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[0225] Step 2:

[0226] The device's camera begins recording the user's actions as video.

[0227] Step 3:

[0228] The device's microphone begins recording the user's voice.

[0229] Step 4:

[0230] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[0231] Step 5:

[0232] The device stores the recorded video and audio data in local storage.

[0233] Step 6:

[0234] The device uploads the stored video and audio data to a central server via the Internet.

[0235] Step 7:

[0236] The server receives the uploaded video and audio data and stores it in storage.

[0237] Step 8:

[0238] The server analyzes the video data and extracts frames, which are then used to perform facial expression recognition and analyze the user's facial features.

[0239] Step 9:

[0240] The server performs gesture recognition from video data and analyzes the user's hand and body movements.

[0241] Step 10:

[0242] The server analyzes the voice data, analyzes the voice tone and the content of the speech, specifically extracting voice waveform features and performing language analysis.

[0243] Step 11:

[0244] The server uses an emotion engine to perform emotion analysis based on the video and audio data. It identifies the user's emotional state by analyzing facial expressions from the video data and tone of voice and content of speech from the audio data.

[0245] Step 12:

[0246] The server generates a score based on the analysis results according to a preset evaluation criterion, which also incorporates the emotion analysis results.

[0247] Step 13:

[0248] The server stores the generated scores and analysis details in a database.

[0249] Step 14:

[0250] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[0251] Step 15:

[0252] The administrator uses the administrator terminal to check each test taker's score, analysis details, and emotion analysis results.

[0253] Step 16:

[0254] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[0255] Example 2

[0256] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0257] Traditional role-playing tests often rely on the examiner's subjective evaluation, making it difficult to achieve fair and objective evaluation. Furthermore, because it is difficult to analyze emotions, it is not possible to perform sophisticated evaluations that take into account the examinee's emotional state. Furthermore, there are challenges in that it is difficult for test administrators to consistently apply evaluation criteria.

[0258] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring audio and video by an information terminal used by the examinee, means for transferring the audio and video acquired by the information terminal to a central data processing device, means for the central data processing device to receive and analyze the transferred data, means for the central data processing device to evaluate the examinee based on the analysis results, and means for the central data processing device to provide the evaluation results to an administrator's information terminal. This makes it possible to evaluate the examinee's performance fairly and objectively, eliminate the subjective bias of the examiner, and perform consistent evaluations.

[0259] "Information terminal" means a device used by a test taker during the test to obtain audio and video.

[0260] The "central data processing device" is a device that receives, analyzes, and evaluates audio and video data transferred from the examinee's information terminal.

[0261] The "analysis means" is a function that allows the central data processing unit to process audio and video data and perform facial expression recognition, action recognition, tone analysis, and analysis of audio content.

[0262] The "evaluation means" is a function that allows the central data processing unit to evaluate and score the performance of the examinee based on the analysis results.

[0263] The "administrator's information terminal" is a device that displays the evaluation results and analysis results provided by the central data processing unit and enables the evaluation criteria to be changed.

[0264] "Acquisition means" refers to the function that enables an information terminal to acquire audio and video.

[0265] The "transfer means" is a function for transmitting data acquired by the information terminal to the central data processing unit.

[0266] The "memory device" is a storage device for saving information such as analysis results and evaluation criteria in the central data processing unit.

[0267] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Furthermore, by combining an emotion engine, more precise evaluations can be performed. The system of the present invention is primarily composed of an information terminal used by the examinee, a central data processing unit, and an information terminal for the administrator.

[0268] Exam taker information terminal

[0269] The terminal is a device used by the test taker to capture audio and video during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the terminal and starts recording by pressing the "Start Role-Playing" button. The application uses the camera to record the test taker's actions and the microphone to record audio. When the test is finished, the user presses the "Stop Recording" button, and the terminal automatically uploads the recorded data to the central data processing unit.

[0270] Central Data Processing Unit

[0271] The server receives the data uploaded from the test taker's device and stores it in storage. Next, frames are extracted from the video data and facial expression and movement recognition is performed. This is done using video analysis software such as OpenCV. Furthermore, the audio data is analyzed for tone and audio content using audio analysis software such as LIBROSA. Furthermore, an emotion engine (e.g., an emotion recognition AI module) is used to analyze the test taker's emotional state. Based on the results of these analyses, a score is generated according to the evaluation criteria and stored in a database.

[0272] Administrator information terminal

[0273] The administrator terminal is a device used by the test administrator and examiners. The administrator terminal obtains the evaluation results from the central data processing device and displays them on the management screen. On this management screen, each examinee's score, detailed analysis results, and related information can be viewed. The administrator can adjust the evaluation criteria as needed. New evaluation criteria are entered from the administrator terminal and sent to the central data processing device. The server saves the new evaluation criteria in storage and applies them to the next evaluation.

[0274] Specific examples

[0275] For example, consider a test taker taking a customer service etiquette test. First, the test taker uses a device to begin the test and records their performance using a camera and microphone. The user introduces themselves and role-plays interacting with virtual customers. Once the recording is complete, the device uploads the data to a central data processing unit. The server analyzes the uploaded video data and recognizes the test taker's facial expressions and movements. It also analyzes the audio data to evaluate speaking style, tone, and content. It also uses an emotion engine to analyze the test taker's emotions and evaluates factors such as stress and friendliness. A score is generated based on these results, and the test taker is assigned a grade according to the evaluation criteria.

[0276] Prompt Sentence Examples

[0277] "This system records video and audio of test takers taking customer service etiquette tests and uploads them to a central server. The server analyzes facial expressions and gestures from the video and the content of speech from the audio. It also uses an emotion engine to analyze the test takers' emotions. Finally, it generates a score based on the evaluation criteria and displays it on an administrator's terminal."

[0278] In this way, test takers' performance can be evaluated fairly and objectively, eliminating subjective bias and ensuring consistent evaluation. Using an emotion engine enables advanced evaluation that takes test takers' emotional states into account, reducing the workload of test administrators and enabling more precise evaluation.

[0279] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0280] Program processing steps

[0281] Step 1:

[0282] The user launches a dedicated recording application on their information terminal. When the application launches, it checks the camera and microphone readiness and displays that they are available for use. This allows real-time data of the examinee's movements and speech to be obtained as input.

[0283] Step 2:

[0284] The user presses the "Start Role-Playing" button to begin recording. The device uses the camera to record the examinee's actions as video and the microphone to record audio. Once video and audio data begins to be recorded, it is continuously saved. The input is real-time video and audio, and the output is recorded video and audio files.

[0285] Step 3:

[0286] When the test is finished, the user presses the "End Recording" button. The terminal saves the recorded data as a file and prepares it for upload to the central data processing unit. The input is the recorded video and audio files, and the output is a data package for uploading.

[0287] Step 4:

[0288] The terminals upload the recorded video and audio data to the central data processing unit. The data is transferred over the network and reaches the central data processing unit. The input is the data package for uploading, and the output is the data received by the central data processing unit.

[0289] Step 5:

[0290] The server receives the video and audio data uploaded from the terminal and stores it in storage. The input is the received data package, and the output is a data file stored in the storage.

[0291] Step 6:

[0292] The server extracts frames from the video data and performs facial expression and movement recognition. This uses dedicated video analysis software (e.g., OpenCV). The input is a saved video file, and the output is facial expression and movement data.

[0293] Step 7:

[0294] The server analyzes the audio data and performs a timbre and audio content analysis using audio analysis software (e.g., LIBROSA). The input is a saved audio file, and the output is timbre and audio content data.

[0295] Step 8:

[0296] The server uses an emotion engine to perform emotion analysis based on video and audio data. The inputs are facial expression data, movement data, tone data, and audio content data, and the output is emotion data.

[0297] Step 9:

[0298] The server generates a score based on the analysis results and evaluation criteria. The evaluation criteria are set in advance and points are calculated based on these. The inputs are facial expression data, movement data, tone data, voice content data, and emotion data, and the output is a final score and an evaluation report.

[0299] Step 10:

[0300] The server stores the generated scores and analysis details in a database. The input is the final score and the evaluation report, and the output is the evaluation results stored in the database.

[0301] Step 11:

[0302] The administrator terminal obtains the evaluation results from the central data processing unit and displays them on the management screen. The administrator can check each examinee's score and detailed analysis results. The input is the evaluation results from the database, and the output is the information displayed on the management screen.

[0303] Step 12:

[0304] The administrator adjusts the evaluation criteria as needed. The administrator terminal sends the new evaluation criteria to the central data processing unit. The input is the new evaluation criteria entered by the administrator, and the output is the updated evaluation criteria stored in the central data processing unit.

[0305] The above processing steps realize a system for evaluating the performance of test takers fairly and objectively.

[0306] (Application example 2)

[0307] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0308] Current customer service skill evaluation systems often rely on subjective evaluations, making it difficult to provide fair and objective evaluations. Furthermore, they lack real-time analysis of customer service behavior and detailed emotion analysis using emotion engines, making it cumbersome to provide feedback on specific areas for improvement to staff. This makes it difficult to efficiently support staff growth.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording video and audio using a terminal used by the test taker; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the test taker based on the analysis results; means for the server to provide the evaluation results to an administrator terminal; means for the store staff to record video and audio in real time using smart glasses while serving customers; and means for transmitting the recorded data to the server and analyzing facial expressions, gestures, and tone of voice using an emotion engine. This enables objective and fair evaluation of customer service skills and also enables efficient real-time feedback of specific areas for improvement to be provided to staff.

[0310] "Device used by the examinee" refers to the device used by the examinee during the examination to record video and audio.

[0311] "Central Server" is a computer system established to receive, analyze, and evaluate data uploaded from the terminals.

[0312] The "emotion engine" is software that analyzes facial expressions, gestures, and tone of voice from video and audio data to determine the user's emotions.

[0313] "Smart glasses" are glasses-type mobile devices worn by users that can record video and audio in real time.

[0314] "Facial expression recognition" is a technology that analyzes facial expressions from recorded video data.

[0315] "Gesture recognition" is a technology that analyzes hand and body movements from recorded video data.

[0316] "Voice tone analysis" is a technology that analyzes speaking style and tone of voice from recorded audio data.

[0317] "Evaluation criteria" are the criteria for evaluating test takers' skills and performance based on the recorded and analyzed data.

[0318] An "administrator terminal" is a device used by an administrator to check the evaluation results and adjust the evaluation criteria.

[0319] "Real-time feedback" is the process of providing staff with evaluation and analysis results obtained during customer service on the spot.

[0320] The present invention provides a system for fairly and objectively evaluating test takers in customer service skills and qualification examinations. Specific embodiments of this system are described below.

[0321] Subsystem 1: Examinee's terminal

[0322] The device is used by the test taker to record video and audio during the test. It includes a camera, microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0323] Subsystem 2: Central Server

[0324] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0325] Subsystem 3: Administrator terminal

[0326] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0327] Application example: Smart customer service evaluation assistant in brick-and-mortar stores

[0328] This system can be applied as an application installed on smart glasses to support the evaluation of customer service staff in brick-and-mortar stores. The system records video and audio in real time while the staff are serving customers and transmits the data to a server.

[0329] Specifically, staff wear smart glasses and record video and audio data while serving customers. The server receives this data and uses an emotion engine to analyze facial expressions, gestures, and tone of voice. Based on the analysis results, staff are then scored for their customer service skills. Managers can check the scores and analysis results and provide specific feedback to staff on areas for improvement in real time.

[0330] Hardware and Software

[0331] Hardware

[0332] Smart glasses (e.g. Google Glass)

[0333] Exam taker's device (camera, microphone)

[0334] software

[0335] Recording application (video and audio recording)

[0336] Emotion engine (analysis of facial expressions, gestures, and voice tone)

[0337] Server system (receiving, analyzing, and evaluating data)

[0338] Examples of prompt statements

[0339] "For customer service training evaluation, please provide specific program code for analyzing recorded audio and video data and recognizing facial expressions, gestures, and tone of voice."

[0340] In this way, we provide a system that uses smart glasses to enable real-time customer service evaluation and efficiently support staff in improving their skills.

[0341] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0342] Step 1:

[0343] The user puts on the smart glasses and begins serving customers. While serving customers, the smart glasses' camera and microphone record video and audio in real time. The recorded video is saved frame by frame, and the audio is continuously saved in a buffer as data. The input is the actual customer service scene, and the output is the recorded video and audio data.

[0344] Step 2:

[0345] The device's recording application starts recording by pressing the "Start Role-Playing" button. The camera captures video as successive frames, and the microphone captures audio data. When recording is complete, the user presses the "Stop Recording" button to stop recording. The input is the user's actions, and the output is the stopped video and audio files.

[0346] Step 3:

[0347] The device uploads the recorded video and audio data to a central server. After recording is complete, the device application sends the data to the server using the FTP protocol or HTTP POST request. The input is the recorded video and audio files, and the output is the data stored on the server.

[0348] Step 4:

[0349] The server receives the uploaded video and audio data and saves it in storage. The server receives the HTTP request and saves the video and audio files in the specified directory. The input is the data sent from the device, and the output is the data saved in storage.

[0350] Step 5:

[0351] The server extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. The emotion engine uses, for example, Google Cloud Vision API or Microsoft Azure Face API. The input is each frame of the video data, and the output is recognized facial expression data, gesture data, and emotion data.

[0352] Step 6:

[0353] The server analyzes the recorded voice data, analyzing voice tone, speech content, and emotion using an emotion engine. Voice analysis uses Google Cloud Speech-to-Text API and IBM Watson Speech to Text, among others. The input is voice data, and the output is voice tone data, speech content data, and emotion data.

[0354] Step 7:

[0355] The server evaluates the examinee's skills and performance based on the analysis results and generates a score according to pre-set evaluation criteria. The evaluation criteria are set by the system administrator. The inputs are facial expression data, gesture data, voice tone data, and speech content data, and the output is an evaluation score.

[0356] Step 8:

[0357] The server stores the generated scores and analysis details in a database. The inputs are the assessment scores and analysis results, and the output is the assessment data stored in the database.

[0358] Step 9:

[0359] The administrator terminal obtains the evaluation results from the server and displays them on the management screen. The administrator checks the evaluation results and analysis details and adjusts the evaluation criteria as necessary. The input is the evaluation data, and the output is the evaluation results displayed on the administrator terminal.

[0360] Step 10:

[0361] The server receives the evaluation criteria data adjusted by the administrator terminal and stores it in the database. The new criteria are applied at the next evaluation. The input is the new evaluation criteria data, and the output is the evaluation criteria stored in the database.

[0362] The above steps enable real-time customer service skill evaluation using smart glasses, providing objective and fair evaluations.

[0363] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0364] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0365] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0366] [Second embodiment]

[0367] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0368] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0369] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0370] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0371] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0372] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0373] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0374] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0375] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0376] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0377] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0378] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0379] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Specific embodiments of this system are described below.

[0380] Subsystem 1: Examinee's terminal

[0381] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0382] Subsystem 2: Central Server

[0383] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the recorded audio data for tone and speech content. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0384] Subsystem 3: Administrator terminal

[0385] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0386] Specific examples

[0387] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content from the audio data. Based on this analysis, a score is generated according to the evaluation criteria. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[0388] In this way, the present invention provides a system for fair and objective evaluation of test-taker performance, eliminating human subjective bias, reducing examiner workload and enabling consistent evaluation.

[0389] The processing flow will be explained below.

[0390] Step 1:

[0391] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[0392] Step 2:

[0393] The device's camera begins recording the user's actions as video.

[0394] Step 3:

[0395] The device's microphone begins recording the user's voice.

[0396] Step 4:

[0397] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[0398] Step 5:

[0399] The device stores the recorded video and audio data in local storage.

[0400] Step 6:

[0401] The device uploads the stored video and audio data to a central server via the Internet.

[0402] Step 7:

[0403] The server receives the uploaded video and audio data and stores it in storage.

[0404] Step 8:

[0405] The server analyzes the video data and extracts frames, which are then used to perform facial expression and gesture recognition.

[0406] Step 9:

[0407] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[0408] Step 10:

[0409] The server generates a score based on the analysis results and in accordance with preset evaluation criteria.

[0410] Step 11:

[0411] The server stores the generated scores and analysis details in a database.

[0412] Step 12:

[0413] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[0414] Step 13:

[0415] The administrator uses the administrator terminal to check each test taker's score, analysis details, and related information.

[0416] Step 14:

[0417] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[0418] Example 1

[0419] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0420] In conventional role-playing tests, evaluations are easily influenced by subjective bias, making it difficult to evaluate test takers' performance fairly and objectively. Furthermore, the burden on examiners is heavy, making it difficult to maintain consistency in evaluations.

[0421] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0422] In this invention, the server includes means for recording video and audio using a terminal used by the examinee, means for the terminal to upload the recorded video and audio to a central server, means for the server to analyze the data and perform facial expression recognition and gesture recognition, means for the server to analyze the data and perform voice tone analysis and speech content analysis, means for the server to evaluate the examinee based on the analysis results and generate a score, and means for the server to provide the evaluation results to an administrator terminal. This enables test evaluation to be conducted fairly and objectively, reduces the burden on examiners, and enables consistent evaluation standards to be maintained.

[0423] A "candidate" is a person who takes part in an examination and is assessed on a particular task or skill.

[0424] "Device" means an electronic device used by a candidate for the test, which is equipped with the capability to record video and audio.

[0425] "Recording" refers to the act of using a terminal to record video and audio data during the test.

[0426] "Central Server" means a central data processing device that receives, analyzes, and evaluates recorded video and audio data.

[0427] "Uploading" is the act of sending recorded data from a terminal to a central server.

[0428] "Storage" refers to a storage device for temporarily or permanently storing received data.

[0429] "Analysis" is the process of processing the recorded data and extracting the information necessary for evaluation.

[0430] "Facial expression recognition" is a technology that detects the facial expressions of test takers from video data and analyzes their emotional state.

[0431] "Gesture recognition" is a technology that detects the examinee's hand movements and posture from video data and analyzes their meaning.

[0432] "Voice tone" refers to characteristics of voice data such as pitch, strength, and intonation.

[0433] "Speech content" is text information of words and phrases contained in the voice data.

[0434] "Evaluation criteria" refers to the standards and scales established to fairly and objectively evaluate test takers' performance.

[0435] A "score" is a numerical value or evaluation index generated from the analysis results based on the evaluation criteria.

[0436] An "administrator terminal" is an electronic device used by the test administrator or examiner to check evaluation results and adjust evaluation criteria.

[0437] "Display" refers to the act of showing the evaluation results and analysis details on the screen of the administrator's terminal.

[0438] A "database" is a system that systematically stores and manages data such as analysis results and evaluation criteria.

[0439] "Application" is the act of reflecting the newly adjusted evaluation criteria in the next evaluation.

[0440] This is an evaluation system consisting of multiple subsystems for the purpose of fairly and objectively evaluating examinee performance. The system consists of three main components: examinee terminals, a central server, and an administrator terminal.

[0441] Examinee's device

[0442] The user uses a device with a dedicated recording application installed. This device is equipped with a camera and microphone, which are used to record video and audio during the test. The user starts the recording application and presses the "Start role-playing" button to start recording, and the "Stop recording" button to end recording. Once recording is complete, the device automatically uploads the recorded video and audio data to a central server.

[0443] Central Server

[0444] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs various analyses. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the audio data for tone and speech content. These analyses are performed using dedicated software such as facial expression recognition libraries and audio analysis tools. Based on the analysis results, a score is generated according to pre-set evaluation criteria. The generated score and analysis details are stored in a database.

[0445] Administrator terminal

[0446] The administrator uses the administrator terminal to retrieve the assessment results from the central server. The management screen of the administrator terminal displays each examinee's score, analysis details, and related information. In addition, the administrator can adjust the assessment criteria. When new assessment criteria are entered, they are saved in the database and applied to the next assessment.

[0447] Specific examples

[0448] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content of the speech from the audio data. Based on these analyses, a score is generated according to the evaluation criteria. The test administrator can view this score using the administrator device and adjust the evaluation criteria as necessary.

[0449] Examples of prompt statements

[0450] Example prompt 1: "Analyze the test-taker's video to recognize facial expressions and gestures. Then analyze the audio data to analyze speaking style and content to generate a score based on the assessment criteria."

[0451] Example prompt 2: "Analyze the uploaded test data and generate a score based on the test taker's facial expressions, gestures, tone, and speech content. Then store the generated score along with the analysis results in a database."

[0452] In this way, the system of the present invention allows exam evaluation to be carried out fairly and objectively, reduces the burden on examiners, and maintains consistent evaluation standards.

[0453] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0454] Step 1:

[0455] The user launches a dedicated recording application on their device. As input, a launch command is given, which starts preparation of the device's camera and microphone. As output, the initial screen of the application is displayed, ready for recording.

[0456] Step 2:

[0457] The user presses the "Start Role-Playing" button. The user's operation instruction (pressing the button) is given as input, and the device activates the camera and microphone to start recording video and audio. The output is that the recorded video and audio data is saved in the device's local storage in real time.

[0458] Step 3:

[0459] The user presses the "Stop Recording" button. The user's operation instruction is given as input, and the device stops recording. The output is a state in which the stopped video and audio data has been completely saved to local storage.

[0460] Step 4:

[0461] The device makes an API call to the central server to send the recorded data. The input is the recorded video and audio data that the device uploads to the central server. The output is the video and audio data received by the server.

[0462] Step 5:

[0463] The server receives data sent from the device and stores it in storage. The input is the uploaded video and audio data, which the server stores in temporary storage. The output is the data stored in storage.

[0464] Step 6:

[0465] The server extracts frames from the video data and performs facial expression and gesture recognition. The input is video data stored in storage, which the server passes to the facial expression recognition library for analysis. The output is analyzed facial expression and gesture data.

[0466] Step 7:

[0467] The server analyzes the voice data and performs voice tone analysis and speech content analysis. The input is the voice data stored in the storage, and the server passes it to the voice analysis tool. The output is the analyzed voice tone and speech content data.

[0468] Step 8:

[0469] The server generates a score based on the analysis results according to the set evaluation criteria. The input is the analyzed facial expression, gesture, voice tone, and speech content data, which the server inputs into the evaluation algorithm. The output is the generated score.

[0470] Step 9:

[0471] The server stores the generated score in a database. The input is the generated score, which the server records in the database. The output is the score stored in the database.

[0472] Step 10:

[0473] The administrator uses the administrator terminal to obtain the evaluation results from the central server. The input is a request to obtain the evaluation results, which the administrator terminal sends to the server. The output is to display the evaluation results sent from the server on the screen of the administrator terminal.

[0474] Step 11:

[0475] The administrator can adjust the evaluation criteria using the administrator terminal. The input is the adjusted new evaluation criteria, which the administrator terminal sends to the server. The output is that the new evaluation criteria are stored in the database, ready to be applied at the next evaluation.

[0476] (Application example 1)

[0477] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0478] It is difficult to fairly and objectively evaluate the customer service skills of employees in brick-and-mortar stores. Conventional methods rely on the subjective judgment of the evaluator, which can lead to inconsistency and a lack of fairness. Furthermore, employee training is time-consuming and costly, and providing effective feedback is difficult. The present invention solves these problems and provides a system for efficiently and objectively evaluating and training customer service skills in brick-and-mortar stores.

[0479] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0480] In this invention, the server includes: means for recording video and audio using a terminal used by the examinee; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the examinee based on the analysis results; and means for the server to provide the evaluation results to a manager's terminal. In a customer service skill evaluation system used in a brick-and-mortar store, means for recording the actions and speech of employees; means for uploading the recorded data to the central server and analyzing it to generate evaluation results; and means for a manager to check the evaluation results and adjust the evaluation criteria. This makes it possible to evaluate the customer service skills of employees fairly and objectively and provide effective feedback.

[0481] "Examination candidate terminal" refers to a device used by an examination candidate to record video and audio, and includes a camera, microphone, etc.

[0482] The "central server" is a computer system that receives and analyzes data uploaded from test taker terminals.

[0483] An "administrator terminal" is a device that displays the evaluation results and adjusts the evaluation criteria as necessary, and is generally a PC or tablet terminal.

[0484] "Facial expression recognition" is the process of analyzing facial features extracted from frames of video data to identify emotions and expressions.

[0485] "Gesture recognition" is the process of analyzing physical movements extracted from frames of video data and identifying their meaning.

[0486] "Voice tone analysis" is the process of analyzing audio data to evaluate the pitch, intensity, rhythm, etc. of a speaker's voice.

[0487] "Speech analysis" is the process of analyzing the content of audio data to identify the words being spoken and their content.

[0488] The "customer service skills evaluation system" is a system for fairly and objectively evaluating the customer service skills of employees in physical stores based on video and audio data.

[0489] "Recording means" refers to a method or device for recording video and audio using a camera or microphone.

[0490] "Uploading means" refers to a method or device for transmitting recorded video and audio data over a network to a central server.

[0491] "Analysis means" refers to software or algorithms that the central server uses to analyze the uploaded video and audio data.

[0492] An "assessment tool" is a method or algorithm used to evaluate candidates or employees based on the results of the analysis.

[0493] "Evaluation criteria" are the indicators or standards used to evaluate candidates or employees and are used for scoring purposes.

[0494] The present invention provides a system for fairly and objectively evaluating the customer service skills of employees in a brick-and-mortar store. Specific embodiments of this system will be described in detail below.

[0495] 1. Exam-taker's device

[0496] The devices used by employees include smartphones, smart glasses, or head-mounted displays. These devices are equipped with cameras and microphones, allowing them to record video and audio. Recording begins by launching a dedicated application installed on the device and pressing the "Start Role-Playing" button. When the customer service simulation is finished, the user presses the "Stop Recording" button, and the recorded video and audio data is automatically uploaded to the central server.

[0497] 2. Central Server

[0498] The central server receives the data uploaded from the test taker's device and stores it in storage. The server then performs several processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. This is done using the OpenCV library. The recorded audio data is also analyzed for tone and speech content. This is done using the PyAudio library. Based on the results of these analyses, a score is generated according to the evaluation criteria, and the evaluation results are stored in a database.

[0499] 3. Administrator terminal

[0500] The manager terminal is used by the store manager. The manager terminal retrieves the evaluation results from the central server and displays them on a dedicated management screen. On this management screen, each employee's score, analysis details, and related information can be viewed. In addition, the manager can adjust the evaluation criteria through this screen. New evaluation criteria are entered from the manager terminal, and the central server saves them in the database and applies them the next time an employee is evaluated.

[0501] 4. Specific Examples

[0502] For example, consider a case where a newly hired employee at a store is undergoing customer service training. The employee wears smart glasses and performs customer service role-playing with a colleague playing the role of a customer. The recorded video and audio are uploaded to a server, where facial expressions, speech patterns, and gestures are analyzed in detail. A manager can view the results on a manager's device, evaluate the employee's customer service skills, and provide necessary feedback. This system enables consistent and fair evaluations, leading to improved employee skills.

[0503] Generative AI model prompt example

[0504] Analyze the following video and audio data to evaluate an employee's customer service skills. Generate scores for each of the following items: facial expressions, gestures, tone of voice, and content of speech, and output an overall score:

[0505] <Video data URL>

[0506] <Audio data URL>

[0507] In this way, the system of the present invention makes it possible to fairly and objectively evaluate test takers' performance and provide effective feedback, thereby enabling employees to quickly and effectively improve their customer service skills.

[0508] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0509] Step 1:

[0510] Employees will use smart devices to conduct customer service simulations.

[0511] Specifically, employees wear a smartphone, smart glasses, or a head-mounted display and perform a customer service role-play. To do so, they launch the recording application and press the "Start role-playing" button to begin recording video and audio.

[0512] Input: Video and audio data collected from the camera and microphone built into your smart device.

[0513] Output: Recorded video and audio data.

[0514] Step 2:

[0515] After the device finishes recording, it uploads the video and audio data to a central server.

[0516] Specifically, when the user presses the "Stop Recording" button, the device automatically transmits the recorded data to a central server via secure communication, ensuring the safety of the data.

[0517] Input: Video and audio data after recording has finished.

[0518] Output: Data is uploaded to a central server.

[0519] Step 3:

[0520] The server receives the uploaded data and stores it in storage.

[0521] Specifically, the server converts the received data into an appropriate format and stores it in a database or file system.

[0522] Input: Uploaded video and audio data.

[0523] Output: Securely stored video and audio data.

[0524] Step 4:

[0525] The server extracts frames from the video data and performs facial expression and gesture recognition.

[0526] Specifically, it uses the OpenCV library to split the video data into frames, analyzes the facial features in each frame, and identifies the employee's facial expressions and gestures.

[0527] Input: Stored video data.

[0528] Output: Analysis results of facial expression and gesture recognition.

[0529] Step 5:

[0530] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[0531] Specifically, it uses the PyAudio library to extract features from audio data, evaluate the pitch, intensity, and rhythm of the speaker's voice, and identify what is being said.

[0532] Input: Stored audio data.

[0533] Output: Voice tone analysis and speech content analysis.

[0534] Step 6:

[0535] The server generates a score based on the analysis results and in accordance with the evaluation criteria.

[0536] Specifically, based on the analysis results, pre-set evaluation criteria are applied to generate a numerical score for the employee's customer service skills.

[0537] Input: Results of facial expression recognition, gesture recognition, voice tone analysis, and speech content analysis.

[0538] Output: Scored evaluation results.

[0539] Step 7:

[0540] The server provides the evaluation results to the administrator terminal.

[0541] Specifically, the generated scores and analysis details will be displayed on a dedicated management screen so that administrators can check them.

[0542] Input: Scored assessment results.

[0543] Output: Evaluation results displayed on the administrator's terminal.

[0544] Step 8:

[0545] The administrator adjusts the evaluation criteria and saves the new criteria on the server.

[0546] Specifically, the administrator inputs the evaluation criteria using the management terminal, and the server stores them in the database and applies them the next time the evaluation is conducted.

[0547] Input: New evaluation criteria from the administrator terminal.

[0548] Output: The new saved criterion.

[0549] The above are the specific processing steps for fairly and objectively evaluating the customer service skills of the examinee.

[0550] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0551] The present invention provides a system for fairly and objectively evaluating test takers in role-playing tests such as customer service skills tests and certification tests. Furthermore, the present invention enables more precise evaluations by incorporating an emotion engine that analyzes the user's emotions. A specific embodiment of this system is described below.

[0552] Subsystem 1: Examinee's terminal

[0553] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0554] Subsystem 2: Central Server

[0555] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0556] Subsystem 3: Administrator terminal

[0557] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0558] Specific examples

[0559] Let's take an example where a candidate is taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes their speaking style and content from the audio data. In addition, an emotion engine analyzes the candidate's emotions using the video and audio data. Based on this analysis, a score is generated according to the evaluation criteria, and additional evaluation criteria are applied based on the emotion analysis results. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[0560] In this way, the present invention provides a system that evaluates test takers' performance fairly and objectively, eliminating human subjective bias. Furthermore, the use of an emotion engine enables advanced evaluation that takes test takers' emotional states into account. The system of the present invention reduces the workload of examiners and ensures consistent evaluation.

[0561] The processing flow will be explained below.

[0562] Step 1:

[0563] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[0564] Step 2:

[0565] The device's camera begins recording the user's actions as video.

[0566] Step 3:

[0567] The device's microphone begins recording the user's voice.

[0568] Step 4:

[0569] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[0570] Step 5:

[0571] The device stores the recorded video and audio data in local storage.

[0572] Step 6:

[0573] The device uploads the stored video and audio data to a central server via the Internet.

[0574] Step 7:

[0575] The server receives the uploaded video and audio data and stores it in storage.

[0576] Step 8:

[0577] The server analyzes the video data and extracts frames, which are then used to perform facial expression recognition and analyze the user's facial features.

[0578] Step 9:

[0579] The server performs gesture recognition from video data and analyzes the user's hand and body movements.

[0580] Step 10:

[0581] The server analyzes the voice data, analyzes the voice tone and the content of the speech, specifically extracting voice waveform features and performing language analysis.

[0582] Step 11:

[0583] The server uses an emotion engine to perform emotion analysis based on the video and audio data. It identifies the user's emotional state by analyzing facial expressions from the video data and tone of voice and content of speech from the audio data.

[0584] Step 12:

[0585] The server generates a score based on the analysis results according to a preset evaluation criterion, which also incorporates the emotion analysis results.

[0586] Step 13:

[0587] The server stores the generated scores and analysis details in a database.

[0588] Step 14:

[0589] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[0590] Step 15:

[0591] The administrator uses the administrator terminal to check each test taker's score, analysis details, and emotion analysis results.

[0592] Step 16:

[0593] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[0594] Example 2

[0595] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0596] Traditional role-playing tests often rely on the examiner's subjective evaluation, making it difficult to achieve fair and objective evaluation. Furthermore, because it is difficult to analyze emotions, it is not possible to perform sophisticated evaluations that take into account the examinee's emotional state. Furthermore, there are challenges in that it is difficult for test administrators to consistently apply evaluation criteria.

[0597] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring audio and video by an information terminal used by the examinee, means for transferring the audio and video acquired by the information terminal to a central data processing device, means for the central data processing device to receive and analyze the transferred data, means for the central data processing device to evaluate the examinee based on the analysis results, and means for the central data processing device to provide the evaluation results to an administrator's information terminal. This makes it possible to evaluate the examinee's performance fairly and objectively, eliminate the subjective bias of the examiner, and perform consistent evaluations.

[0598] "Information terminal" means a device used by a test taker during the test to obtain audio and video.

[0599] The "central data processing device" is a device that receives, analyzes, and evaluates audio and video data transferred from the examinee's information terminal.

[0600] The "analysis means" is a function that allows the central data processing unit to process audio and video data and perform facial expression recognition, action recognition, tone analysis, and analysis of audio content.

[0601] The "evaluation means" is a function that allows the central data processing unit to evaluate and score the performance of the examinee based on the analysis results.

[0602] The "administrator's information terminal" is a device that displays the evaluation results and analysis results provided by the central data processing unit and enables the evaluation criteria to be changed.

[0603] "Acquisition means" refers to the function that enables an information terminal to acquire audio and video.

[0604] The "transfer means" is a function for transmitting data acquired by the information terminal to the central data processing unit.

[0605] The "memory device" is a storage device for saving information such as analysis results and evaluation criteria in the central data processing unit.

[0606] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Furthermore, by combining an emotion engine, more precise evaluations can be performed. The system of the present invention is primarily composed of an information terminal used by the examinee, a central data processing unit, and an information terminal for the administrator.

[0607] Exam taker information terminal

[0608] The terminal is a device used by the test taker to capture audio and video during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the terminal and starts recording by pressing the "Start Role-Playing" button. The application uses the camera to record the test taker's actions and the microphone to record audio. When the test is finished, the user presses the "Stop Recording" button, and the terminal automatically uploads the recorded data to the central data processing unit.

[0609] Central Data Processing Unit

[0610] The server receives the data uploaded from the test taker's device and stores it in storage. Next, frames are extracted from the video data and facial expression and movement recognition is performed. This is done using video analysis software such as OpenCV. Furthermore, the audio data is analyzed for tone and audio content using audio analysis software such as LIBROSA. Furthermore, an emotion engine (e.g., an emotion recognition AI module) is used to analyze the test taker's emotional state. Based on the results of these analyses, a score is generated according to the evaluation criteria and stored in a database.

[0611] Administrator information terminal

[0612] The administrator terminal is a device used by the test administrator and examiners. The administrator terminal obtains the evaluation results from the central data processing device and displays them on the management screen. On this management screen, each examinee's score, detailed analysis results, and related information can be viewed. The administrator can adjust the evaluation criteria as needed. New evaluation criteria are entered from the administrator terminal and sent to the central data processing device. The server saves the new evaluation criteria in storage and applies them to the next evaluation.

[0613] Specific examples

[0614] For example, consider a test taker taking a customer service etiquette test. First, the test taker uses a device to begin the test and records their performance using a camera and microphone. The user introduces themselves and role-plays interacting with virtual customers. Once the recording is complete, the device uploads the data to a central data processing unit. The server analyzes the uploaded video data and recognizes the test taker's facial expressions and movements. It also analyzes the audio data to evaluate speaking style, tone, and content. It also uses an emotion engine to analyze the test taker's emotions and evaluates factors such as stress and friendliness. A score is generated based on these results, and the test taker is assigned a grade according to the evaluation criteria.

[0615] Prompt Sentence Examples

[0616] "This system records video and audio of test takers taking customer service etiquette tests and uploads them to a central server. The server analyzes facial expressions and gestures from the video and the content of speech from the audio. It also uses an emotion engine to analyze the test takers' emotions. Finally, it generates a score based on the evaluation criteria and displays it on an administrator's terminal."

[0617] In this way, test takers' performance can be evaluated fairly and objectively, eliminating subjective bias and ensuring consistent evaluation. Using an emotion engine enables advanced evaluation that takes test takers' emotional states into account, reducing the workload of test administrators and enabling more precise evaluation.

[0618] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0619] Program processing steps

[0620] Step 1:

[0621] The user launches a dedicated recording application on their information terminal. When the application launches, it checks the camera and microphone readiness and displays that they are available for use. This allows real-time data of the examinee's movements and speech to be obtained as input.

[0622] Step 2:

[0623] The user presses the "Start Role-Playing" button to begin recording. The device uses the camera to record the examinee's actions as video and the microphone to record audio. Once video and audio data begins to be recorded, it is continuously saved. The input is real-time video and audio, and the output is recorded video and audio files.

[0624] Step 3:

[0625] When the test is finished, the user presses the "End Recording" button. The terminal saves the recorded data as a file and prepares it for upload to the central data processing unit. The input is the recorded video and audio files, and the output is a data package for uploading.

[0626] Step 4:

[0627] The terminals upload the recorded video and audio data to the central data processing unit. The data is transferred over the network and reaches the central data processing unit. The input is the data package for uploading, and the output is the data received by the central data processing unit.

[0628] Step 5:

[0629] The server receives the video and audio data uploaded from the terminal and stores it in storage. The input is the received data package, and the output is a data file stored in the storage.

[0630] Step 6:

[0631] The server extracts frames from the video data and performs facial expression and movement recognition. This uses dedicated video analysis software (e.g., OpenCV). The input is a saved video file, and the output is facial expression and movement data.

[0632] Step 7:

[0633] The server analyzes the audio data and performs a timbre and audio content analysis using audio analysis software (e.g., LIBROSA). The input is a saved audio file, and the output is timbre and audio content data.

[0634] Step 8:

[0635] The server uses an emotion engine to perform emotion analysis based on video and audio data. The inputs are facial expression data, movement data, tone data, and audio content data, and the output is emotion data.

[0636] Step 9:

[0637] The server generates a score based on the analysis results and evaluation criteria. The evaluation criteria are set in advance and points are calculated based on these. The inputs are facial expression data, movement data, tone data, voice content data, and emotion data, and the output is a final score and an evaluation report.

[0638] Step 10:

[0639] The server stores the generated scores and analysis details in a database. The input is the final score and the evaluation report, and the output is the evaluation results stored in the database.

[0640] Step 11:

[0641] The administrator terminal obtains the evaluation results from the central data processing unit and displays them on the management screen. The administrator can check each examinee's score and detailed analysis results. The input is the evaluation results from the database, and the output is the information displayed on the management screen.

[0642] Step 12:

[0643] The administrator adjusts the evaluation criteria as needed. The administrator terminal sends the new evaluation criteria to the central data processing unit. The input is the new evaluation criteria entered by the administrator, and the output is the updated evaluation criteria stored in the central data processing unit.

[0644] The above processing steps realize a system for evaluating the performance of test takers fairly and objectively.

[0645] (Application example 2)

[0646] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0647] Current customer service skill evaluation systems often rely on subjective evaluations, making it difficult to provide fair and objective evaluations. Furthermore, they lack real-time analysis of customer service behavior and detailed emotion analysis using emotion engines, making it cumbersome to provide feedback on specific areas for improvement to staff. This makes it difficult to efficiently support staff growth.

[0648] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording video and audio using a terminal used by the test taker; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the test taker based on the analysis results; means for the server to provide the evaluation results to an administrator terminal; means for the store staff to record video and audio in real time using smart glasses while serving customers; and means for transmitting the recorded data to the server and analyzing facial expressions, gestures, and tone of voice using an emotion engine. This enables objective and fair evaluation of customer service skills and also enables efficient real-time feedback of specific areas for improvement to be provided to staff.

[0649] "Device used by the examinee" refers to the device used by the examinee during the examination to record video and audio.

[0650] "Central Server" is a computer system established to receive, analyze, and evaluate data uploaded from the terminals.

[0651] The "emotion engine" is software that analyzes facial expressions, gestures, and tone of voice from video and audio data to determine the user's emotions.

[0652] "Smart glasses" are glasses-type mobile devices worn by users that can record video and audio in real time.

[0653] "Facial expression recognition" is a technology that analyzes facial expressions from recorded video data.

[0654] "Gesture recognition" is a technology that analyzes hand and body movements from recorded video data.

[0655] "Voice tone analysis" is a technology that analyzes speaking style and tone of voice from recorded audio data.

[0656] "Evaluation criteria" are the criteria for evaluating test takers' skills and performance based on the recorded and analyzed data.

[0657] An "administrator terminal" is a device used by an administrator to check the evaluation results and adjust the evaluation criteria.

[0658] "Real-time feedback" is the process of providing staff with evaluation and analysis results obtained during customer service on the spot.

[0659] The present invention provides a system for fairly and objectively evaluating test takers in customer service skills and qualification examinations. Specific embodiments of this system are described below.

[0660] Subsystem 1: Examinee's terminal

[0661] The device is used by the test taker to record video and audio during the test. It includes a camera, microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0662] Subsystem 2: Central Server

[0663] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0664] Subsystem 3: Administrator terminal

[0665] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0666] Application example: Smart customer service evaluation assistant in brick-and-mortar stores

[0667] This system can be applied as an application installed on smart glasses to support the evaluation of customer service staff in brick-and-mortar stores. The system records video and audio in real time while the staff are serving customers and transmits the data to a server.

[0668] Specifically, staff wear smart glasses and record video and audio data while serving customers. The server receives this data and uses an emotion engine to analyze facial expressions, gestures, and tone of voice. Based on the analysis results, staff are then scored for their customer service skills. Managers can check the scores and analysis results and provide specific feedback to staff on areas for improvement in real time.

[0669] Hardware and Software

[0670] Hardware

[0671] Smart glasses (e.g. Google Glass)

[0672] Exam taker's device (camera, microphone)

[0673] software

[0674] Recording application (video and audio recording)

[0675] Emotion engine (analysis of facial expressions, gestures, and voice tone)

[0676] Server system (receiving, analyzing, and evaluating data)

[0677] Examples of prompt statements

[0678] "For customer service training evaluation, please provide specific program code for analyzing recorded audio and video data and recognizing facial expressions, gestures, and tone of voice."

[0679] In this way, we provide a system that uses smart glasses to enable real-time customer service evaluation and efficiently support staff in improving their skills.

[0680] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0681] Step 1:

[0682] The user puts on the smart glasses and begins serving customers. While serving customers, the smart glasses' camera and microphone record video and audio in real time. The recorded video is saved frame by frame, and the audio is continuously saved in a buffer as data. The input is the actual customer service scene, and the output is the recorded video and audio data.

[0683] Step 2:

[0684] The device's recording application starts recording by pressing the "Start Role-Playing" button. The camera captures video as successive frames, and the microphone captures audio data. When recording is complete, the user presses the "Stop Recording" button to stop recording. The input is the user's actions, and the output is the stopped video and audio files.

[0685] Step 3:

[0686] The device uploads the recorded video and audio data to a central server. After recording is complete, the device application sends the data to the server using the FTP protocol or HTTP POST request. The input is the recorded video and audio files, and the output is the data stored on the server.

[0687] Step 4:

[0688] The server receives the uploaded video and audio data and saves it in storage. The server receives the HTTP request and saves the video and audio files in the specified directory. The input is the data sent from the device, and the output is the data saved in storage.

[0689] Step 5:

[0690] The server extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. The emotion engine uses, for example, Google Cloud Vision API or Microsoft Azure Face API. The input is each frame of the video data, and the output is recognized facial expression data, gesture data, and emotion data.

[0691] Step 6:

[0692] The server analyzes the recorded voice data, analyzing voice tone, speech content, and emotion using an emotion engine. Voice analysis uses Google Cloud Speech-to-Text API and IBM Watson Speech to Text, among others. The input is voice data, and the output is voice tone data, speech content data, and emotion data.

[0693] Step 7:

[0694] The server evaluates the examinee's skills and performance based on the analysis results and generates a score according to pre-set evaluation criteria. The evaluation criteria are set by the system administrator. The inputs are facial expression data, gesture data, voice tone data, and speech content data, and the output is an evaluation score.

[0695] Step 8:

[0696] The server stores the generated scores and analysis details in a database. The inputs are the assessment scores and analysis results, and the output is the assessment data stored in the database.

[0697] Step 9:

[0698] The administrator terminal obtains the evaluation results from the server and displays them on the management screen. The administrator checks the evaluation results and analysis details and adjusts the evaluation criteria as necessary. The input is the evaluation data, and the output is the evaluation results displayed on the administrator terminal.

[0699] Step 10:

[0700] The server receives the evaluation criteria data adjusted by the administrator terminal and stores it in the database. The new criteria are applied at the next evaluation. The input is the new evaluation criteria data, and the output is the evaluation criteria stored in the database.

[0701] The above steps enable real-time customer service skill evaluation using smart glasses, providing objective and fair evaluations.

[0702] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0703] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0704] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0705] [Third embodiment]

[0706] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0707] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0708] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0709] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0710] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0711] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0712] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0713] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0714] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0715] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0716] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0717] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0718] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Specific embodiments of this system are described below.

[0719] Subsystem 1: Examinee's terminal

[0720] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0721] Subsystem 2: Central Server

[0722] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the recorded audio data for tone and speech content. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0723] Subsystem 3: Administrator terminal

[0724] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0725] Specific examples

[0726] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content from the audio data. Based on this analysis, a score is generated according to the evaluation criteria. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[0727] In this way, the present invention provides a system for fair and objective evaluation of test-taker performance, eliminating human subjective bias, reducing examiner workload and enabling consistent evaluation.

[0728] The processing flow will be explained below.

[0729] Step 1:

[0730] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[0731] Step 2:

[0732] The device's camera begins recording the user's actions as video.

[0733] Step 3:

[0734] The device's microphone begins recording the user's voice.

[0735] Step 4:

[0736] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[0737] Step 5:

[0738] The device stores the recorded video and audio data in local storage.

[0739] Step 6:

[0740] The device uploads the stored video and audio data to a central server via the Internet.

[0741] Step 7:

[0742] The server receives the uploaded video and audio data and stores it in storage.

[0743] Step 8:

[0744] The server analyzes the video data and extracts frames, which are then used to perform facial expression and gesture recognition.

[0745] Step 9:

[0746] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[0747] Step 10:

[0748] The server generates a score based on the analysis results and in accordance with preset evaluation criteria.

[0749] Step 11:

[0750] The server stores the generated scores and analysis details in a database.

[0751] Step 12:

[0752] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[0753] Step 13:

[0754] The administrator uses the administrator terminal to check each test taker's score, analysis details, and related information.

[0755] Step 14:

[0756] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[0757] Example 1

[0758] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0759] In conventional role-playing tests, evaluations are easily influenced by subjective bias, making it difficult to evaluate test takers' performance fairly and objectively. Furthermore, the burden on examiners is heavy, making it difficult to maintain consistency in evaluations.

[0760] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0761] In this invention, the server includes means for recording video and audio using a terminal used by the examinee, means for the terminal to upload the recorded video and audio to a central server, means for the server to analyze the data and perform facial expression recognition and gesture recognition, means for the server to analyze the data and perform voice tone analysis and speech content analysis, means for the server to evaluate the examinee based on the analysis results and generate a score, and means for the server to provide the evaluation results to an administrator terminal. This enables test evaluation to be conducted fairly and objectively, reduces the burden on examiners, and enables consistent evaluation standards to be maintained.

[0762] A "candidate" is a person who takes part in an examination and is assessed on a particular task or skill.

[0763] "Device" means an electronic device used by a candidate for the test, which is equipped with the capability to record video and audio.

[0764] "Recording" refers to the act of using a terminal to record video and audio data during the test.

[0765] "Central Server" means a central data processing device that receives, analyzes, and evaluates recorded video and audio data.

[0766] "Uploading" is the act of sending recorded data from a terminal to a central server.

[0767] "Storage" refers to a storage device for temporarily or permanently storing received data.

[0768] "Analysis" is the process of processing the recorded data and extracting the information necessary for evaluation.

[0769] "Facial expression recognition" is a technology that detects the facial expressions of test takers from video data and analyzes their emotional state.

[0770] "Gesture recognition" is a technology that detects the examinee's hand movements and posture from video data and analyzes their meaning.

[0771] "Voice tone" refers to characteristics of voice data such as pitch, strength, and intonation.

[0772] "Speech content" is text information of words and phrases contained in the voice data.

[0773] "Evaluation criteria" refers to the standards and scales established to fairly and objectively evaluate test takers' performance.

[0774] A "score" is a numerical value or evaluation index generated from the analysis results based on the evaluation criteria.

[0775] An "administrator terminal" is an electronic device used by the test administrator or examiner to check evaluation results and adjust evaluation criteria.

[0776] "Display" refers to the act of showing the evaluation results and analysis details on the screen of the administrator's terminal.

[0777] A "database" is a system that systematically stores and manages data such as analysis results and evaluation criteria.

[0778] "Application" is the act of reflecting the newly adjusted evaluation criteria in the next evaluation.

[0779] This is an evaluation system consisting of multiple subsystems for the purpose of fairly and objectively evaluating examinee performance. The system consists of three main components: examinee terminals, a central server, and an administrator terminal.

[0780] Examinee's device

[0781] The user uses a device with a dedicated recording application installed. This device is equipped with a camera and microphone, which are used to record video and audio during the test. The user starts the recording application and presses the "Start role-playing" button to start recording, and the "Stop recording" button to end recording. Once recording is complete, the device automatically uploads the recorded video and audio data to a central server.

[0782] Central Server

[0783] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs various analyses. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the audio data for tone and speech content. These analyses are performed using dedicated software such as facial expression recognition libraries and audio analysis tools. Based on the analysis results, a score is generated according to pre-set evaluation criteria. The generated score and analysis details are stored in a database.

[0784] Administrator terminal

[0785] The administrator uses the administrator terminal to retrieve the assessment results from the central server. The management screen of the administrator terminal displays each examinee's score, analysis details, and related information. In addition, the administrator can adjust the assessment criteria. When new assessment criteria are entered, they are saved in the database and applied to the next assessment.

[0786] Specific examples

[0787] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content of the speech from the audio data. Based on these analyses, a score is generated according to the evaluation criteria. The test administrator can view this score using the administrator device and adjust the evaluation criteria as necessary.

[0788] Examples of prompt statements

[0789] Example prompt 1: "Analyze the test-taker's video to recognize facial expressions and gestures. Then analyze the audio data to analyze speaking style and content to generate a score based on the assessment criteria."

[0790] Example prompt 2: "Analyze the uploaded test data and generate a score based on the test taker's facial expressions, gestures, tone, and speech content. Then store the generated score along with the analysis results in a database."

[0791] In this way, the system of the present invention allows exam evaluation to be carried out fairly and objectively, reduces the burden on examiners, and maintains consistent evaluation standards.

[0792] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0793] Step 1:

[0794] The user launches a dedicated recording application on their device. As input, a launch command is given, which starts preparation of the device's camera and microphone. As output, the initial screen of the application is displayed, ready for recording.

[0795] Step 2:

[0796] The user presses the "Start Role-Playing" button. The user's operation instruction (pressing the button) is given as input, and the device activates the camera and microphone to start recording video and audio. The output is that the recorded video and audio data is saved in the device's local storage in real time.

[0797] Step 3:

[0798] The user presses the "Stop Recording" button. The user's operation instruction is given as input, and the device stops recording. The output is a state in which the stopped video and audio data has been completely saved to local storage.

[0799] Step 4:

[0800] The device makes an API call to the central server to send the recorded data. The input is the recorded video and audio data that the device uploads to the central server. The output is the video and audio data received by the server.

[0801] Step 5:

[0802] The server receives data sent from the device and stores it in storage. The input is the uploaded video and audio data, which the server stores in temporary storage. The output is the data stored in storage.

[0803] Step 6:

[0804] The server extracts frames from the video data and performs facial expression and gesture recognition. The input is video data stored in storage, which the server passes to the facial expression recognition library for analysis. The output is analyzed facial expression and gesture data.

[0805] Step 7:

[0806] The server analyzes the voice data and performs voice tone analysis and speech content analysis. The input is the voice data stored in the storage, and the server passes it to the voice analysis tool. The output is the analyzed voice tone and speech content data.

[0807] Step 8:

[0808] The server generates a score based on the analysis results according to the set evaluation criteria. The input is the analyzed facial expression, gesture, voice tone, and speech content data, which the server inputs into the evaluation algorithm. The output is the generated score.

[0809] Step 9:

[0810] The server stores the generated score in a database. The input is the generated score, which the server records in the database. The output is the score stored in the database.

[0811] Step 10:

[0812] The administrator uses the administrator terminal to obtain the evaluation results from the central server. The input is a request to obtain the evaluation results, which the administrator terminal sends to the server. The output is to display the evaluation results sent from the server on the screen of the administrator terminal.

[0813] Step 11:

[0814] The administrator can adjust the evaluation criteria using the administrator terminal. The input is the adjusted new evaluation criteria, which the administrator terminal sends to the server. The output is that the new evaluation criteria are stored in the database, ready to be applied at the next evaluation.

[0815] (Application example 1)

[0816] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0817] It is difficult to fairly and objectively evaluate the customer service skills of employees in brick-and-mortar stores. Conventional methods rely on the subjective judgment of the evaluator, which can lead to inconsistency and a lack of fairness. Furthermore, employee training is time-consuming and costly, and providing effective feedback is difficult. The present invention solves these problems and provides a system for efficiently and objectively evaluating and training customer service skills in brick-and-mortar stores.

[0818] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0819] In this invention, the server includes: means for recording video and audio using a terminal used by the examinee; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the examinee based on the analysis results; and means for the server to provide the evaluation results to a manager's terminal. In a customer service skill evaluation system used in a brick-and-mortar store, means for recording the actions and speech of employees; means for uploading the recorded data to the central server and analyzing it to generate evaluation results; and means for a manager to check the evaluation results and adjust the evaluation criteria. This makes it possible to evaluate the customer service skills of employees fairly and objectively and provide effective feedback.

[0820] "Examination candidate terminal" refers to a device used by an examination candidate to record video and audio, and includes a camera, microphone, etc.

[0821] The "central server" is a computer system that receives and analyzes data uploaded from test taker terminals.

[0822] An "administrator terminal" is a device that displays the evaluation results and adjusts the evaluation criteria as necessary, and is generally a PC or tablet terminal.

[0823] "Facial expression recognition" is the process of analyzing facial features extracted from frames of video data to identify emotions and expressions.

[0824] "Gesture recognition" is the process of analyzing physical movements extracted from frames of video data and identifying their meaning.

[0825] "Voice tone analysis" is the process of analyzing audio data to evaluate the pitch, intensity, rhythm, etc. of a speaker's voice.

[0826] "Speech analysis" is the process of analyzing the content of audio data to identify the words being spoken and their content.

[0827] The "customer service skills evaluation system" is a system for fairly and objectively evaluating the customer service skills of employees in physical stores based on video and audio data.

[0828] "Recording means" refers to a method or device for recording video and audio using a camera or microphone.

[0829] "Uploading means" refers to a method or device for transmitting recorded video and audio data over a network to a central server.

[0830] "Analysis means" refers to software or algorithms that the central server uses to analyze the uploaded video and audio data.

[0831] An "assessment tool" is a method or algorithm used to evaluate candidates or employees based on the results of the analysis.

[0832] "Evaluation criteria" are the indicators or standards used to evaluate candidates or employees and are used for scoring purposes.

[0833] The present invention provides a system for fairly and objectively evaluating the customer service skills of employees in a brick-and-mortar store. Specific embodiments of this system will be described in detail below.

[0834] 1. Exam-taker's device

[0835] The devices used by employees include smartphones, smart glasses, or head-mounted displays. These devices are equipped with cameras and microphones, allowing them to record video and audio. Recording begins by launching a dedicated application installed on the device and pressing the "Start Role-Playing" button. When the customer service simulation is finished, the user presses the "Stop Recording" button, and the recorded video and audio data is automatically uploaded to the central server.

[0836] 2. Central Server

[0837] The central server receives the data uploaded from the test taker's device and stores it in storage. The server then performs several processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. This is done using the OpenCV library. The recorded audio data is also analyzed for tone and speech content. This is done using the PyAudio library. Based on the results of these analyses, a score is generated according to the evaluation criteria, and the evaluation results are stored in a database.

[0838] 3. Administrator terminal

[0839] The manager terminal is used by the store manager. The manager terminal retrieves the evaluation results from the central server and displays them on a dedicated management screen. On this management screen, each employee's score, analysis details, and related information can be viewed. In addition, the manager can adjust the evaluation criteria through this screen. New evaluation criteria are entered from the manager terminal, and the central server saves them in the database and applies them the next time an employee is evaluated.

[0840] 4. Specific Examples

[0841] For example, consider a case where a newly hired employee at a store is undergoing customer service training. The employee wears smart glasses and performs customer service role-playing with a colleague playing the role of a customer. The recorded video and audio are uploaded to a server, where facial expressions, speech patterns, and gestures are analyzed in detail. A manager can view the results on a manager's device, evaluate the employee's customer service skills, and provide necessary feedback. This system enables consistent and fair evaluations, leading to improved employee skills.

[0842] Generative AI model prompt example

[0843] Analyze the following video and audio data to evaluate an employee's customer service skills. Generate scores for each of the following items: facial expressions, gestures, tone of voice, and content of speech, and output an overall score:

[0844] <Video data URL>

[0845] <Audio data URL>

[0846] In this way, the system of the present invention makes it possible to fairly and objectively evaluate test takers' performance and provide effective feedback, thereby enabling employees to quickly and effectively improve their customer service skills.

[0847] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0848] Step 1:

[0849] Employees will use smart devices to conduct customer service simulations.

[0850] Specifically, employees wear a smartphone, smart glasses, or a head-mounted display and perform a customer service role-play. To do so, they launch the recording application and press the "Start role-playing" button to begin recording video and audio.

[0851] Input: Video and audio data collected from the camera and microphone built into your smart device.

[0852] Output: Recorded video and audio data.

[0853] Step 2:

[0854] After the device finishes recording, it uploads the video and audio data to a central server.

[0855] Specifically, when the user presses the "Stop Recording" button, the device automatically transmits the recorded data to a central server via secure communication, ensuring the safety of the data.

[0856] Input: Video and audio data after recording has finished.

[0857] Output: Data is uploaded to a central server.

[0858] Step 3:

[0859] The server receives the uploaded data and stores it in storage.

[0860] Specifically, the server converts the received data into an appropriate format and stores it in a database or file system.

[0861] Input: Uploaded video and audio data.

[0862] Output: Securely stored video and audio data.

[0863] Step 4:

[0864] The server extracts frames from the video data and performs facial expression and gesture recognition.

[0865] Specifically, it uses the OpenCV library to split the video data into frames, analyzes the facial features in each frame, and identifies the employee's facial expressions and gestures.

[0866] Input: Stored video data.

[0867] Output: Analysis results of facial expression and gesture recognition.

[0868] Step 5:

[0869] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[0870] Specifically, it uses the PyAudio library to extract features from audio data, evaluate the pitch, intensity, and rhythm of the speaker's voice, and identify what is being said.

[0871] Input: Stored audio data.

[0872] Output: Voice tone analysis and speech content analysis.

[0873] Step 6:

[0874] The server generates a score based on the analysis results and in accordance with the evaluation criteria.

[0875] Specifically, based on the analysis results, pre-set evaluation criteria are applied to generate a numerical score for the employee's customer service skills.

[0876] Input: Results of facial expression recognition, gesture recognition, voice tone analysis, and speech content analysis.

[0877] Output: Scored evaluation results.

[0878] Step 7:

[0879] The server provides the evaluation results to the administrator terminal.

[0880] Specifically, the generated scores and analysis details will be displayed on a dedicated management screen so that administrators can check them.

[0881] Input: Scored assessment results.

[0882] Output: Evaluation results displayed on the administrator's terminal.

[0883] Step 8:

[0884] The administrator adjusts the evaluation criteria and saves the new criteria on the server.

[0885] Specifically, the administrator inputs the evaluation criteria using the management terminal, and the server stores them in the database and applies them the next time the evaluation is conducted.

[0886] Input: New evaluation criteria from the administrator terminal.

[0887] Output: The new saved criterion.

[0888] The above are the specific processing steps for fairly and objectively evaluating the customer service skills of the examinee.

[0889] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0890] The present invention provides a system for fairly and objectively evaluating test takers in role-playing tests such as customer service skills tests and certification tests. Furthermore, the present invention enables more precise evaluations by incorporating an emotion engine that analyzes the user's emotions. A specific embodiment of this system is described below.

[0891] Subsystem 1: Examinee's terminal

[0892] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[0893] Subsystem 2: Central Server

[0894] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[0895] Subsystem 3: Administrator terminal

[0896] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[0897] Specific examples

[0898] Let's take an example where a candidate is taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes their speaking style and content from the audio data. In addition, an emotion engine analyzes the candidate's emotions using the video and audio data. Based on this analysis, a score is generated according to the evaluation criteria, and additional evaluation criteria are applied based on the emotion analysis results. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[0899] In this way, the present invention provides a system that evaluates test takers' performance fairly and objectively, eliminating human subjective bias. Furthermore, the use of an emotion engine enables advanced evaluation that takes test takers' emotional states into account. The system of the present invention reduces the workload of examiners and ensures consistent evaluation.

[0900] The processing flow will be explained below.

[0901] Step 1:

[0902] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[0903] Step 2:

[0904] The device's camera begins recording the user's actions as video.

[0905] Step 3:

[0906] The device's microphone begins recording the user's voice.

[0907] Step 4:

[0908] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[0909] Step 5:

[0910] The device stores the recorded video and audio data in local storage.

[0911] Step 6:

[0912] The device uploads the stored video and audio data to a central server via the Internet.

[0913] Step 7:

[0914] The server receives the uploaded video and audio data and stores it in storage.

[0915] Step 8:

[0916] The server analyzes the video data and extracts frames, which are then used to perform facial expression recognition and analyze the user's facial features.

[0917] Step 9:

[0918] The server performs gesture recognition from video data and analyzes the user's hand and body movements.

[0919] Step 10:

[0920] The server analyzes the voice data, analyzes the voice tone and the content of the speech, specifically extracting voice waveform features and performing language analysis.

[0921] Step 11:

[0922] The server uses an emotion engine to perform emotion analysis based on the video and audio data. It identifies the user's emotional state by analyzing facial expressions from the video data and tone of voice and content of speech from the audio data.

[0923] Step 12:

[0924] The server generates a score based on the analysis results according to a preset evaluation criterion, which also incorporates the emotion analysis results.

[0925] Step 13:

[0926] The server stores the generated scores and analysis details in a database.

[0927] Step 14:

[0928] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[0929] Step 15:

[0930] The administrator uses the administrator terminal to check each test taker's score, analysis details, and emotion analysis results.

[0931] Step 16:

[0932] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[0933] Example 2

[0934] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0935] Traditional role-playing tests often rely on the examiner's subjective evaluation, making it difficult to achieve fair and objective evaluation. Furthermore, because it is difficult to analyze emotions, it is not possible to perform sophisticated evaluations that take into account the examinee's emotional state. Furthermore, there are challenges in that it is difficult for test administrators to consistently apply evaluation criteria.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring audio and video by an information terminal used by the examinee, means for transferring the audio and video acquired by the information terminal to a central data processing device, means for the central data processing device to receive and analyze the transferred data, means for the central data processing device to evaluate the examinee based on the analysis results, and means for the central data processing device to provide the evaluation results to an administrator's information terminal. This makes it possible to evaluate the examinee's performance fairly and objectively, eliminate the subjective bias of the examiner, and perform consistent evaluations.

[0937] "Information terminal" means a device used by a test taker during the test to obtain audio and video.

[0938] The "central data processing device" is a device that receives, analyzes, and evaluates audio and video data transferred from the examinee's information terminal.

[0939] The "analysis means" is a function that allows the central data processing unit to process audio and video data and perform facial expression recognition, action recognition, tone analysis, and analysis of audio content.

[0940] The "evaluation means" is a function that allows the central data processing unit to evaluate and score the performance of the examinee based on the analysis results.

[0941] The "administrator's information terminal" is a device that displays the evaluation results and analysis results provided by the central data processing unit and enables the evaluation criteria to be changed.

[0942] "Acquisition means" refers to the function that enables an information terminal to acquire audio and video.

[0943] The "transfer means" is a function for transmitting data acquired by the information terminal to the central data processing unit.

[0944] The "memory device" is a storage device for saving information such as analysis results and evaluation criteria in the central data processing unit.

[0945] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Furthermore, by combining an emotion engine, more precise evaluations can be performed. The system of the present invention is primarily composed of an information terminal used by the examinee, a central data processing unit, and an information terminal for the administrator.

[0946] Exam taker information terminal

[0947] The terminal is a device used by the test taker to capture audio and video during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the terminal and starts recording by pressing the "Start Role-Playing" button. The application uses the camera to record the test taker's actions and the microphone to record audio. When the test is finished, the user presses the "Stop Recording" button, and the terminal automatically uploads the recorded data to the central data processing unit.

[0948] Central Data Processing Unit

[0949] The server receives the data uploaded from the test taker's device and stores it in storage. Next, frames are extracted from the video data and facial expression and movement recognition is performed. This is done using video analysis software such as OpenCV. Furthermore, the audio data is analyzed for tone and audio content using audio analysis software such as LIBROSA. Furthermore, an emotion engine (e.g., an emotion recognition AI module) is used to analyze the test taker's emotional state. Based on the results of these analyses, a score is generated according to the evaluation criteria and stored in a database.

[0950] Administrator information terminal

[0951] The administrator terminal is a device used by the test administrator and examiners. The administrator terminal obtains the evaluation results from the central data processing device and displays them on the management screen. On this management screen, each examinee's score, detailed analysis results, and related information can be viewed. The administrator can adjust the evaluation criteria as needed. New evaluation criteria are entered from the administrator terminal and sent to the central data processing device. The server saves the new evaluation criteria in storage and applies them to the next evaluation.

[0952] Specific examples

[0953] For example, consider a test taker taking a customer service etiquette test. First, the test taker uses a device to begin the test and records their performance using a camera and microphone. The user introduces themselves and role-plays interacting with virtual customers. Once the recording is complete, the device uploads the data to a central data processing unit. The server analyzes the uploaded video data and recognizes the test taker's facial expressions and movements. It also analyzes the audio data to evaluate speaking style, tone, and content. It also uses an emotion engine to analyze the test taker's emotions and evaluates factors such as stress and friendliness. A score is generated based on these results, and the test taker is assigned a grade according to the evaluation criteria.

[0954] Prompt Sentence Examples

[0955] "This system records video and audio of test takers taking customer service etiquette tests and uploads them to a central server. The server analyzes facial expressions and gestures from the video and the content of speech from the audio. It also uses an emotion engine to analyze the test takers' emotions. Finally, it generates a score based on the evaluation criteria and displays it on an administrator's terminal."

[0956] In this way, test takers' performance can be evaluated fairly and objectively, eliminating subjective bias and ensuring consistent evaluation. Using an emotion engine enables advanced evaluation that takes test takers' emotional states into account, reducing the workload of test administrators and enabling more precise evaluation.

[0957] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0958] Program processing steps

[0959] Step 1:

[0960] The user launches a dedicated recording application on their information terminal. When the application launches, it checks the camera and microphone readiness and displays that they are available for use. This allows real-time data of the examinee's movements and speech to be obtained as input.

[0961] Step 2:

[0962] The user presses the "Start Role-Playing" button to begin recording. The device uses the camera to record the examinee's actions as video and the microphone to record audio. Once video and audio data begins to be recorded, it is continuously saved. The input is real-time video and audio, and the output is recorded video and audio files.

[0963] Step 3:

[0964] When the test is finished, the user presses the "End Recording" button. The terminal saves the recorded data as a file and prepares it for upload to the central data processing unit. The input is the recorded video and audio files, and the output is a data package for uploading.

[0965] Step 4:

[0966] The terminals upload the recorded video and audio data to the central data processing unit. The data is transferred over the network and reaches the central data processing unit. The input is the data package for uploading, and the output is the data received by the central data processing unit.

[0967] Step 5:

[0968] The server receives the video and audio data uploaded from the terminal and stores it in storage. The input is the received data package, and the output is a data file stored in the storage.

[0969] Step 6:

[0970] The server extracts frames from the video data and performs facial expression and movement recognition. This uses dedicated video analysis software (e.g., OpenCV). The input is a saved video file, and the output is facial expression and movement data.

[0971] Step 7:

[0972] The server analyzes the audio data and performs a timbre and audio content analysis using audio analysis software (e.g., LIBROSA). The input is a saved audio file, and the output is timbre and audio content data.

[0973] Step 8:

[0974] The server uses an emotion engine to perform emotion analysis based on video and audio data. The inputs are facial expression data, movement data, tone data, and audio content data, and the output is emotion data.

[0975] Step 9:

[0976] The server generates a score based on the analysis results and evaluation criteria. The evaluation criteria are set in advance and points are calculated based on these. The inputs are facial expression data, movement data, tone data, voice content data, and emotion data, and the output is a final score and an evaluation report.

[0977] Step 10:

[0978] The server stores the generated scores and analysis details in a database. The input is the final score and the evaluation report, and the output is the evaluation results stored in the database.

[0979] Step 11:

[0980] The administrator terminal obtains the evaluation results from the central data processing unit and displays them on the management screen. The administrator can check each examinee's score and detailed analysis results. The input is the evaluation results from the database, and the output is the information displayed on the management screen.

[0981] Step 12:

[0982] The administrator adjusts the evaluation criteria as needed. The administrator terminal sends the new evaluation criteria to the central data processing unit. The input is the new evaluation criteria entered by the administrator, and the output is the updated evaluation criteria stored in the central data processing unit.

[0983] The above processing steps realize a system for evaluating the performance of test takers fairly and objectively.

[0984] (Application example 2)

[0985] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0986] Current customer service skill evaluation systems often rely on subjective evaluations, making it difficult to provide fair and objective evaluations. Furthermore, they lack real-time analysis of customer service behavior and detailed emotion analysis using emotion engines, making it cumbersome to provide feedback on specific areas for improvement to staff. This makes it difficult to efficiently support staff growth.

[0987] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording video and audio using a terminal used by the test taker; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the test taker based on the analysis results; means for the server to provide the evaluation results to an administrator terminal; means for the store staff to record video and audio in real time using smart glasses while serving customers; and means for transmitting the recorded data to the server and analyzing facial expressions, gestures, and tone of voice using an emotion engine. This enables objective and fair evaluation of customer service skills and also enables efficient real-time feedback of specific areas for improvement to be provided to staff.

[0988] "Device used by the examinee" refers to the device used by the examinee during the examination to record video and audio.

[0989] "Central Server" is a computer system established to receive, analyze, and evaluate data uploaded from the terminals.

[0990] The "emotion engine" is software that analyzes facial expressions, gestures, and tone of voice from video and audio data to determine the user's emotions.

[0991] "Smart glasses" are glasses-type mobile devices worn by users that can record video and audio in real time.

[0992] "Facial expression recognition" is a technology that analyzes facial expressions from recorded video data.

[0993] "Gesture recognition" is a technology that analyzes hand and body movements from recorded video data.

[0994] "Voice tone analysis" is a technology that analyzes speaking style and tone of voice from recorded audio data.

[0995] "Evaluation criteria" are the criteria for evaluating test takers' skills and performance based on the recorded and analyzed data.

[0996] An "administrator terminal" is a device used by an administrator to check the evaluation results and adjust the evaluation criteria.

[0997] "Real-time feedback" is the process of providing staff with evaluation and analysis results obtained during customer service on the spot.

[0998] The present invention provides a system for fairly and objectively evaluating test takers in customer service skills and qualification examinations. Specific embodiments of this system are described below.

[0999] Subsystem 1: Examinee's terminal

[1000] The device is used by the test taker to record video and audio during the test. It includes a camera, microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[1001] Subsystem 2: Central Server

[1002] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[1003] Subsystem 3: Administrator terminal

[1004] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[1005] Application example: Smart customer service evaluation assistant in brick-and-mortar stores

[1006] This system can be applied as an application installed on smart glasses to support the evaluation of customer service staff in brick-and-mortar stores. The system records video and audio in real time while the staff are serving customers and transmits the data to a server.

[1007] Specifically, staff wear smart glasses and record video and audio data while serving customers. The server receives this data and uses an emotion engine to analyze facial expressions, gestures, and tone of voice. Based on the analysis results, staff are then scored for their customer service skills. Managers can check the scores and analysis results and provide specific feedback to staff on areas for improvement in real time.

[1008] Hardware and Software

[1009] Hardware

[1010] Smart glasses (e.g. Google Glass)

[1011] Exam taker's device (camera, microphone)

[1012] software

[1013] Recording application (video and audio recording)

[1014] Emotion engine (analysis of facial expressions, gestures, and voice tone)

[1015] Server system (receiving, analyzing, and evaluating data)

[1016] Examples of prompt statements

[1017] "For customer service training evaluation, please provide specific program code for analyzing recorded audio and video data and recognizing facial expressions, gestures, and tone of voice."

[1018] In this way, we provide a system that uses smart glasses to enable real-time customer service evaluation and efficiently support staff in improving their skills.

[1019] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1020] Step 1:

[1021] The user puts on the smart glasses and begins serving customers. While serving customers, the smart glasses' camera and microphone record video and audio in real time. The recorded video is saved frame by frame, and the audio is continuously saved in a buffer as data. The input is the actual customer service scene, and the output is the recorded video and audio data.

[1022] Step 2:

[1023] The device's recording application starts recording by pressing the "Start Role-Playing" button. The camera captures video as successive frames, and the microphone captures audio data. When recording is complete, the user presses the "Stop Recording" button to stop recording. The input is the user's actions, and the output is the stopped video and audio files.

[1024] Step 3:

[1025] The device uploads the recorded video and audio data to a central server. After recording is complete, the device application sends the data to the server using the FTP protocol or HTTP POST request. The input is the recorded video and audio files, and the output is the data stored on the server.

[1026] Step 4:

[1027] The server receives the uploaded video and audio data and saves it in storage. The server receives the HTTP request and saves the video and audio files in the specified directory. The input is the data sent from the device, and the output is the data saved in storage.

[1028] Step 5:

[1029] The server extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. The emotion engine uses, for example, Google Cloud Vision API or Microsoft Azure Face API. The input is each frame of the video data, and the output is recognized facial expression data, gesture data, and emotion data.

[1030] Step 6:

[1031] The server analyzes the recorded voice data, analyzing voice tone, speech content, and emotion using an emotion engine. Voice analysis uses Google Cloud Speech-to-Text API and IBM Watson Speech to Text, among others. The input is voice data, and the output is voice tone data, speech content data, and emotion data.

[1032] Step 7:

[1033] The server evaluates the examinee's skills and performance based on the analysis results and generates a score according to pre-set evaluation criteria. The evaluation criteria are set by the system administrator. The inputs are facial expression data, gesture data, voice tone data, and speech content data, and the output is an evaluation score.

[1034] Step 8:

[1035] The server stores the generated scores and analysis details in a database. The inputs are the assessment scores and analysis results, and the output is the assessment data stored in the database.

[1036] Step 9:

[1037] The administrator terminal obtains the evaluation results from the server and displays them on the management screen. The administrator checks the evaluation results and analysis details and adjusts the evaluation criteria as necessary. The input is the evaluation data, and the output is the evaluation results displayed on the administrator terminal.

[1038] Step 10:

[1039] The server receives the evaluation criteria data adjusted by the administrator terminal and stores it in the database. The new criteria are applied at the next evaluation. The input is the new evaluation criteria data, and the output is the evaluation criteria stored in the database.

[1040] The above steps enable real-time customer service skill evaluation using smart glasses, providing objective and fair evaluations.

[1041] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1042] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1043] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1044] [Fourth embodiment]

[1045] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1046] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1047] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1048] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1049] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1050] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1051] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1052] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1053] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1054] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1055] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1056] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1057] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1058] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Specific embodiments of this system are described below.

[1059] Subsystem 1: Examinee's terminal

[1060] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[1061] Subsystem 2: Central Server

[1062] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the recorded audio data for tone and speech content. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[1063] Subsystem 3: Administrator terminal

[1064] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[1065] Specific examples

[1066] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content from the audio data. Based on this analysis, a score is generated according to the evaluation criteria. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[1067] In this way, the present invention provides a system for fair and objective evaluation of test-taker performance, eliminating human subjective bias, reducing examiner workload and enabling consistent evaluation.

[1068] The processing flow will be explained below.

[1069] Step 1:

[1070] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[1071] Step 2:

[1072] The device's camera begins recording the user's actions as video.

[1073] Step 3:

[1074] The device's microphone begins recording the user's voice.

[1075] Step 4:

[1076] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[1077] Step 5:

[1078] The device stores the recorded video and audio data in local storage.

[1079] Step 6:

[1080] The device uploads the stored video and audio data to a central server via the Internet.

[1081] Step 7:

[1082] The server receives the uploaded video and audio data and stores it in storage.

[1083] Step 8:

[1084] The server analyzes the video data and extracts frames, which are then used to perform facial expression and gesture recognition.

[1085] Step 9:

[1086] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[1087] Step 10:

[1088] The server generates a score based on the analysis results and in accordance with preset evaluation criteria.

[1089] Step 11:

[1090] The server stores the generated scores and analysis details in a database.

[1091] Step 12:

[1092] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[1093] Step 13:

[1094] The administrator uses the administrator terminal to check each test taker's score, analysis details, and related information.

[1095] Step 14:

[1096] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[1097] Example 1

[1098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1099] In conventional role-playing tests, evaluations are easily influenced by subjective bias, making it difficult to evaluate test takers' performance fairly and objectively. Furthermore, the burden on examiners is heavy, making it difficult to maintain consistency in evaluations.

[1100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1101] In this invention, the server includes means for recording video and audio using a terminal used by the examinee, means for the terminal to upload the recorded video and audio to a central server, means for the server to analyze the data and perform facial expression recognition and gesture recognition, means for the server to analyze the data and perform voice tone analysis and speech content analysis, means for the server to evaluate the examinee based on the analysis results and generate a score, and means for the server to provide the evaluation results to an administrator terminal. This enables test evaluation to be conducted fairly and objectively, reduces the burden on examiners, and enables consistent evaluation standards to be maintained.

[1102] A "candidate" is a person who takes part in an examination and is assessed on a particular task or skill.

[1103] "Device" means an electronic device used by a candidate for the test, which is equipped with the capability to record video and audio.

[1104] "Recording" refers to the act of using a terminal to record video and audio data during the test.

[1105] "Central Server" means a central data processing device that receives, analyzes, and evaluates recorded video and audio data.

[1106] "Uploading" is the act of sending recorded data from a terminal to a central server.

[1107] "Storage" refers to a storage device for temporarily or permanently storing received data.

[1108] "Analysis" is the process of processing the recorded data and extracting the information necessary for evaluation.

[1109] "Facial expression recognition" is a technology that detects the facial expressions of test takers from video data and analyzes their emotional state.

[1110] "Gesture recognition" is a technology that detects the examinee's hand movements and posture from video data and analyzes their meaning.

[1111] "Voice tone" refers to characteristics of voice data such as pitch, strength, and intonation.

[1112] "Speech content" is text information of words and phrases contained in the voice data.

[1113] "Evaluation criteria" refers to the standards and scales established to fairly and objectively evaluate test takers' performance.

[1114] A "score" is a numerical value or evaluation index generated from the analysis results based on the evaluation criteria.

[1115] An "administrator terminal" is an electronic device used by the test administrator or examiner to check evaluation results and adjust evaluation criteria.

[1116] "Display" refers to the act of showing the evaluation results and analysis details on the screen of the administrator's terminal.

[1117] A "database" is a system that systematically stores and manages data such as analysis results and evaluation criteria.

[1118] "Application" is the act of reflecting the newly adjusted evaluation criteria in the next evaluation.

[1119] This is an evaluation system consisting of multiple subsystems for the purpose of fairly and objectively evaluating examinee performance. The system consists of three main components: examinee terminals, a central server, and an administrator terminal.

[1120] Examinee's device

[1121] The user uses a device with a dedicated recording application installed. This device is equipped with a camera and microphone, which are used to record video and audio during the test. The user starts the recording application and presses the "Start role-playing" button to start recording, and the "Stop recording" button to end recording. Once recording is complete, the device automatically uploads the recorded video and audio data to a central server.

[1122] Central Server

[1123] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs various analyses. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. It also analyzes the audio data for tone and speech content. These analyses are performed using dedicated software such as facial expression recognition libraries and audio analysis tools. Based on the analysis results, a score is generated according to pre-set evaluation criteria. The generated score and analysis details are stored in a database.

[1124] Administrator terminal

[1125] The administrator uses the administrator terminal to retrieve the assessment results from the central server. The management screen of the administrator terminal displays each examinee's score, analysis details, and related information. In addition, the administrator can adjust the assessment criteria. When new assessment criteria are entered, they are saved in the database and applied to the next assessment.

[1126] Specific examples

[1127] Let's take the example of a candidate taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes the speaking style and content of the speech from the audio data. Based on these analyses, a score is generated according to the evaluation criteria. The test administrator can view this score using the administrator device and adjust the evaluation criteria as necessary.

[1128] Examples of prompt statements

[1129] Example prompt 1: "Analyze the test-taker's video to recognize facial expressions and gestures. Then analyze the audio data to analyze speaking style and content to generate a score based on the assessment criteria."

[1130] Example prompt 2: "Analyze the uploaded test data and generate a score based on the test taker's facial expressions, gestures, tone, and speech content. Then store the generated score along with the analysis results in a database."

[1131] In this way, the system of the present invention allows exam evaluation to be carried out fairly and objectively, reduces the burden on examiners, and maintains consistent evaluation standards.

[1132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1133] Step 1:

[1134] The user launches a dedicated recording application on their device. As input, a launch command is given, which starts preparation of the device's camera and microphone. As output, the initial screen of the application is displayed, ready for recording.

[1135] Step 2:

[1136] The user presses the "Start Role-Playing" button. The user's operation instruction (pressing the button) is given as input, and the device activates the camera and microphone to start recording video and audio. The output is that the recorded video and audio data is saved in the device's local storage in real time.

[1137] Step 3:

[1138] The user presses the "Stop Recording" button. The user's operation instruction is given as input, and the device stops recording. The output is a state in which the stopped video and audio data has been completely saved to local storage.

[1139] Step 4:

[1140] The device makes an API call to the central server to send the recorded data. The input is the recorded video and audio data that the device uploads to the central server. The output is the video and audio data received by the server.

[1141] Step 5:

[1142] The server receives data sent from the device and stores it in storage. The input is the uploaded video and audio data, which the server stores in temporary storage. The output is the data stored in storage.

[1143] Step 6:

[1144] The server extracts frames from the video data and performs facial expression and gesture recognition. The input is video data stored in storage, which the server passes to the facial expression recognition library for analysis. The output is analyzed facial expression and gesture data.

[1145] Step 7:

[1146] The server analyzes the voice data and performs voice tone analysis and speech content analysis. The input is the voice data stored in the storage, and the server passes it to the voice analysis tool. The output is the analyzed voice tone and speech content data.

[1147] Step 8:

[1148] The server generates a score based on the analysis results according to the set evaluation criteria. The input is the analyzed facial expression, gesture, voice tone, and speech content data, which the server inputs into the evaluation algorithm. The output is the generated score.

[1149] Step 9:

[1150] The server stores the generated score in a database. The input is the generated score, which the server records in the database. The output is the score stored in the database.

[1151] Step 10:

[1152] The administrator uses the administrator terminal to obtain the evaluation results from the central server. The input is a request to obtain the evaluation results, which the administrator terminal sends to the server. The output is to display the evaluation results sent from the server on the screen of the administrator terminal.

[1153] Step 11:

[1154] The administrator can adjust the evaluation criteria using the administrator terminal. The input is the adjusted new evaluation criteria, which the administrator terminal sends to the server. The output is that the new evaluation criteria are stored in the database, ready to be applied at the next evaluation.

[1155] (Application example 1)

[1156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1157] It is difficult to fairly and objectively evaluate the customer service skills of employees in brick-and-mortar stores. Conventional methods rely on the subjective judgment of the evaluator, which can lead to inconsistency and a lack of fairness. Furthermore, employee training is time-consuming and costly, and providing effective feedback is difficult. The present invention solves these problems and provides a system for efficiently and objectively evaluating and training customer service skills in brick-and-mortar stores.

[1158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1159] In this invention, the server includes: means for recording video and audio using a terminal used by the examinee; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the examinee based on the analysis results; and means for the server to provide the evaluation results to a manager's terminal. In a customer service skill evaluation system used in a brick-and-mortar store, means for recording the actions and speech of employees; means for uploading the recorded data to the central server and analyzing it to generate evaluation results; and means for a manager to check the evaluation results and adjust the evaluation criteria. This makes it possible to evaluate the customer service skills of employees fairly and objectively and provide effective feedback.

[1160] "Examination candidate terminal" refers to a device used by an examination candidate to record video and audio, and includes a camera, microphone, etc.

[1161] The "central server" is a computer system that receives and analyzes data uploaded from test taker terminals.

[1162] An "administrator terminal" is a device that displays the evaluation results and adjusts the evaluation criteria as necessary, and is generally a PC or tablet terminal.

[1163] "Facial expression recognition" is the process of analyzing facial features extracted from frames of video data to identify emotions and expressions.

[1164] "Gesture recognition" is the process of analyzing physical movements extracted from frames of video data and identifying their meaning.

[1165] "Voice tone analysis" is the process of analyzing audio data to evaluate the pitch, intensity, rhythm, etc. of a speaker's voice.

[1166] "Speech analysis" is the process of analyzing the content of audio data to identify the words being spoken and their content.

[1167] The "customer service skills evaluation system" is a system for fairly and objectively evaluating the customer service skills of employees in physical stores based on video and audio data.

[1168] "Recording means" refers to a method or device for recording video and audio using a camera or microphone.

[1169] "Uploading means" refers to a method or device for transmitting recorded video and audio data over a network to a central server.

[1170] "Analysis means" refers to software or algorithms that the central server uses to analyze the uploaded video and audio data.

[1171] An "assessment tool" is a method or algorithm used to evaluate candidates or employees based on the results of the analysis.

[1172] "Evaluation criteria" are the indicators or standards used to evaluate candidates or employees and are used for scoring purposes.

[1173] The present invention provides a system for fairly and objectively evaluating the customer service skills of employees in a brick-and-mortar store. Specific embodiments of this system will be described in detail below.

[1174] 1. Exam-taker's device

[1175] The devices used by employees include smartphones, smart glasses, or head-mounted displays. These devices are equipped with cameras and microphones, allowing them to record video and audio. Recording begins by launching a dedicated application installed on the device and pressing the "Start Role-Playing" button. When the customer service simulation is finished, the user presses the "Stop Recording" button, and the recorded video and audio data is automatically uploaded to the central server.

[1176] 2. Central Server

[1177] The central server receives the data uploaded from the test taker's device and stores it in storage. The server then performs several processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression and gesture recognition. This is done using the OpenCV library. The recorded audio data is also analyzed for tone and speech content. This is done using the PyAudio library. Based on the results of these analyses, a score is generated according to the evaluation criteria, and the evaluation results are stored in a database.

[1178] 3. Administrator terminal

[1179] The manager terminal is used by the store manager. The manager terminal retrieves the evaluation results from the central server and displays them on a dedicated management screen. On this management screen, each employee's score, analysis details, and related information can be viewed. In addition, the manager can adjust the evaluation criteria through this screen. New evaluation criteria are entered from the manager terminal, and the central server saves them in the database and applies them the next time an employee is evaluated.

[1180] 4. Specific Examples

[1181] For example, consider a case where a newly hired employee at a store is undergoing customer service training. The employee wears smart glasses and performs customer service role-playing with a colleague playing the role of a customer. The recorded video and audio are uploaded to a server, where facial expressions, speech patterns, and gestures are analyzed in detail. A manager can view the results on a manager's device, evaluate the employee's customer service skills, and provide necessary feedback. This system enables consistent and fair evaluations, leading to improved employee skills.

[1182] Generative AI model prompt example

[1183] Analyze the following video and audio data to evaluate an employee's customer service skills. Generate scores for each of the following items: facial expressions, gestures, tone of voice, and content of speech, and output an overall score:

[1184] <Video data URL>

[1185] <Audio data URL>

[1186] In this way, the system of the present invention makes it possible to fairly and objectively evaluate test takers' performance and provide effective feedback, thereby enabling employees to quickly and effectively improve their customer service skills.

[1187] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1188] Step 1:

[1189] Employees will use smart devices to conduct customer service simulations.

[1190] Specifically, employees wear a smartphone, smart glasses, or a head-mounted display and perform a customer service role-play. To do so, they launch the recording application and press the "Start role-playing" button to begin recording video and audio.

[1191] Input: Video and audio data collected from the camera and microphone built into your smart device.

[1192] Output: Recorded video and audio data.

[1193] Step 2:

[1194] After the device finishes recording, it uploads the video and audio data to a central server.

[1195] Specifically, when the user presses the "Stop Recording" button, the device automatically transmits the recorded data to a central server via secure communication, ensuring the safety of the data.

[1196] Input: Video and audio data after recording has finished.

[1197] Output: Data is uploaded to a central server.

[1198] Step 3:

[1199] The server receives the uploaded data and stores it in storage.

[1200] Specifically, the server converts the received data into an appropriate format and stores it in a database or file system.

[1201] Input: Uploaded video and audio data.

[1202] Output: Securely stored video and audio data.

[1203] Step 4:

[1204] The server extracts frames from the video data and performs facial expression and gesture recognition.

[1205] Specifically, it uses the OpenCV library to split the video data into frames, analyzes the facial features in each frame, and identifies the employee's facial expressions and gestures.

[1206] Input: Stored video data.

[1207] Output: Analysis results of facial expression and gesture recognition.

[1208] Step 5:

[1209] The server analyzes the voice data and performs voice tone analysis and speech content analysis.

[1210] Specifically, it uses the PyAudio library to extract features from audio data, evaluate the pitch, intensity, and rhythm of the speaker's voice, and identify what is being said.

[1211] Input: Stored audio data.

[1212] Output: Voice tone analysis and speech content analysis.

[1213] Step 6:

[1214] The server generates a score based on the analysis results and in accordance with the evaluation criteria.

[1215] Specifically, based on the analysis results, pre-set evaluation criteria are applied to generate a numerical score for the employee's customer service skills.

[1216] Input: Results of facial expression recognition, gesture recognition, voice tone analysis, and speech content analysis.

[1217] Output: Scored evaluation results.

[1218] Step 7:

[1219] The server provides the evaluation results to the administrator terminal.

[1220] Specifically, the generated scores and analysis details will be displayed on a dedicated management screen so that administrators can check them.

[1221] Input: Scored assessment results.

[1222] Output: Evaluation results displayed on the administrator's terminal.

[1223] Step 8:

[1224] The administrator adjusts the evaluation criteria and saves the new criteria on the server.

[1225] Specifically, the administrator inputs the evaluation criteria using the management terminal, and the server stores them in the database and applies them the next time the evaluation is conducted.

[1226] Input: New evaluation criteria from the administrator terminal.

[1227] Output: The new saved criterion.

[1228] The above are the specific processing steps for fairly and objectively evaluating the customer service skills of the examinee.

[1229] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1230] The present invention provides a system for fairly and objectively evaluating test takers in role-playing tests such as customer service skills tests and certification tests. Furthermore, the present invention enables more precise evaluations by incorporating an emotion engine that analyzes the user's emotions. A specific embodiment of this system is described below.

[1231] Subsystem 1: Examinee's terminal

[1232] The device is used by the test taker to record video and audio during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[1233] Subsystem 2: Central Server

[1234] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[1235] Subsystem 3: Administrator terminal

[1236] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[1237] Specific examples

[1238] Let's take an example where a candidate is taking a customer service etiquette test. First, the candidate starts the test using a device and records their performance using a camera and microphone. Once the recording is complete, the device uploads the data to a central server. The server analyzes the candidate's facial expressions and gestures from the received video data, and analyzes their speaking style and content from the audio data. In addition, an emotion engine analyzes the candidate's emotions using the video and audio data. Based on this analysis, a score is generated according to the evaluation criteria, and additional evaluation criteria are applied based on the emotion analysis results. The test administrator can view the score using the administrator device and adjust the evaluation criteria as necessary.

[1239] In this way, the present invention provides a system that evaluates test takers' performance fairly and objectively, eliminating human subjective bias. Furthermore, the use of an emotion engine enables advanced evaluation that takes test takers' emotional states into account. The system of the present invention reduces the workload of examiners and ensures consistent evaluation.

[1240] The processing flow will be explained below.

[1241] Step 1:

[1242] The user starts the dedicated application on the examinee's device and presses the "Start role-playing" button, which starts recording.

[1243] Step 2:

[1244] The device's camera begins recording the user's actions as video.

[1245] Step 3:

[1246] The device's microphone begins recording the user's voice.

[1247] Step 4:

[1248] When the user finishes role-playing, they press the "Stop Recording" button, which stops the video and audio recording.

[1249] Step 5:

[1250] The device stores the recorded video and audio data in local storage.

[1251] Step 6:

[1252] The device uploads the stored video and audio data to a central server via the Internet.

[1253] Step 7:

[1254] The server receives the uploaded video and audio data and stores it in storage.

[1255] Step 8:

[1256] The server analyzes the video data and extracts frames, which are then used to perform facial expression recognition and analyze the user's facial features.

[1257] Step 9:

[1258] The server performs gesture recognition from video data and analyzes the user's hand and body movements.

[1259] Step 10:

[1260] The server analyzes the voice data, analyzes the voice tone and the content of the speech, specifically extracting voice waveform features and performing language analysis.

[1261] Step 11:

[1262] The server uses an emotion engine to perform emotion analysis based on the video and audio data. It identifies the user's emotional state by analyzing facial expressions from the video data and tone of voice and content of speech from the audio data.

[1263] Step 12:

[1264] The server generates a score based on the analysis results according to a preset evaluation criterion, which also incorporates the emotion analysis results.

[1265] Step 13:

[1266] The server stores the generated scores and analysis details in a database.

[1267] Step 14:

[1268] The administrator terminal obtains the evaluation results from the central server and displays them on the management screen.

[1269] Step 15:

[1270] The administrator uses the administrator terminal to check each test taker's score, analysis details, and emotion analysis results.

[1271] Step 16:

[1272] The administrator adjusts the evaluation criteria as needed, and the adjusted evaluation criteria are sent to the server, which stores them in a database and applies them the next time the evaluation is conducted.

[1273] Example 2

[1274] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1275] Traditional role-playing tests often rely on the examiner's subjective evaluation, making it difficult to achieve fair and objective evaluation. Furthermore, because it is difficult to analyze emotions, it is not possible to perform sophisticated evaluations that take into account the examinee's emotional state. Furthermore, there are challenges in that it is difficult for test administrators to consistently apply evaluation criteria.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring audio and video by an information terminal used by the examinee, means for transferring the audio and video acquired by the information terminal to a central data processing device, means for the central data processing device to receive and analyze the transferred data, means for the central data processing device to evaluate the examinee based on the analysis results, and means for the central data processing device to provide the evaluation results to an administrator's information terminal. This makes it possible to evaluate the examinee's performance fairly and objectively, eliminate the subjective bias of the examiner, and perform consistent evaluations.

[1277] "Information terminal" means a device used by a test taker during the test to obtain audio and video.

[1278] The "central data processing device" is a device that receives, analyzes, and evaluates audio and video data transferred from the examinee's information terminal.

[1279] The "analysis means" is a function that allows the central data processing unit to process audio and video data and perform facial expression recognition, action recognition, tone analysis, and analysis of audio content.

[1280] The "evaluation means" is a function that allows the central data processing unit to evaluate and score the performance of the examinee based on the analysis results.

[1281] The "administrator's information terminal" is a device that displays the evaluation results and analysis results provided by the central data processing unit and enables the evaluation criteria to be changed.

[1282] "Acquisition means" refers to the function that enables an information terminal to acquire audio and video.

[1283] The "transfer means" is a function for transmitting data acquired by the information terminal to the central data processing unit.

[1284] The "memory device" is a storage device for saving information such as analysis results and evaluation criteria in the central data processing unit.

[1285] The present invention provides a system for fairly and objectively evaluating examinees in role-playing tests such as customer service skills tests and certification tests. Furthermore, by combining an emotion engine, more precise evaluations can be performed. The system of the present invention is primarily composed of an information terminal used by the examinee, a central data processing unit, and an information terminal for the administrator.

[1286] Exam taker information terminal

[1287] The terminal is a device used by the test taker to capture audio and video during the test. It includes a camera, a microphone, and a dedicated recording application. The user starts the recording application on the terminal and starts recording by pressing the "Start Role-Playing" button. The application uses the camera to record the test taker's actions and the microphone to record audio. When the test is finished, the user presses the "Stop Recording" button, and the terminal automatically uploads the recorded data to the central data processing unit.

[1288] Central Data Processing Unit

[1289] The server receives the data uploaded from the test taker's device and stores it in storage. Next, frames are extracted from the video data and facial expression and movement recognition is performed. This is done using video analysis software such as OpenCV. Furthermore, the audio data is analyzed for tone and audio content using audio analysis software such as LIBROSA. Furthermore, an emotion engine (e.g., an emotion recognition AI module) is used to analyze the test taker's emotional state. Based on the results of these analyses, a score is generated according to the evaluation criteria and stored in a database.

[1290] Administrator information terminal

[1291] The administrator terminal is a device used by the test administrator and examiners. The administrator terminal obtains the evaluation results from the central data processing device and displays them on the management screen. On this management screen, each examinee's score, detailed analysis results, and related information can be viewed. The administrator can adjust the evaluation criteria as needed. New evaluation criteria are entered from the administrator terminal and sent to the central data processing device. The server saves the new evaluation criteria in storage and applies them to the next evaluation.

[1292] Specific examples

[1293] For example, consider a test taker taking a customer service etiquette test. First, the test taker uses a device to begin the test and records their performance using a camera and microphone. The user introduces themselves and role-plays interacting with virtual customers. Once the recording is complete, the device uploads the data to a central data processing unit. The server analyzes the uploaded video data and recognizes the test taker's facial expressions and movements. It also analyzes the audio data to evaluate speaking style, tone, and content. It also uses an emotion engine to analyze the test taker's emotions and evaluates factors such as stress and friendliness. A score is generated based on these results, and the test taker is assigned a grade according to the evaluation criteria.

[1294] Prompt Sentence Examples

[1295] "This system records video and audio of test takers taking customer service etiquette tests and uploads them to a central server. The server analyzes facial expressions and gestures from the video and the content of speech from the audio. It also uses an emotion engine to analyze the test takers' emotions. Finally, it generates a score based on the evaluation criteria and displays it on an administrator's terminal."

[1296] In this way, test takers' performance can be evaluated fairly and objectively, eliminating subjective bias and ensuring consistent evaluation. Using an emotion engine enables advanced evaluation that takes test takers' emotional states into account, reducing the workload of test administrators and enabling more precise evaluation.

[1297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1298] Program processing steps

[1299] Step 1:

[1300] The user launches a dedicated recording application on their information terminal. When the application launches, it checks the camera and microphone readiness and displays that they are available for use. This allows real-time data of the examinee's movements and speech to be obtained as input.

[1301] Step 2:

[1302] The user presses the "Start Role-Playing" button to begin recording. The device uses the camera to record the examinee's actions as video and the microphone to record audio. Once video and audio data begins to be recorded, it is continuously saved. The input is real-time video and audio, and the output is recorded video and audio files.

[1303] Step 3:

[1304] When the test is finished, the user presses the "End Recording" button. The terminal saves the recorded data as a file and prepares it for upload to the central data processing unit. The input is the recorded video and audio files, and the output is a data package for uploading.

[1305] Step 4:

[1306] The terminals upload the recorded video and audio data to the central data processing unit. The data is transferred over the network and reaches the central data processing unit. The input is the data package for uploading, and the output is the data received by the central data processing unit.

[1307] Step 5:

[1308] The server receives the video and audio data uploaded from the terminal and stores it in storage. The input is the received data package, and the output is a data file stored in the storage.

[1309] Step 6:

[1310] The server extracts frames from the video data and performs facial expression and movement recognition. This uses dedicated video analysis software (e.g., OpenCV). The input is a saved video file, and the output is facial expression and movement data.

[1311] Step 7:

[1312] The server analyzes the audio data and performs a timbre and audio content analysis using audio analysis software (e.g., LIBROSA). The input is a saved audio file, and the output is timbre and audio content data.

[1313] Step 8:

[1314] The server uses an emotion engine to perform emotion analysis based on video and audio data. The inputs are facial expression data, movement data, tone data, and audio content data, and the output is emotion data.

[1315] Step 9:

[1316] The server generates a score based on the analysis results and evaluation criteria. The evaluation criteria are set in advance and points are calculated based on these. The inputs are facial expression data, movement data, tone data, voice content data, and emotion data, and the output is a final score and an evaluation report.

[1317] Step 10:

[1318] The server stores the generated scores and analysis details in a database. The input is the final score and the evaluation report, and the output is the evaluation results stored in the database.

[1319] Step 11:

[1320] The administrator terminal obtains the evaluation results from the central data processing unit and displays them on the management screen. The administrator can check each examinee's score and detailed analysis results. The input is the evaluation results from the database, and the output is the information displayed on the management screen.

[1321] Step 12:

[1322] The administrator adjusts the evaluation criteria as needed. The administrator terminal sends the new evaluation criteria to the central data processing unit. The input is the new evaluation criteria entered by the administrator, and the output is the updated evaluation criteria stored in the central data processing unit.

[1323] The above processing steps realize a system for evaluating the performance of test takers fairly and objectively.

[1324] (Application example 2)

[1325] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1326] Current customer service skill evaluation systems often rely on subjective evaluations, making it difficult to provide fair and objective evaluations. Furthermore, they lack real-time analysis of customer service behavior and detailed emotion analysis using emotion engines, making it cumbersome to provide feedback on specific areas for improvement to staff. This makes it difficult to efficiently support staff growth.

[1327] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording video and audio using a terminal used by the test taker; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the test taker based on the analysis results; means for the server to provide the evaluation results to an administrator terminal; means for the store staff to record video and audio in real time using smart glasses while serving customers; and means for transmitting the recorded data to the server and analyzing facial expressions, gestures, and tone of voice using an emotion engine. This enables objective and fair evaluation of customer service skills and also enables efficient real-time feedback of specific areas for improvement to be provided to staff.

[1328] "Device used by the examinee" refers to the device used by the examinee during the examination to record video and audio.

[1329] "Central Server" is a computer system established to receive, analyze, and evaluate data uploaded from the terminals.

[1330] The "emotion engine" is software that analyzes facial expressions, gestures, and tone of voice from video and audio data to determine the user's emotions.

[1331] "Smart glasses" are glasses-type mobile devices worn by users that can record video and audio in real time.

[1332] "Facial expression recognition" is a technology that analyzes facial expressions from recorded video data.

[1333] "Gesture recognition" is a technology that analyzes hand and body movements from recorded video data.

[1334] "Voice tone analysis" is a technology that analyzes speaking style and tone of voice from recorded audio data.

[1335] "Evaluation criteria" are the criteria for evaluating test takers' skills and performance based on the recorded and analyzed data.

[1336] An "administrator terminal" is a device used by an administrator to check the evaluation results and adjust the evaluation criteria.

[1337] "Real-time feedback" is the process of providing staff with evaluation and analysis results obtained during customer service on the spot.

[1338] The present invention provides a system for fairly and objectively evaluating test takers in customer service skills and qualification examinations. Specific embodiments of this system are described below.

[1339] Subsystem 1: Examinee's terminal

[1340] The device is used by the test taker to record video and audio during the test. It includes a camera, microphone, and a dedicated recording application. The user starts the recording application on the device and presses the "Start Role-Playing" button to begin recording. The application uses the camera to record video of the test taker's actions and the microphone to record audio. When recording is complete, the user presses the "Stop Recording" button, and the device automatically uploads the recorded video and audio to a central server.

[1341] Subsystem 2: Central Server

[1342] The server receives the video and audio data uploaded from the device and stores it in storage. The server then performs multiple processes to analyze this data. Specifically, it extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. For the recorded audio data, it also analyzes the voice tone, speech content, and emotion using an emotion engine. Based on the results of these analyses, it generates a score according to pre-set evaluation criteria. The server then stores the generated score and analysis details in a database.

[1343] Subsystem 3: Administrator terminal

[1344] The administrator terminal is used by the test administrator and examiners. The administrator terminal retrieves the evaluation results from the central server and displays them on the management screen. On this management screen, each examinee's score, analysis details, and related information can be viewed. In addition, the administrator can adjust the evaluation criteria. New evaluation criteria are entered from the administrator terminal, and the server saves them in the database and applies them to the next evaluation.

[1345] Application example: Smart customer service evaluation assistant in brick-and-mortar stores

[1346] This system can be applied as an application installed on smart glasses to support the evaluation of customer service staff in brick-and-mortar stores. The system records video and audio in real time while the staff are serving customers and transmits the data to a server.

[1347] Specifically, staff wear smart glasses and record video and audio data while serving customers. The server receives this data and uses an emotion engine to analyze facial expressions, gestures, and tone of voice. Based on the analysis results, staff are then scored for their customer service skills. Managers can check the scores and analysis results and provide specific feedback to staff on areas for improvement in real time.

[1348] Hardware and Software

[1349] Hardware

[1350] Smart glasses (e.g. Google Glass)

[1351] Exam taker's device (camera, microphone)

[1352] software

[1353] Recording application (video and audio recording)

[1354] Emotion engine (analysis of facial expressions, gestures, and voice tone)

[1355] Server system (receiving, analyzing, and evaluating data)

[1356] Examples of prompt statements

[1357] "For customer service training evaluation, please provide specific program code for analyzing recorded audio and video data and recognizing facial expressions, gestures, and tone of voice."

[1358] In this way, we provide a system that uses smart glasses to enable real-time customer service evaluation and efficiently support staff in improving their skills.

[1359] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1360] Step 1:

[1361] The user puts on the smart glasses and begins serving customers. While serving customers, the smart glasses' camera and microphone record video and audio in real time. The recorded video is saved frame by frame, and the audio is continuously saved in a buffer as data. The input is the actual customer service scene, and the output is the recorded video and audio data.

[1362] Step 2:

[1363] The device's recording application starts recording by pressing the "Start Role-Playing" button. The camera captures video as successive frames, and the microphone captures audio data. When recording is complete, the user presses the "Stop Recording" button to stop recording. The input is the user's actions, and the output is the stopped video and audio files.

[1364] Step 3:

[1365] The device uploads the recorded video and audio data to a central server. After recording is complete, the device application sends the data to the server using the FTP protocol or HTTP POST request. The input is the recorded video and audio files, and the output is the data stored on the server.

[1366] Step 4:

[1367] The server receives the uploaded video and audio data and saves it in storage. The server receives the HTTP request and saves the video and audio files in the specified directory. The input is the data sent from the device, and the output is the data saved in storage.

[1368] Step 5:

[1369] The server extracts frames from the video data and performs facial expression recognition, gesture recognition, and emotion analysis using an emotion engine. The emotion engine uses, for example, Google Cloud Vision API or Microsoft Azure Face API. The input is each frame of the video data, and the output is recognized facial expression data, gesture data, and emotion data.

[1370] Step 6:

[1371] The server analyzes the recorded voice data, analyzing voice tone, speech content, and emotion using an emotion engine. Voice analysis uses Google Cloud Speech-to-Text API and IBM Watson Speech to Text, among others. The input is voice data, and the output is voice tone data, speech content data, and emotion data.

[1372] Step 7:

[1373] The server evaluates the examinee's skills and performance based on the analysis results and generates a score according to pre-set evaluation criteria. The evaluation criteria are set by the system administrator. The inputs are facial expression data, gesture data, voice tone data, and speech content data, and the output is an evaluation score.

[1374] Step 8:

[1375] The server stores the generated scores and analysis details in a database. The inputs are the assessment scores and analysis results, and the output is the assessment data stored in the database.

[1376] Step 9:

[1377] The administrator terminal obtains the evaluation results from the server and displays them on the management screen. The administrator checks the evaluation results and analysis details and adjusts the evaluation criteria as necessary. The input is the evaluation data, and the output is the evaluation results displayed on the administrator terminal.

[1378] Step 10:

[1379] The server receives the evaluation criteria data adjusted by the administrator terminal and stores it in the database. The new criteria are applied at the next evaluation. The input is the new evaluation criteria data, and the output is the evaluation criteria stored in the database.

[1380] The above steps enable real-time customer service skill evaluation using smart glasses, providing objective and fair evaluations.

[1381] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1382] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1383] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1384] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1385] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1386] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1387] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1388] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1389] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1390] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1391] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1392] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1393] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1394] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1395] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1396] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1397] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1398] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1399] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1400] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1401] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1402] The following is further disclosed regarding the above embodiment.

[1403] (Claim 1)

[1404] A means for recording video and audio using a device used by the test taker;

[1405] means for the terminal to upload the recorded video and audio to a central server;

[1406] means for the server to receive and analyze the uploaded data;

[1407] means for the server to evaluate the examinee based on the analysis results;

[1408] a means for the server to provide the evaluation result to an administrator terminal;

[1409] A system including:

[1410] (Claim 2)

[1411] A means for extracting frames from the recorded video data and performing facial expression recognition and gesture recognition;

[1412] means for analyzing the recorded voice data and performing voice tone analysis and speech content analysis;

[1413] means for scoring the analysis results based on the evaluation criteria;

[1414] The system of claim 1 further comprising:

[1415] (Claim 3)

[1416] a means for displaying the evaluation results on the administrator terminal and enabling adjustment of evaluation criteria;

[1417] means for the server to store and apply the adjusted evaluation criteria in a database;

[1418] The system of claim 1 further comprising:

[1419] "Example 1"

[1420] (Claim 1)

[1421] A means for recording video and audio using a device used by the test taker;

[1422] means for the terminal to upload the recorded video and audio to a central server;

[1423] a means for the server to receive the uploaded data and store it in a storage;

[1424] means for the server to analyze the data and perform facial expression recognition and gesture recognition;

[1425] means for the server to analyze the data and perform voice tone analysis and speech content analysis;

[1426] means for the server to evaluate the examinee based on the analysis results and generate a score;

[1427] a means for the server to provide the evaluation result to an administrator terminal;

[1428] A system including:

[1429] (Claim 2)

[1430] means for extracting frames from the recorded video data and performing facial expression recognition and gesture recognition;

[1431] means for analyzing the recorded voice data and performing voice tone analysis and speech content analysis;

[1432] means for scoring the analysis results based on the evaluation criteria;

[1433] The system of claim 1 further comprising:

[1434] (Claim 3)

[1435] a means for displaying the evaluation results on the administrator terminal and enabling adjustment of evaluation criteria;

[1436] means for the server to store and apply the adjusted evaluation criteria in a database;

[1437] The system of claim 1 further comprising:

[1438] "Application Example 1"

[1439] (Claim 1)

[1440] A means for recording video and audio using a device used by the test taker;

[1441] means for the terminal to upload the recorded video and audio to a central server;

[1442] means for the server to receive and analyze the uploaded data;

[1443] means for the server to evaluate the examinee based on the analysis results;

[1444] a means for the server to provide the evaluation result to an administrator terminal;

[1445] A customer service skill evaluation system used in a real store, comprising: a means for recording the actions and speech of employees;

[1446] means for uploading the recorded data to a central server for analysis and generation of evaluation results;

[1447] A means for managers to review evaluation results and adjust evaluation criteria;

[1448] A system including:

[1449] (Claim 2)

[1450] A means for extracting frames from the recorded video data and performing facial expression recognition and gesture recognition;

[1451] means for analyzing the recorded voice data and performing voice tone analysis and speech content analysis;

[1452] means for scoring the analysis results based on the evaluation criteria;

[1453] A means for a manager to provide necessary feedback to employees based on the evaluation results;

[1454] The system of claim 1 further comprising:

[1455] (Claim 3)

[1456] a means for displaying the evaluation results on the administrator terminal and enabling adjustment of evaluation criteria;

[1457] means for the server to store and apply the adjusted evaluation criteria in a database;

[1458] The system of claim 1 further comprising:

[1459] "Example 2: Combining Emotion Engines"

[1460] (Claim 1)

[1461] a means for acquiring audio and video by an information terminal used by the examinee;

[1462] means for transferring the audio and video captured by the information terminal to a central data processing unit;

[1463] means for said central data processing unit to receive and analyze said transferred data;

[1464] means for the central data processing unit to evaluate the examinee based on the analysis results;

[1465] a means for the central data processing device to provide the evaluation results to an information terminal for an administrator;

[1466] A system including:

[1467] (Claim 2)

[1468] means for extracting still images from the acquired video data and performing facial expression recognition and action recognition;

[1469] means for analyzing the acquired voice data and performing a timbre analysis and a voice content analysis;

[1470] means for scoring the analysis results based on evaluation criteria;

[1471] The system of claim 1 further comprising:

[1472] (Claim 3)

[1473] a means for the manager's information terminal to display the evaluation results and enable the evaluation criteria to be changed;

[1474] means for the central data processing unit to store and apply the modified evaluation criteria in a storage device;

[1475] The system of claim 1 further comprising:

[1476] "Application example 2 when combining emotion engines"

[1477] (Claim 1)

[1478] A means for recording video and audio using a device used by the test taker;

[1479] means for the terminal to upload the recorded video and audio to a central server;

[1480] means for the server to receive and analyze the uploaded data;

[1481] means for the server to evaluate the examinee based on the analysis results;

[1482] a means for the server to provide the evaluation result to an administrator terminal;

[1483] A means for real-time video and audio recording by store staff using smart glasses while serving customers;

[1484] means for transmitting the recorded data to a server and analyzing facial expressions, gestures, and tone of voice using an emotion engine;

[1485] A system including:

[1486] (Claim 2)

[1487] A means for extracting frames from the recorded video data and performing facial expression recognition and gesture recognition;

[1488] means for analyzing the recorded voice data and performing voice tone analysis and speech content analysis;

[1489] means for scoring the analysis results based on the evaluation criteria;

[1490] A means to automatically evaluate staff customer service skills based on the analysis results, and

[1491] The system of claim 1 further comprising:

[1492] (Claim 3)

[1493] a means for displaying the evaluation results on the administrator terminal and enabling adjustment of evaluation criteria;

[1494] means for the server to store and apply the adjusted evaluation criteria in a database;

[1495] A means for managers to provide real-time feedback on staff evaluations,

[1496] The system of claim 1 further comprising: [Explanation of symbols]

[1497] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for recording video and audio using a device used by the test taker; means for the terminal to upload the recorded video and audio to a central server; means for the server to receive and analyze the uploaded data; means for the server to evaluate the examinee based on the analysis results; a means for the server to provide the evaluation result to an administrator terminal; A system including:

2. A means for extracting frames from the recorded video data and performing facial expression recognition and gesture recognition; means for analyzing the recorded voice data and performing voice tone analysis and speech content analysis; means for scoring the analysis results based on the evaluation criteria; The system of claim 1 further comprising:

3. a means for displaying the evaluation results on the administrator terminal and enabling adjustment of evaluation criteria; means for the server to store and apply the adjusted evaluation criteria in a database; The system of claim 1 further comprising:

Images