System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring, from a server, a question list generated by a generation AI based on a specific interview scenario selected from a plurality of interview scenarios selected by a user; means for inquiring of the user by voice about the question list acquired from the server by the terminal; means for converting, by the terminal, contents answered by the user by voice into text data by using a voice recognition system; means for transmitting, to the server, an evaluation result and a feedback generated by the generation AI and evaluating the contents by the user, and transmitting the evaluation result and the feedback to the terminal; and means for providing, to the user, the evaluation result and the feedback as voice or display.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, face-to-face practice has been the mainstream approach to preparing for interviews and presentations, but this has significant time and location constraints, making it difficult to achieve high-quality practice. Furthermore, opportunities for growth have been limited because self-evaluation and feedback are not provided appropriately. The objective of this invention is to provide a system that allows interview and presentation practice without being restricted by location or time, while receiving effective feedback. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means: a system including: means for acquiring from a server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user; means for a terminal to ask the user verbally the questions from the list acquired from the server; means for the terminal to convert the user's verbal responses into text data using a voice recognition system; means for sending the user's responses to the server, evaluating the responses using the generation AI, generating evaluation results and feedback, and sending them to the terminal; and means for the terminal to provide the evaluation results and feedback to the user as voice or on a display. This allows users to practice interviews and presentations while receiving high-quality feedback, regardless of location or time.

[0006] A "user" is an individual who uses the system to practice interviews or presentations.

[0007] A "terminal" is an electronic device used by a user, such as a computer, smartphone, or tablet.

[0008] The "server" is a computer system that uses a generation AI to generate a list of questions and receives and evaluates user answers.

[0009] An "interview scenario" is a practice scenario that includes a list of questions tailored to a specific job or situation.

[0010] "Generative AI" is an artificial intelligence system that generates new questions based on past example questions and best practices, and evaluates the user's answers.

[0011] A "question list" is a series of questions created by generative AI based on an interview scenario.

[0012] A "voice recognition system" is a system that converts the content of a user's voice response into text data.

[0013] "Text data" refers to data obtained by converting a user's voice response into text information.

[0014] "Feedback" is information that the server uses to evaluate the user's answers using generated AI and provides the user with areas for improvement and strengths. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that provides practical interview practice via a terminal using a question list generated by a generation AI based on a specific interview scenario selected by the user.

[0037] Program processing

[0038] 1. The user logs in to the system using a terminal and begins the interview practice.

[0039] The user selects the desired scenario from multiple interview scenarios on the terminal screen.

[0040] 2. The terminal sends the user's selection information to the server.

[0041] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0042] 3. The server uses the generation AI to generate a list of questions corresponding to the scenario and sends it to the device.

[0043] The list of questions corresponds to the interview scenario selected by the user (e.g., technical interview or sales interview).

[0044] 4. The device receives the list of questions and uses the ChatBot function to ask the questions to the user via voice.

[0045] The questions are presented to the user one by one in turn by voice, prompting the user to respond.

[0046] 5. The user answers the question by voice.

[0047] The user's response is converted into text data by a voice recognition system via the device's microphone.

[0048] 6. The device sends the converted text data to the server.

[0049] The server has the generation AI evaluate the received text data.

[0050] 7. The server uses the generated AI to evaluate the user's answers.

[0051] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity of the answer.

[0052] 8. The server generates the evaluation results and feedback and sends them to the device.

[0053] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[0054] 9. The device displays or audibly conveys the evaluation results and feedback to the user.

[0055] Users can improve their answers through feedback and incorporate it into their next practice.

[0056] Specific examples

[0057] (Example) For a technical interview:

[0058] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[0059] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[0060] 3. The user responds verbally, "I'm a software engineer..."

[0061] 4. The device converts the voice into text data and sends it to the server.

[0062] 5. The server receives the answer and uses the generation AI to generate an evaluation result such as "The answer was clear and consistent."

[0063] 6. The device will provide the user with audible feedback on the evaluation results and proceed to the next question.

[0064] In this way, the system allows users to practice interviews at high quality, regardless of location or time. This system is expected to increase users' confidence through practical practice and improve their success rate in interviews.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[0068] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[0069] Step 2:

[0070] The terminal transmits the user's scenario selection information to the server.

[0071] The request sent to the server includes the user ID and the selected scenario type.

[0072] Step 3:

[0073] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0074] The generative AI generates a list of questions based on past question examples and best practices.

[0075] Step 4:

[0076] The server transmits the generated question list to the terminal.

[0077] The list of questions was tailored to the selected interview scenario.

[0078] Step 5:

[0079] The device receives a list of questions and asks the user aloud using the ChatBot function.

[0080] The questions are presented to the user aloud one by one in turn.

[0081] Step 6:

[0082] The user answers the questions by voice.

[0083] The user's voice response is recorded through the device's microphone.

[0084] Step 7:

[0085] The terminal uses a voice recognition system to convert the user's voice response into text data.

[0086] The converted text data is sent to a server for evaluation.

[0087] Step 8:

[0088] The server receives the user's text data and evaluates the content using a generation AI.

[0089] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[0090] Step 9:

[0091] The server generates evaluation results and feedback and sends them to the device.

[0092] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[0093] Step 10:

[0094] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[0095] Users can review the feedback and use it to improve their answers.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Conventional interview practice systems have the problem that it is difficult for users to practice in conditions similar to a real interview environment, and in particular, they do not provide sufficient individual feedback or evaluation, which limits the user's skill improvement.In addition, existing systems often do not provide comprehensive evaluation based on evaluation criteria such as consistency, concreteness, and clarity of answers required in a real interview.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user verbally the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a speech recognition system, means for evaluating the acquired answers with the generative AI model using prompt sentences created based on the generative AI model, means for transmitting the user's answers to the server, evaluating the contents of the answers with the generative AI model, generating evaluation results and feedback and transmitting them to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to engage in more practical and high-quality interview practice and receive detailed feedback.

[0101] A "user" is an individual or organization that uses the interview practice system to practice for an interview.

[0102] A "terminal" is an electronic device used by a user, such as a computer, smartphone, or tablet, that functions as an interface with the system.

[0103] The "server" is a central computer system that works in conjunction with the terminal to generate a list of questions using a generative AI model, evaluate answers, and provide feedback.

[0104] An "interview scenario" refers to multiple interview situations or themes that a user can select, such as different situations such as a technical interview or a sales interview.

[0105] A "generative AI model" is an artificial intelligence algorithm that generates a list of questions and evaluates answers based on user input data, and primarily uses natural language processing technology.

[0106] A "question list" is a set of questions generated by a generative AI model based on an interview scenario, and is presented to the user.

[0107] A "voice recognition system" is a technology that converts a user's verbal response from voice data into text data.

[0108] A "prompt" is a document that defines the basic instructions and conditions that a generative AI model uses to conduct a question list or evaluation.

[0109] "Feedback" is information that includes the evaluation results of the user's answers and points for improvement, and is provided to the user.

[0110] This invention is a system that uses a generative AI model to generate a list of questions based on a specific interview scenario selected by the user, and provides practical interview practice via a terminal. This system is composed of a server, a terminal, a generative AI model, a speech recognition system, and speech synthesis technology. Detailed embodiments of this system are described below.

[0111] The user logs in to the terminal and starts the interview practice. The user selects the desired scenario from multiple interview scenarios displayed on the terminal. For example, "technical interview" or "sales interview" can be selected. The selected scenario information is sent from the terminal to the server.

[0112] The server sends a request to the generative AI model to generate a list of questions based on the selected scenario. Here, a generative AI model that is strong in natural language processing (e.g., GPT-4) is used. For example, a prompt such as "Please generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10" is sent to the generative AI model. The generative AI model generates a list of questions based on this prompt.

[0113] The server sends the generated question list to the terminal, which then uses voice synthesis technology (e.g., Google Text-to-Speech) to ask the user aloud the questions. The questions are presented to the user one by one in order, prompting them to answer.

[0114] The user answers questions by voice. The user's answers are captured through the device's microphone and converted into text data by a voice recognition system (e.g., Google Cloud Speech-to-Text).

[0115] The device sends the converted text data to a server, which passes the text data to a generative AI model to evaluate the answer. For example, the generative AI model might be provided with a prompt such as, "Please rate the answer based on criteria such as consistency, specificity, and clarity." The generative AI model then analyzes the answer based on the evaluation criteria and generates an evaluation result and feedback.

[0116] The server sends the evaluation results and feedback to the device. The feedback includes the user's strengths and areas for improvement, as well as advice for the next practice. The device then tells the user the evaluation results and feedback by voice or displays them on the screen. This allows the user to improve their answers and reflect them in their next practice.

[0117] As a concrete example, the flow of a technical interview is shown below.

[0118] 1. The user logs in to the terminal and selects "Technical Interview Practice."

[0119] 2. The server sends a prompt to the generative AI model: "Generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10."

[0120] 3. The generative AI model generates a list of questions, which the server sends to the device.

[0121] 4. The device will ask the user aloud, "Please introduce yourself."

[0122] 5. The user responds verbally, "I'm a software engineer..."

[0123] 6. The device converts the voice into text data and sends it to the server.

[0124] 7. The server evaluates the answer using the generative AI model and generates a rating result that says, "The answer was clear and consistent."

[0125] 8. The device provides the user with audio feedback on the evaluation results.

[0126] In this way, the system allows users to conduct high-quality interview practice regardless of location or time.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1:

[0129] A user logs in to a terminal

[0130] Input: User ID and password.

[0131] How it works: The user enters their ID and password on the device's login screen and presses the "Login" button.

[0132] Data processing: The device sends the user ID and password to the server, which checks them against the authentication information stored in the database.

[0133] Output: If authentication is successful, the server generates an authentication token and sends it to the device, which then displays the dashboard screen to the user.

[0134] Step 2:

[0135] The user selects an interview scenario

[0136] Input: Interview scenario (e.g. technical interview, sales interview).

[0137] How it works: The user clicks on the desired scenario from the list displayed on the device screen.

[0138] Data processing: The device sends the selected scenario information to the server.

[0139] Output: The server receives the scenario information.

[0140] Step 3:

[0141] The server sends a question list generation request to the generative AI model.

[0142] Input: Selected scenario information.

[0143] Operation: Based on the selected scenario information, the server sends a request to the generative AI model to generate a list of questions.

[0144] Data processing: The generative AI model generates a list of questions based on the prompt.

[0145] Example prompt: "Generate a list of 10 questions for a technical interview at an intermediate level."

[0146] Output: The generative AI model generates a list of questions and returns it to the server.

[0147] Step 4:

[0148] The server sends the question list to the terminal.

[0149] Input: The list of questions received from the generative AI model.

[0150] Operation: The server converts the generated question list into JSON format and sends it to the terminal.

[0151] Data processing: Format conversion and data transmission on the server.

[0152] Output: The terminal receives the list of questions.

[0153] Step 5:

[0154] The device asks the user a list of questions by voice

[0155] Input: Questionnaire.

[0156] How it works: The device uses speech synthesis technology to read the list of questions to the user.

[0157] Text-to-speech example: Google Text-to-Speech.

[0158] Data processing: Text to speech conversion.

[0159] Output: Presents the question to the user by voice, prompting the user for an answer.

[0160] Step 6:

[0161] The user answers the question

[0162] Input: User speaks the answer.

[0163] How it works: The user speaks the answer to a question into the terminal.

[0164] Data processing: The user's voice is recorded through the device's microphone.

[0165] Output: Recorded audio data.

[0166] Step 7:

[0167] The device converts the voice data into text data.

[0168] Input: Recorded audio data.

[0169] How it works: The device uses a voice recognition system to convert voice data into text data.

[0170] Speech recognition example: Google Cloud Speech-to-Text.

[0171] Data processing: Converting audio data into text data.

[0172] Output: The converted text data.

[0173] Step 8:

[0174] The device sends text data to the server.

[0175] Input: Text data.

[0176] Operation: The terminal converts the converted text data into JSON format and sends it to the server.

[0177] Data processing: format conversion and data transmission.

[0178] Output: The server receives the text data.

[0179] Step 9:

[0180] The server evaluates the answers using a generative AI model

[0181] Input: The converted text data.

[0182] How it works: The server passes the received text data to the generative AI model for evaluation.

[0183] Data processing: The generative AI model analyzes the answers based on the prompt sentence and evaluation criteria.

[0184] Sample prompt: "Please rate your answers based on criteria such as coherence, specificity, and clarity."

[0185] Output: The generative AI model generates evaluation results and feedback and returns them to the server.

[0186] Step 10:

[0187] The server sends the evaluation results and feedback to the device.

[0188] Input: Assessment results and feedback.

[0189] Operation: The server formats the evaluation information obtained from the generative AI model and sends it to the device.

[0190] Data processing: Data formatting and transmission.

[0191] Output: The device receives the evaluation results and feedback.

[0192] Step 11:

[0193] The device provides the evaluation results and feedback to the user.

[0194] Input: Assessment results and feedback.

[0195] What it does: The device will read out the assessment results and feedback aloud or display them on the screen.

[0196] Data processing: Using voice synthesis technology.

[0197] Output: User receives evaluation results and feedback.

[0198] (Application example 1)

[0199] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0200] Conventional interview practice systems have difficulty in providing situations similar to those of a real interview, limiting the effectiveness of user practice. Furthermore, in the actual hiring process, there are issues with the heavy burden on interviewers and the inconsistency of evaluations. Given these circumstances, there is a need for a system that can improve the quality of interview practice and ensure efficient and fair hiring of staff for brick-and-mortar stores.

[0201] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0202] In this invention, the server includes means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user the list of questions acquired from the server by voice or text, means for the terminal to convert the user's voice or text answers into text data using a voice recognition system, means for sending the user's answers to the server, evaluating the answers by the generation AI, generating evaluation results and feedback and sending them to the terminal, and means for the terminal to provide the user with the evaluation results and feedback by voice or display. This makes it possible to provide practical interview practice and enables fair and efficient evaluation even in staff recruitment at physical stores.

[0203] "User" means any individual or organization that uses the System.

[0204] An "interview scenario" is a set of questions designed to fit a specific interview situation.

[0205] "Generative AI" refers to a system that uses artificial intelligence technology to solve problems and generate questions.

[0206] A "question list" refers to a collection of questions provided for a user to answer.

[0207] "Server" refers to the central processing unit responsible for data processing and storage within a system.

[0208] "Terminal" refers to a device (smartphone, tablet, computer, etc.) that is directly operated by a user.

[0209] A "voice recognition system" refers to a technology that converts voice data into text data.

[0210] "Text data" refers to character information converted by a voice recognition system.

[0211] "Evaluation result" refers to the result of the evaluation made by the generation AI on the user's answer.

[0212] "Feedback" refers to advice or suggestions provided to the user based on the evaluation results.

[0213] "Audio or display" refers to the means for providing output information to the user, and includes both audio output and visual display.

[0214] This invention is a system that provides practical interview practice using a question list generated by a generation AI based on an interview scenario selected by the user. A specific embodiment of this system is described below.

[0215] First, a user must log in to a terminal to start the interview practice. Next, the user selects a specific scenario from multiple interview scenarios (e.g., technical interview, sales interview, etc.) through the terminal. This selection information is sent from the terminal to the server.

[0216] The server sends a request to a generative AI to generate a list of questions based on the interview scenario selected by the user, using a generative AI model such as OpenAI's GPT-3, and then sends the generated list of questions to the device.

[0217] The device asks the user questions in order based on the received question list. These questions are provided in the form of voice or text. When the user responds by voice, the device's voice recognition system converts the voice data into text data. Here, Python's speech_recognition library, etc., is used.

[0218] The user's answer, converted into text data, is sent back to the server, where the generation AI evaluates the answer based on criteria such as consistency, specificity, and clarity. The server then uses the generation AI to generate evaluation results and feedback, which are then sent to the device.

[0219] The device provides the user with the evaluation results and feedback, which can be presented as audio or visual, allowing the user to improve their answers and use them for their next practice.

[0220] Specific examples

[0221] For example, in a cashier interview scenario, the following questions might be generated:

[0222] Example prompt sentence:

[0223] Generate interview questions based on the interview scenario "Cashier."

[0224] The resulting questionnaire:

[0225] 1. Please introduce yourself

[0226] 2. Tell us about your experience in treating all customers with kindness.

[0227] 3. How do you manage stress during busy periods?

[0228] In this way, users can conduct high-quality interview practice regardless of location or time. By applying this to staff recruitment in brick-and-mortar stores, it is expected to realize a fair and efficient interview process and reduce the burden on interviewers.

[0229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0230] Step 1:

[0231] A user logs in to a terminal. The user launches an application on the terminal, enters the required credentials on the login screen, and performs the login operation, completing the authentication. The input is the user's credentials, and the output is an authentication success message.

[0232] Step 2:

[0233] The user selects the desired interview scenario. The user makes a selection from multiple interview scenarios (e.g., technical interview, sales interview, cashier, etc.) displayed on the terminal screen. The input is the user's selection information, and the output is data indicating the selected scenario.

[0234] Step 3:

[0235] The terminal sends the selected scenario information to the server. The terminal converts the user-selected scenario information into a data packet and sends a request to the server. The input is the selected scenario data, and the output is a request to the server.

[0236] Step 4:

[0237] The server requests the generative AI model to generate a list of questions. The server sends a prompt to the generative AI model (e.g., GPT-3) based on the selected scenario, causing it to create a list of questions. The input is the prompt, and the output is the generated list of questions.

[0238] Step 5:

[0239] The server sends the generated question list to the terminal. The server returns the question list obtained from the generative AI model to the terminal. The input is the generated question list, and the output is the data sent to the terminal.

[0240] Step 6:

[0241] The terminal asks the user a list of questions by voice or text. Based on the received list of questions, the terminal uses a speech synthesis system (e.g., gTTS) to present the questions to the user by voice or displays them on the screen as text. The input is the question list data, and the output is voice or text display.

[0242] Step 7:

[0243] The user answers the question. The user answers the question displayed on the terminal by voice or text. The input is the user's voice response to the question, and the output is voice data.

[0244] Step 8:

[0245] The device converts the user's voice into text data. The device's voice recognition system (e.g., speech_recognition) converts the user's voice response into text data. The input is voice data, and the output is text data.

[0246] Step 9:

[0247] The terminal sends the converted text data to the server. The terminal sends the converted text data to the server and makes an evaluation request. The input is the text data, and the output is a request to the server.

[0248] Step 10:

[0249] The server requests the generative AI model to evaluate the answer. The server provides the received text data to the generative AI model and asks for an evaluation. The input is the text data, and the output is the evaluation result and feedback.

[0250] Step 11:

[0251] The server generates evaluation results and feedback and sends them to the terminal. The server evaluates based on evaluation criteria (e.g., consistency, specificity, clarity), creates feedback, and sends it to the terminal. The input is the evaluation result data, and the output is the feedback data.

[0252] Step 12:

[0253] The terminal provides the evaluation results and feedback to the user. The terminal presents the received evaluation results and feedback by voice or displays them on a screen. The input is the feedback data, and the output is voice or text display.

[0254] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0255] This invention is a system that combines an emotion engine and provides practical interview practice via a terminal using a question list generated by a generative AI based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotions into the evaluation.

[0256] Program processing

[0257] 1. The user logs in to the system using a terminal and begins the interview practice.

[0258] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[0259] 2. The terminal sends the user's selection information to the server.

[0260] The request sent to the server includes the user ID and the selected scenario type.

[0261] 3. The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0262] The generative AI generates a list of questions based on past question examples and best practices.

[0263] 4. The server sends the generated question list to the terminal.

[0264] The list of questions was tailored to the selected interview scenario.

[0265] 5. The device receives the list of questions and asks the user voice questions using the ChatBot function.

[0266] The questions are presented to the user aloud one by one in turn.

[0267] 6. The user answers the question by voice.

[0268] The user's voice response is recorded through the device's microphone.

[0269] 7. The device uses a voice recognition system to convert the user's voice response into text data.

[0270] The converted text data is sent to a server for evaluation.

[0271] 8. The server receives the user's text data and evaluates the content using a generation AI.

[0272] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[0273] 9. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0274] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[0275] 10. The emotion engine sends the recognized emotion data to the device, and the device then sends the data to the server.

[0276] 11. The server uses generated AI to evaluate the user's answers, including emotional data.

[0277] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[0278] 12. The server generates the evaluation results and feedback and sends them to the device.

[0279] Feedback includes user strengths, areas for improvement, and emotional advice.

[0280] 13. The device displays or audibly conveys the evaluation results and feedback to the user.

[0281] Through feedback, users can improve their answers and emotional state and reflect this in their next practice.

[0282] Specific examples

[0283] (Example) For a technical interview:

[0284] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[0285] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[0286] 3. The user responds verbally, "I'm a software engineer..."

[0287] 4. The device converts the voice into text data and sends it to the server.

[0288] 5. The server receives the answer and evaluates it using the generation AI.

[0289] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0290] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[0291] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[0292] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[0293] In this way, users can not only practice for interviews at high quality regardless of time or place, but also understand their emotional state and make comprehensive improvements, including their mental state.

[0294] The processing flow will be explained below.

[0295] Step 1:

[0296] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[0297] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[0298] Step 2:

[0299] The terminal transmits the user's scenario selection information to the server.

[0300] The request sent to the server includes the user ID and the selected scenario type.

[0301] Step 3:

[0302] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0303] The generative AI generates a list of questions based on past question examples and best practices.

[0304] Step 4:

[0305] The server transmits the generated question list to the terminal.

[0306] The list of questions was tailored to the selected interview scenario.

[0307] Step 5:

[0308] The device receives a list of questions and asks the user aloud using the ChatBot function.

[0309] The questions are presented to the user aloud one by one.

[0310] Step 6:

[0311] The user answers the questions by voice.

[0312] The user's voice response is recorded through the device's microphone.

[0313] Step 7:

[0314] The terminal uses a voice recognition system to convert the user's voice response into text data.

[0315] The converted text data is sent to the server.

[0316] Step 8:

[0317] The server receives the user's text data and evaluates the content using a generation AI.

[0318] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[0319] Step 9:

[0320] The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0321] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[0322] Step 10:

[0323] The emotion engine transmits the recognized emotion data to the terminal, which then transmits the data to the server.

[0324] Emotional data is used as additional information for evaluation.

[0325] Step 11:

[0326] The server uses generated AI to evaluate the user's answers, including emotional data.

[0327] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[0328] Step 12:

[0329] The server generates evaluation results and feedback and sends them to the device.

[0330] Feedback includes specific recommendations based on user strengths, areas for improvement, and feelings.

[0331] Step 13:

[0332] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[0333] Users can review the feedback and use it to improve their answers and emotional state.

[0334] Specific examples

[0335] (Example) For a technical interview:

[0336] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[0337] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[0338] 3. The user responds verbally, "I'm a software engineer..."

[0339] 4. The device converts the voice into text data and sends it to the server.

[0340] 5. The server receives the answer and evaluates it using the generation AI.

[0341] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0342] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[0343] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[0344] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[0345] In this way, users can not only practice for interviews at high quality without being tied down to a specific place or time, but also understand their emotional state and make comprehensive improvements, including their own emotional state.

[0346] Example 2

[0347] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0348] Conventional interview practice systems evaluate only the content of the user's voice responses, making it difficult to provide comprehensive feedback that takes into account the user's emotional state and mental state. This leaves users with no way to effectively overcome feelings of tension and anxiety, making it difficult for them to maximize their performance in an actual interview.

[0349] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0350] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for the terminal to ask the user verbally the questions from the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a voice recognition system, means for transmitting the user's answers to the server, evaluating the content of the answers using the generative AI model, and analyzing the voice data using an emotion engine to recognize the user's emotional state, means for generating and transmitting evaluation results and feedback to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to receive multifaceted evaluation and feedback that takes into account not only the content of their answers but also their emotional state.

[0351] A "generative AI model" is an artificial intelligence that uses machine learning technology to automatically generate a list of questions and answer evaluations based on user input.

[0352] A "terminal" is an electronic device used by a user to practice for an interview, such as a smartphone, tablet, or PC.

[0353] The "server" is a central computer system that processes data for the entire system, generates questionnaires, and executes evaluations.

[0354] An "interview scenario" is a mock interview situation consisting of a number of predetermined questions and situations.

[0355] A "question list" is a set of interview questions generated by a generative AI model.

[0356] A "voice recognition system" is a technology that analyzes a user's voice and converts it into text data.

[0357] "Emotion engine" is an artificial intelligence technology that analyzes a user's voice data to recognize their emotional state.

[0358] The "evaluation results" are a collection of feedback obtained by the generative AI model by analyzing the user's responses and emotional state.

[0359] "Feedback" is information about evaluations and advice regarding the user's interview performance.

[0360] This invention is an interview practice system that combines a generative AI model and an emotion engine to provide practical interview practice based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotional state into the evaluation.

[0361] First, the terminal used by the user is an electronic device such as a smartphone, tablet, or PC, which is used to access and log in to the system. After logging in, the user selects the desired scenario from multiple interview scenarios (e.g., technical interview, sales interview, behavioral interview, etc.). The specific operation of the system is explained below.

[0362] The server requests the generative AI model to generate a list of questions based on the interview scenario selected by the user. For example, if a technical interview scenario is selected, the prompt message is "Please generate a list of questions for the technical interview." The generative AI model references past example questions and best practices to generate a list of questions appropriate to the selected scenario. The generated list of questions is sent from the server to the device in a format such as JSON.

[0363] Based on the received question list, the device uses its ChatBot function to present questions to the user by voice. For example, questions such as "Please introduce yourself" are asked in sequence. When the user responds by voice, the response is recorded through the device's microphone. The device then uses a voice recognition system to convert the voice response into text data in real time and transmits this data to the server.

[0364] The server receives the user's text data and sends the recorded voice data to the emotion engine, which analyzes the voice data and recognizes the user's emotional state (e.g., nervousness, anxiety, confidence, etc.). The recognized emotion data is then sent to the server.

[0365] The server uses a generative AI model to comprehensively evaluate the text data and emotional data, taking into account the consistency, specificity, and clarity of the answers, as well as the user's emotional state. For example, the server may generate an evaluation such as, "Your answers were consistent and specific, but you seemed a little nervous." Once the evaluation results and feedback are generated, they are sent to the device, which then provides them to the user in audio or display format. The user can use this feedback to improve their answers and emotional state and reflect them in their next practice.

[0366] As a concrete example, if a user selects a technical interview scenario, the following prompt is sent to the generative AI model: "Generate a list of questions for a technical interview." A list of questions is generated, and the question "Please introduce yourself" is presented. The user responds, "I'm a software engineer..." and the audio data is recorded and converted into text data. At the same time, the emotion engine generates a recognition result such as "The user is nervous." The generative AI model evaluates the content of the response and the user's emotional state, and generates feedback such as "The response was clear and consistent, but you seem a little nervous."

[0367] As described above, this system combines voice recognition technology and emotion analysis technology to provide users with comprehensive interview practice and feedback.

[0368] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0369] Step 1:

[0370] A user logs into the system using a terminal and selects an interview scenario.

[0371] Specifically, the user enters their ID and password on the login screen and clicks the login button. After logging in, a dashboard is displayed where the user selects the desired interview scenario from among "technical interview" and "sales interview" scenarios. The input is the user ID and password, and the output is information about the selected scenario.

[0372] Step 2:

[0373] The terminal transmits the user's selection information to the server.

[0374] The input is the user ID and the selected scenario information, which the device sends to the server as a request. The specific operation is to send an HTTP request to the server. The output is the request data sent to the server.

[0375] Step 3:

[0376] The server sends a request to the generative AI model to generate a list of questions.

[0377] The input is the user's selection information, and the server sends a prompt to the generative AI model based on this data. For example, the prompt might say, "Please generate a list of questions for a technical interview." Specifically, an API request is sent to the generative AI model. The output is the generated list of questions.

[0378] Step 4:

[0379] The server transmits the generated question list to the terminal.

[0380] The input is a list of questions returned by the generative AI model, and the server sends this list to the device. Specifically, an HTTP response is sent to the device. The output is the list of questions that arrives at the device.

[0381] Step 5:

[0382] The device receives the list of questions and asks the user voice questions using the ChatBot function.

[0383] The input is a list of questions received from the server, and the ChatBot function presents the questions to the user one by one by voice. Specifically, speech synthesis software is used. The output is the voice question presented to the user.

[0384] Step 6:

[0385] The user answers the questions by voice.

[0386] The input is a voice question from the terminal, and the user answers by voice. Specifically, the terminal's microphone records the user's voice. The output is the recorded voice data.

[0387] Step 7:

[0388] The terminal uses a voice recognition system to convert the voice response into text data and transmits it to the server.

[0389] The input is the user's voice data, which the device converts into text data using a voice recognition system. Specifically, voice recognition software is used. The output is the converted text data, which is sent to the server.

[0390] Step 8:

[0391] The server receives the user's text and voice data and analyzes it using a generative AI model and emotion engine.

[0392] The input is the user's text data and voice data, and the server has the generative AI model analyze the text data and the emotion engine analyze the voice data. Specifically, an API request is sent to the generative AI model and the emotion engine. The output is the evaluation result and emotion data.

[0393] Step 9:

[0394] The server generates evaluation results and feedback and sends them to the device.

[0395] The input is the analyzed evaluation results and emotion data, and the server synthesizes these to generate feedback. Specifically, the generative AI model automatically generates the feedback. The output is feedback data, which is sent to the device.

[0396] Step 10:

[0397] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[0398] The input is the feedback data received from the server, which the device displays or speaks to the user. Specific actions include presenting the feedback on a display or through voice synthesis. The output is the feedback provided to the user.

[0399] (Application example 2)

[0400] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] Conventional interview practice systems focus on evaluating the content of the user's answers, but are unable to incorporate the user's emotional state into the evaluation. This has led to the problem of being unable to properly manage emotions such as tension and anxiety that users may experience during an actual interview. Furthermore, in the customer service industry, the emotional state of store clerks significantly affects the quality of customer service, so an evaluation system that includes emotion recognition has been in demand.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0403] In this invention, the server includes: means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user; means for the terminal to ask the user aloud questions from the list of questions acquired from the server; means for the terminal to convert the user's spoken answers into text data using a voice recognition system; means for transmitting the user's answers to the server, evaluating the content of the answers using the generation AI, generating evaluation results and feedback and transmitting them to the terminal; means for the terminal to transmit recorded voice data to an emotion recognition engine and analyze the user's emotions; and means for transmitting emotional data recognized by the emotion recognition engine to the server and for the generation AI to evaluate the user's answers, including the emotional data. This makes it possible to incorporate not only the content of the user's answers but also their emotional state at the time of answering into the evaluation. This allows users to understand their own emotional state and engage in more practical and comprehensive interview practice and customer service training.

[0404] "User" refers to an individual who uses the system to practice interviews and customer service training.

[0405] An "interview scenario" is a practice scenario that assumes a specific interview situation, and includes different scenarios such as a technical interview and a sales interview.

[0406] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate question lists and assessments.

[0407] A "question list" is a collection of multiple questions generated by generative AI and is constructed based on an interview scenario.

[0408] The "server" refers to a central processing unit that processes user selection information and response data, and generates evaluations and feedback by working with a generative AI and emotion recognition engine.

[0409] A "terminal" is a device operated by a user, including a smartphone or a head-mounted display (HMD).

[0410] "Speech recognition system" refers to technology that converts a user's voice response into text data.

[0411] "Text data" refers to sentence data converted from speech by a speech recognition system.

[0412] An "emotion recognition engine" refers to a system that analyzes a user's emotional state (e.g., tension, anxiety, confidence, etc.) from their voice data.

[0413] "Feedback" refers to advice or evaluation comments provided as a result of evaluating the content of a user's answers or emotional state.

[0414] This invention provides a system for interview practice and customer service training. The system includes a user, a terminal, and a server, and operates and processes as follows.

[0415] Program Overview

[0416] Users log in to the system using a device such as a smartphone or head-mounted display (HMD). The user selects the desired scenario from multiple interview scenarios or customer service scenarios (for example, technical interview, sales interview, product explanation, complaint handling, etc.). This selection information is sent from the device to the server. The server uses this information to have the generation AI generate a list of questions. The generated list of questions is sent to the device, which then asks the user questions by voice. The user then answers by voice, and the answers are converted into text data using the device's voice recognition system, and this text data is sent to the server.

[0417] Data processing and calculation

[0418] The server uses a generative AI to evaluate the text data and analyze it based on criteria such as consistency, specificity, and clarity of the content. The device also sends the recorded voice data to an emotion recognition engine to analyze the user's emotions. This emotion data is also sent to the server. The server then uses the generative AI to perform a comprehensive evaluation, including the emotion data, and generates and sends the evaluation results and feedback to the device. The device then provides this feedback to the user as voice or display.

[0419] Hardware and software used

[0420] Hardware: Smartphone, Head-Mounted Display (HMD)

[0421] Software: Speech recognition system (e.g., Google Speech Recognition), emotion recognition engine (custom or external API), generative AI (for questionnaire generation and evaluation)

[0422] Specific examples

[0423] For example, consider the case where a store clerk launches a customer service training app and selects the "Handling Complaints" scenario. The device retrieves the question from the server, "Please tell us how you would respond if you received a complaint from a customer," and asks the clerk aloud. When the clerk responds, "First, I'll listen to the customer's story to the end...," the content is converted into text data by a voice recognition system. The server receives this text data and evaluates the content and emotional state using a generative AI and an emotion recognition engine. The server then generates feedback such as, "Your opinion is clear, but you seem a little nervous," and the device presents it to the clerk.

[0424] Prompt Sentence Examples

[0425] How do you explain your products to customers?

[0426] What is the most important point when dealing with a complaint?

[0427] By using these concrete examples and prompts, users can understand their own emotional state while conducting practical and comprehensive interview practice and customer service training.

[0428] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0429] Step 1:

[0430] A user logs in to the system from a smartphone or head-mounted display (HMD). The input is the user's authentication information, and the output is authentication that the login was successful. Specifically, a login form is displayed on the terminal screen, and the user enters their ID and password.

[0431] Step 2:

[0432] The user selects the desired scenario from multiple interview or customer service scenarios. The input is the information about the scenario selected by the user, and the output is the ID of the selected scenario. Specifically, a list of scenarios is displayed on the screen, and the user taps to select the scenario they are interested in.

[0433] Step 3:

[0434] The device sends the user's selection information to the server. The input is the selected scenario ID and user ID, and the output is a receipt confirmation from the server. Specifically, the device sends data to the server using an HTTP request.

[0435] Step 4:

[0436] The server makes the generation AI generate a list of questions based on the selected scenario ID. The input is the scenario ID, and the output is the generated list of questions. Specifically, the server calls the generation AI model and inputs a prompt to generate questions based on the scenario.

[0437] Step 5:

[0438] The server sends the generated question list to the device. The input is the question list generated by the generation AI, and the output is the received question list. Specifically, the server sends the question list to the device in an HTTP response.

[0439] Step 6:

[0440] The device receives the list of questions and asks the user aloud. The input is the received list of questions, and the output is the spoken questions to the user. Specifically, the device uses text-to-speech synthesis (TTS) to play back the questions aloud.

[0441] Step 7:

[0442] The user answers the question by voice. The input is the voice question, and the output is the user's voice response. Specifically, the user responds through the device's microphone.

[0443] Step 8:

[0444] The device uses a voice recognition system to convert the user's voice response into text data. The input is the user's voice response, and the output is text data. Specifically, the device uses a voice recognition library to convert voice to text.

[0445] Step 9:

[0446] The device sends text data to the server. The input is text data converted from speech, and the output is confirmation of transmission to the server. Specifically, the device sends the text data to the server via an HTTP request.

[0447] Step 10:

[0448] The server evaluates the text data using generative AI. The input is text data, and the output is the evaluation result. Specifically, the server feeds the text data to the generative AI model and evaluates the consistency, concreteness, and clarity of the content.

[0449] Step 11:

[0450] The device sends recorded voice data to the emotion recognition engine to analyze the user's emotions. The input is recorded voice data and the output is emotion data. Specifically, the device sends the voice data to the emotion recognition engine via an HTTP request.

[0451] Step 12:

[0452] The emotion recognition engine sends the recognized emotion data to the server. The input is the emotion data, and the output is a confirmation of transmission to the server. Specifically, the emotion recognition engine sends the analysis results to the server via an HTTP request.

[0453] Step 13:

[0454] The server uses a generation AI to evaluate the user's answers, including emotional data. The input is text data and emotional data, and the output is a comprehensive evaluation result. Specifically, the server inputs emotional data into the generation AI and performs a comprehensive evaluation.

[0455] Step 14:

[0456] The server generates evaluation results and feedback and sends them to the terminal. The input is the overall evaluation result, and the output is the evaluation result and feedback. Specifically, the server organizes the evaluation results, generates feedback for the user, and sends it to the terminal.

[0457] Step 15:

[0458] The device provides the evaluation results and feedback to the user by voice or display. The input is the evaluation results and feedback, and the output is provided to the user. Specifically, the device uses text-to-speech (TTS) to play back the feedback by voice or displays it on the screen.

[0459] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0460] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0461] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0462] [Second embodiment]

[0463] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0464] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0465] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0466] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0467] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0468] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0469] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0470] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0471] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0472] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0473] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0474] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0475] The present invention is a system that provides practical interview practice via a terminal using a question list generated by a generation AI based on a specific interview scenario selected by the user.

[0476] Program processing

[0477] 1. The user logs in to the system using a terminal and begins the interview practice.

[0478] The user selects the desired scenario from multiple interview scenarios on the terminal screen.

[0479] 2. The terminal sends the user's selection information to the server.

[0480] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0481] 3. The server uses the generation AI to generate a list of questions corresponding to the scenario and sends it to the device.

[0482] The list of questions corresponds to the interview scenario selected by the user (e.g., technical interview or sales interview).

[0483] 4. The device receives the list of questions and uses the ChatBot function to ask the questions to the user via voice.

[0484] The questions are presented to the user one by one in turn by voice, prompting the user to respond.

[0485] 5. The user answers the question by voice.

[0486] The user's response is converted into text data by a voice recognition system via the device's microphone.

[0487] 6. The device sends the converted text data to the server.

[0488] The server has the generation AI evaluate the received text data.

[0489] 7. The server uses the generated AI to evaluate the user's answers.

[0490] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity of the answer.

[0491] 8. The server generates the evaluation results and feedback and sends them to the device.

[0492] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[0493] 9. The device displays or audibly conveys the evaluation results and feedback to the user.

[0494] Users can improve their answers through feedback and incorporate it into their next practice.

[0495] Specific examples

[0496] (Example) For a technical interview:

[0497] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[0498] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[0499] 3. The user responds verbally, "I'm a software engineer..."

[0500] 4. The device converts the voice into text data and sends it to the server.

[0501] 5. The server receives the answer and uses the generation AI to generate an evaluation result such as "The answer was clear and consistent."

[0502] 6. The device will provide the user with audible feedback on the evaluation results and proceed to the next question.

[0503] In this way, the system allows users to practice interviews at high quality, regardless of location or time. This system is expected to increase users' confidence through practical practice and improve their success rate in interviews.

[0504] The processing flow will be explained below.

[0505] Step 1:

[0506] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[0507] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[0508] Step 2:

[0509] The terminal transmits the user's scenario selection information to the server.

[0510] The request sent to the server includes the user ID and the selected scenario type.

[0511] Step 3:

[0512] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0513] The generative AI generates a list of questions based on past question examples and best practices.

[0514] Step 4:

[0515] The server transmits the generated question list to the terminal.

[0516] The list of questions was tailored to the selected interview scenario.

[0517] Step 5:

[0518] The device receives a list of questions and asks the user aloud using the ChatBot function.

[0519] The questions are presented to the user aloud one by one in turn.

[0520] Step 6:

[0521] The user answers the questions by voice.

[0522] The user's voice response is recorded through the device's microphone.

[0523] Step 7:

[0524] The terminal uses a voice recognition system to convert the user's voice response into text data.

[0525] The converted text data is sent to a server for evaluation.

[0526] Step 8:

[0527] The server receives the user's text data and evaluates the content using a generation AI.

[0528] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[0529] Step 9:

[0530] The server generates evaluation results and feedback and sends them to the device.

[0531] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[0532] Step 10:

[0533] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[0534] Users can review the feedback and use it to improve their answers.

[0535] Example 1

[0536] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0537] Conventional interview practice systems have the problem that it is difficult for users to practice in conditions similar to a real interview environment, and in particular, they do not provide sufficient individual feedback or evaluation, which limits the user's skill improvement.In addition, existing systems often do not provide comprehensive evaluation based on evaluation criteria such as consistency, concreteness, and clarity of answers required in a real interview.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0539] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user verbally the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a speech recognition system, means for evaluating the acquired answers with the generative AI model using prompt sentences created based on the generative AI model, means for transmitting the user's answers to the server, evaluating the contents of the answers with the generative AI model, generating evaluation results and feedback and transmitting them to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to engage in more practical and high-quality interview practice and receive detailed feedback.

[0540] A "user" is an individual or organization that uses the interview practice system to practice for an interview.

[0541] A "terminal" is an electronic device used by a user, such as a computer, smartphone, or tablet, that functions as an interface with the system.

[0542] The "server" is a central computer system that works in conjunction with the terminal to generate a list of questions using a generative AI model, evaluate answers, and provide feedback.

[0543] An "interview scenario" refers to multiple interview situations or themes that a user can select, such as different situations such as a technical interview or a sales interview.

[0544] A "generative AI model" is an artificial intelligence algorithm that generates a list of questions and evaluates answers based on user input data, and primarily uses natural language processing technology.

[0545] A "question list" is a set of questions generated by a generative AI model based on an interview scenario, and is presented to the user.

[0546] A "voice recognition system" is a technology that converts a user's verbal response from voice data into text data.

[0547] A "prompt" is a document that defines the basic instructions and conditions that a generative AI model uses to conduct a question list or evaluation.

[0548] "Feedback" is information that includes the evaluation results of the user's answers and points for improvement, and is provided to the user.

[0549] This invention is a system that uses a generative AI model to generate a list of questions based on a specific interview scenario selected by the user, and provides practical interview practice via a terminal. This system is composed of a server, a terminal, a generative AI model, a speech recognition system, and speech synthesis technology. Detailed embodiments of this system are described below.

[0550] The user logs in to the terminal and starts the interview practice. The user selects the desired scenario from multiple interview scenarios displayed on the terminal. For example, "technical interview" or "sales interview" can be selected. The selected scenario information is sent from the terminal to the server.

[0551] The server sends a request to the generative AI model to generate a list of questions based on the selected scenario. Here, a generative AI model that is strong in natural language processing (e.g., GPT-4) is used. For example, a prompt such as "Please generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10" is sent to the generative AI model. The generative AI model generates a list of questions based on this prompt.

[0552] The server sends the generated question list to the terminal, which then uses voice synthesis technology (e.g., Google Text-to-Speech) to ask the user aloud the questions. The questions are presented to the user one by one in order, prompting them to answer.

[0553] The user answers questions by voice. The user's answers are captured through the device's microphone and converted into text data by a voice recognition system (e.g., Google Cloud Speech-to-Text).

[0554] The device sends the converted text data to a server, which passes the text data to a generative AI model to evaluate the answer. For example, the generative AI model might be provided with a prompt such as, "Please rate the answer based on criteria such as consistency, specificity, and clarity." The generative AI model then analyzes the answer based on the evaluation criteria and generates an evaluation result and feedback.

[0555] The server sends the evaluation results and feedback to the device. The feedback includes the user's strengths and areas for improvement, as well as advice for the next practice. The device then tells the user the evaluation results and feedback by voice or displays them on the screen. This allows the user to improve their answers and reflect them in their next practice.

[0556] As a concrete example, the flow of a technical interview is shown below.

[0557] 1. The user logs in to the terminal and selects "Technical Interview Practice."

[0558] 2. The server sends a prompt to the generative AI model: "Generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10."

[0559] 3. The generative AI model generates a list of questions, which the server sends to the device.

[0560] 4. The device will ask the user aloud, "Please introduce yourself."

[0561] 5. The user responds verbally, "I'm a software engineer..."

[0562] 6. The device converts the voice into text data and sends it to the server.

[0563] 7. The server evaluates the answer using the generative AI model and generates a rating result that says, "The answer was clear and consistent."

[0564] 8. The device provides the user with audio feedback on the evaluation results.

[0565] In this way, the system allows users to conduct high-quality interview practice regardless of location or time.

[0566] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0567] Step 1:

[0568] A user logs in to a terminal

[0569] Input: User ID and password.

[0570] How it works: The user enters their ID and password on the device's login screen and presses the "Login" button.

[0571] Data processing: The device sends the user ID and password to the server, which checks them against the authentication information stored in the database.

[0572] Output: If authentication is successful, the server generates an authentication token and sends it to the device, which then displays the dashboard screen to the user.

[0573] Step 2:

[0574] The user selects an interview scenario

[0575] Input: Interview scenario (e.g. technical interview, sales interview).

[0576] How it works: The user clicks on the desired scenario from the list displayed on the device screen.

[0577] Data processing: The device sends the selected scenario information to the server.

[0578] Output: The server receives the scenario information.

[0579] Step 3:

[0580] The server sends a question list generation request to the generative AI model.

[0581] Input: Selected scenario information.

[0582] Operation: Based on the selected scenario information, the server sends a request to the generative AI model to generate a list of questions.

[0583] Data processing: The generative AI model generates a list of questions based on the prompt.

[0584] Example prompt: "Generate a list of 10 questions for a technical interview at an intermediate level."

[0585] Output: The generative AI model generates a list of questions and returns it to the server.

[0586] Step 4:

[0587] The server sends the question list to the terminal.

[0588] Input: The list of questions received from the generative AI model.

[0589] Operation: The server converts the generated question list into JSON format and sends it to the terminal.

[0590] Data processing: Format conversion and data transmission on the server.

[0591] Output: The terminal receives the list of questions.

[0592] Step 5:

[0593] The device asks the user a list of questions by voice

[0594] Input: Questionnaire.

[0595] How it works: The device uses speech synthesis technology to read the list of questions to the user.

[0596] Text-to-speech example: Google Text-to-Speech.

[0597] Data processing: Text to speech conversion.

[0598] Output: Presents the question to the user by voice, prompting the user for an answer.

[0599] Step 6:

[0600] The user answers the question

[0601] Input: User speaks the answer.

[0602] How it works: The user speaks the answer to a question into the terminal.

[0603] Data processing: The user's voice is recorded through the device's microphone.

[0604] Output: Recorded audio data.

[0605] Step 7:

[0606] The device converts the voice data into text data.

[0607] Input: Recorded audio data.

[0608] How it works: The device uses a voice recognition system to convert voice data into text data.

[0609] Speech recognition example: Google Cloud Speech-to-Text.

[0610] Data processing: Converting audio data into text data.

[0611] Output: The converted text data.

[0612] Step 8:

[0613] The device sends text data to the server.

[0614] Input: Text data.

[0615] Operation: The terminal converts the converted text data into JSON format and sends it to the server.

[0616] Data processing: format conversion and data transmission.

[0617] Output: The server receives the text data.

[0618] Step 9:

[0619] The server evaluates the answers using a generative AI model

[0620] Input: The converted text data.

[0621] How it works: The server passes the received text data to the generative AI model for evaluation.

[0622] Data processing: The generative AI model analyzes the answers based on the prompt sentence and evaluation criteria.

[0623] Sample prompt: "Please rate your answers based on criteria such as coherence, specificity, and clarity."

[0624] Output: The generative AI model generates evaluation results and feedback and returns them to the server.

[0625] Step 10:

[0626] The server sends the evaluation results and feedback to the device.

[0627] Input: Assessment results and feedback.

[0628] Operation: The server formats the evaluation information obtained from the generative AI model and sends it to the device.

[0629] Data processing: Data formatting and transmission.

[0630] Output: The device receives the evaluation results and feedback.

[0631] Step 11:

[0632] The device provides the evaluation results and feedback to the user.

[0633] Input: Assessment results and feedback.

[0634] What it does: The device will read out the assessment results and feedback aloud or display them on the screen.

[0635] Data processing: Using voice synthesis technology.

[0636] Output: User receives evaluation results and feedback.

[0637] (Application example 1)

[0638] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0639] Conventional interview practice systems have difficulty in providing situations similar to those of a real interview, limiting the effectiveness of user practice. Furthermore, in the actual hiring process, there are issues with the heavy burden on interviewers and the inconsistency of evaluations. Given these circumstances, there is a need for a system that can improve the quality of interview practice and ensure efficient and fair hiring of staff for brick-and-mortar stores.

[0640] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0641] In this invention, the server includes means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user the list of questions acquired from the server by voice or text, means for the terminal to convert the user's voice or text answers into text data using a voice recognition system, means for sending the user's answers to the server, evaluating the answers by the generation AI, generating evaluation results and feedback and sending them to the terminal, and means for the terminal to provide the user with the evaluation results and feedback by voice or display. This makes it possible to provide practical interview practice and enables fair and efficient evaluation even in staff recruitment at physical stores.

[0642] "User" means any individual or organization that uses the System.

[0643] An "interview scenario" is a set of questions designed to fit a specific interview situation.

[0644] "Generative AI" refers to a system that uses artificial intelligence technology to solve problems and generate questions.

[0645] A "question list" refers to a collection of questions provided for a user to answer.

[0646] "Server" refers to the central processing unit responsible for data processing and storage within a system.

[0647] "Terminal" refers to a device (smartphone, tablet, computer, etc.) that is directly operated by a user.

[0648] A "voice recognition system" refers to a technology that converts voice data into text data.

[0649] "Text data" refers to character information converted by a voice recognition system.

[0650] "Evaluation result" refers to the result of the evaluation made by the generation AI on the user's answer.

[0651] "Feedback" refers to advice or suggestions provided to the user based on the evaluation results.

[0652] "Audio or display" refers to the means for providing output information to the user, and includes both audio output and visual display.

[0653] This invention is a system that provides practical interview practice using a question list generated by a generation AI based on an interview scenario selected by the user. A specific embodiment of this system is described below.

[0654] First, a user must log in to a terminal to start the interview practice. Next, the user selects a specific scenario from multiple interview scenarios (e.g., technical interview, sales interview, etc.) through the terminal. This selection information is sent from the terminal to the server.

[0655] The server sends a request to a generative AI to generate a list of questions based on the interview scenario selected by the user, using a generative AI model such as OpenAI's GPT-3, and then sends the generated list of questions to the device.

[0656] The device asks the user questions in order based on the received question list. These questions are provided in the form of voice or text. When the user responds by voice, the device's voice recognition system converts the voice data into text data. Here, Python's speech_recognition library, etc., is used.

[0657] The user's answer, converted into text data, is sent back to the server, where the generation AI evaluates the answer based on criteria such as consistency, specificity, and clarity. The server then uses the generation AI to generate evaluation results and feedback, which are then sent to the device.

[0658] The device provides the user with the evaluation results and feedback, which can be presented as audio or visual, allowing the user to improve their answers and use them for their next practice.

[0659] Specific examples

[0660] For example, in a cashier interview scenario, the following questions might be generated:

[0661] Example prompt sentence:

[0662] Generate interview questions based on the interview scenario "Cashier."

[0663] The resulting questionnaire:

[0664] 1. Please introduce yourself

[0665] 2. Tell us about your experience in treating all customers with kindness.

[0666] 3. How do you manage stress during busy periods?

[0667] In this way, users can conduct high-quality interview practice regardless of location or time. By applying this to staff recruitment in brick-and-mortar stores, it is expected to realize a fair and efficient interview process and reduce the burden on interviewers.

[0668] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0669] Step 1:

[0670] A user logs in to a terminal. The user launches an application on the terminal, enters the required credentials on the login screen, and performs the login operation, completing the authentication. The input is the user's credentials, and the output is an authentication success message.

[0671] Step 2:

[0672] The user selects the desired interview scenario. The user makes a selection from multiple interview scenarios (e.g., technical interview, sales interview, cashier, etc.) displayed on the terminal screen. The input is the user's selection information, and the output is data indicating the selected scenario.

[0673] Step 3:

[0674] The terminal sends the selected scenario information to the server. The terminal converts the user-selected scenario information into a data packet and sends a request to the server. The input is the selected scenario data, and the output is a request to the server.

[0675] Step 4:

[0676] The server requests the generative AI model to generate a list of questions. The server sends a prompt to the generative AI model (e.g., GPT-3) based on the selected scenario, causing it to create a list of questions. The input is the prompt, and the output is the generated list of questions.

[0677] Step 5:

[0678] The server sends the generated question list to the terminal. The server returns the question list obtained from the generative AI model to the terminal. The input is the generated question list, and the output is the data sent to the terminal.

[0679] Step 6:

[0680] The terminal asks the user a list of questions by voice or text. Based on the received list of questions, the terminal uses a speech synthesis system (e.g., gTTS) to present the questions to the user by voice or displays them on the screen as text. The input is the question list data, and the output is voice or text display.

[0681] Step 7:

[0682] The user answers the question. The user answers the question displayed on the terminal by voice or text. The input is the user's voice response to the question, and the output is voice data.

[0683] Step 8:

[0684] The device converts the user's voice into text data. The device's voice recognition system (e.g., speech_recognition) converts the user's voice response into text data. The input is voice data, and the output is text data.

[0685] Step 9:

[0686] The terminal sends the converted text data to the server. The terminal sends the converted text data to the server and makes an evaluation request. The input is the text data, and the output is a request to the server.

[0687] Step 10:

[0688] The server requests the generative AI model to evaluate the answer. The server provides the received text data to the generative AI model and asks for an evaluation. The input is the text data, and the output is the evaluation result and feedback.

[0689] Step 11:

[0690] The server generates evaluation results and feedback and sends them to the terminal. The server evaluates based on evaluation criteria (e.g., consistency, specificity, clarity), creates feedback, and sends it to the terminal. The input is the evaluation result data, and the output is the feedback data.

[0691] Step 12:

[0692] The terminal provides the evaluation results and feedback to the user. The terminal presents the received evaluation results and feedback by voice or displays them on a screen. The input is the feedback data, and the output is voice or text display.

[0693] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0694] This invention is a system that combines an emotion engine and provides practical interview practice via a terminal using a question list generated by a generative AI based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotions into the evaluation.

[0695] Program processing

[0696] 1. The user logs in to the system using a terminal and begins the interview practice.

[0697] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[0698] 2. The terminal sends the user's selection information to the server.

[0699] The request sent to the server includes the user ID and the selected scenario type.

[0700] 3. The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0701] The generative AI generates a list of questions based on past question examples and best practices.

[0702] 4. The server sends the generated question list to the terminal.

[0703] The list of questions was tailored to the selected interview scenario.

[0704] 5. The device receives the list of questions and asks the user voice questions using the ChatBot function.

[0705] The questions are presented to the user aloud one by one in turn.

[0706] 6. The user answers the question by voice.

[0707] The user's voice response is recorded through the device's microphone.

[0708] 7. The device uses a voice recognition system to convert the user's voice response into text data.

[0709] The converted text data is sent to a server for evaluation.

[0710] 8. The server receives the user's text data and evaluates the content using a generation AI.

[0711] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[0712] 9. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0713] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[0714] 10. The emotion engine sends the recognized emotion data to the device, and the device then sends the data to the server.

[0715] 11. The server uses generated AI to evaluate the user's answers, including emotional data.

[0716] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[0717] 12. The server generates the evaluation results and feedback and sends them to the device.

[0718] Feedback includes user strengths, areas for improvement, and emotional advice.

[0719] 13. The device displays or audibly conveys the evaluation results and feedback to the user.

[0720] Through feedback, users can improve their answers and emotional state and reflect this in their next practice.

[0721] Specific examples

[0722] (Example) For a technical interview:

[0723] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[0724] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[0725] 3. The user responds verbally, "I'm a software engineer..."

[0726] 4. The device converts the voice into text data and sends it to the server.

[0727] 5. The server receives the answer and evaluates it using the generation AI.

[0728] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0729] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[0730] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[0731] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[0732] In this way, users can not only practice for interviews at high quality regardless of time or place, but also understand their emotional state and make comprehensive improvements, including their mental state.

[0733] The processing flow will be explained below.

[0734] Step 1:

[0735] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[0736] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[0737] Step 2:

[0738] The terminal transmits the user's scenario selection information to the server.

[0739] The request sent to the server includes the user ID and the selected scenario type.

[0740] Step 3:

[0741] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0742] The generative AI generates a list of questions based on past question examples and best practices.

[0743] Step 4:

[0744] The server transmits the generated question list to the terminal.

[0745] The list of questions was tailored to the selected interview scenario.

[0746] Step 5:

[0747] The device receives a list of questions and asks the user aloud using the ChatBot function.

[0748] The questions are presented to the user aloud one by one.

[0749] Step 6:

[0750] The user answers the questions by voice.

[0751] The user's voice response is recorded through the device's microphone.

[0752] Step 7:

[0753] The terminal uses a voice recognition system to convert the user's voice response into text data.

[0754] The converted text data is sent to the server.

[0755] Step 8:

[0756] The server receives the user's text data and evaluates the content using a generation AI.

[0757] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[0758] Step 9:

[0759] The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0760] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[0761] Step 10:

[0762] The emotion engine transmits the recognized emotion data to the terminal, which then transmits the data to the server.

[0763] Emotional data is used as additional information for evaluation.

[0764] Step 11:

[0765] The server uses generated AI to evaluate the user's answers, including emotional data.

[0766] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[0767] Step 12:

[0768] The server generates evaluation results and feedback and sends them to the device.

[0769] Feedback includes specific recommendations based on user strengths, areas for improvement, and feelings.

[0770] Step 13:

[0771] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[0772] Users can review the feedback and use it to improve their answers and emotional state.

[0773] Specific examples

[0774] (Example) For a technical interview:

[0775] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[0776] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[0777] 3. The user responds verbally, "I'm a software engineer..."

[0778] 4. The device converts the voice into text data and sends it to the server.

[0779] 5. The server receives the answer and evaluates it using the generation AI.

[0780] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0781] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[0782] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[0783] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[0784] In this way, users can not only practice for interviews at high quality without being tied down to a specific place or time, but also understand their emotional state and make comprehensive improvements, including their own emotional state.

[0785] Example 2

[0786] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0787] Conventional interview practice systems evaluate only the content of the user's voice responses, making it difficult to provide comprehensive feedback that takes into account the user's emotional state and mental state. This leaves users with no way to effectively overcome feelings of tension and anxiety, making it difficult for them to maximize their performance in an actual interview.

[0788] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0789] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for the terminal to ask the user verbally the questions from the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a voice recognition system, means for transmitting the user's answers to the server, evaluating the content of the answers using the generative AI model, and analyzing the voice data using an emotion engine to recognize the user's emotional state, means for generating and transmitting evaluation results and feedback to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to receive multifaceted evaluation and feedback that takes into account not only the content of their answers but also their emotional state.

[0790] A "generative AI model" is an artificial intelligence that uses machine learning technology to automatically generate a list of questions and answer evaluations based on user input.

[0791] A "terminal" is an electronic device used by a user to practice for an interview, such as a smartphone, tablet, or PC.

[0792] The "server" is a central computer system that processes data for the entire system, generates questionnaires, and executes evaluations.

[0793] An "interview scenario" is a mock interview situation consisting of a number of predetermined questions and situations.

[0794] A "question list" is a set of interview questions generated by a generative AI model.

[0795] A "voice recognition system" is a technology that analyzes a user's voice and converts it into text data.

[0796] "Emotion engine" is an artificial intelligence technology that analyzes a user's voice data to recognize their emotional state.

[0797] The "evaluation results" are a collection of feedback obtained by the generative AI model by analyzing the user's responses and emotional state.

[0798] "Feedback" is information about evaluations and advice regarding the user's interview performance.

[0799] This invention is an interview practice system that combines a generative AI model and an emotion engine to provide practical interview practice based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotional state into the evaluation.

[0800] First, the terminal used by the user is an electronic device such as a smartphone, tablet, or PC, which is used to access and log in to the system. After logging in, the user selects the desired scenario from multiple interview scenarios (e.g., technical interview, sales interview, behavioral interview, etc.). The specific operation of the system is explained below.

[0801] The server requests the generative AI model to generate a list of questions based on the interview scenario selected by the user. For example, if a technical interview scenario is selected, the prompt message is "Please generate a list of questions for the technical interview." The generative AI model references past example questions and best practices to generate a list of questions appropriate to the selected scenario. The generated list of questions is sent from the server to the device in a format such as JSON.

[0802] Based on the received question list, the device uses its ChatBot function to present questions to the user by voice. For example, questions such as "Please introduce yourself" are asked in sequence. When the user responds by voice, the response is recorded through the device's microphone. The device then uses a voice recognition system to convert the voice response into text data in real time and transmits this data to the server.

[0803] The server receives the user's text data and sends the recorded voice data to the emotion engine, which analyzes the voice data and recognizes the user's emotional state (e.g., nervousness, anxiety, confidence, etc.). The recognized emotion data is then sent to the server.

[0804] The server uses a generative AI model to comprehensively evaluate the text data and emotional data, taking into account the consistency, specificity, and clarity of the answers, as well as the user's emotional state. For example, the server may generate an evaluation such as, "Your answers were consistent and specific, but you seemed a little nervous." Once the evaluation results and feedback are generated, they are sent to the device, which then provides them to the user in audio or display format. The user can use this feedback to improve their answers and emotional state and reflect them in their next practice.

[0805] As a concrete example, if a user selects a technical interview scenario, the following prompt is sent to the generative AI model: "Generate a list of questions for a technical interview." A list of questions is generated, and the question "Please introduce yourself" is presented. The user responds, "I'm a software engineer..." and the audio data is recorded and converted into text data. At the same time, the emotion engine generates a recognition result such as "The user is nervous." The generative AI model evaluates the content of the response and the user's emotional state, and generates feedback such as "The response was clear and consistent, but you seem a little nervous."

[0806] As described above, this system combines voice recognition technology and emotion analysis technology to provide users with comprehensive interview practice and feedback.

[0807] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] A user logs into the system using a terminal and selects an interview scenario.

[0810] Specifically, the user enters their ID and password on the login screen and clicks the login button. After logging in, a dashboard is displayed where the user selects the desired interview scenario from among "technical interview" and "sales interview" scenarios. The input is the user ID and password, and the output is information about the selected scenario.

[0811] Step 2:

[0812] The terminal transmits the user's selection information to the server.

[0813] The input is the user ID and the selected scenario information, which the device sends to the server as a request. The specific operation is to send an HTTP request to the server. The output is the request data sent to the server.

[0814] Step 3:

[0815] The server sends a request to the generative AI model to generate a list of questions.

[0816] The input is the user's selection information, and the server sends a prompt to the generative AI model based on this data. For example, the prompt might say, "Please generate a list of questions for a technical interview." Specifically, an API request is sent to the generative AI model. The output is the generated list of questions.

[0817] Step 4:

[0818] The server transmits the generated question list to the terminal.

[0819] The input is a list of questions returned by the generative AI model, and the server sends this list to the device. Specifically, an HTTP response is sent to the device. The output is the list of questions that arrives at the device.

[0820] Step 5:

[0821] The device receives the list of questions and asks the user voice questions using the ChatBot function.

[0822] The input is a list of questions received from the server, and the ChatBot function presents the questions to the user one by one by voice. Specifically, speech synthesis software is used. The output is the voice question presented to the user.

[0823] Step 6:

[0824] The user answers the questions by voice.

[0825] The input is a voice question from the terminal, and the user answers by voice. Specifically, the terminal's microphone records the user's voice. The output is the recorded voice data.

[0826] Step 7:

[0827] The terminal uses a voice recognition system to convert the voice response into text data and transmits it to the server.

[0828] The input is the user's voice data, which the device converts into text data using a voice recognition system. Specifically, voice recognition software is used. The output is the converted text data, which is sent to the server.

[0829] Step 8:

[0830] The server receives the user's text and voice data and analyzes it using a generative AI model and emotion engine.

[0831] The input is the user's text data and voice data, and the server has the generative AI model analyze the text data and the emotion engine analyze the voice data. Specifically, an API request is sent to the generative AI model and the emotion engine. The output is the evaluation result and emotion data.

[0832] Step 9:

[0833] The server generates evaluation results and feedback and sends them to the device.

[0834] The input is the analyzed evaluation results and emotion data, and the server synthesizes these to generate feedback. Specifically, the generative AI model automatically generates the feedback. The output is feedback data, which is sent to the device.

[0835] Step 10:

[0836] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[0837] The input is the feedback data received from the server, which the device displays or speaks to the user. Specific actions include presenting the feedback on a display or through voice synthesis. The output is the feedback provided to the user.

[0838] (Application example 2)

[0839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] Conventional interview practice systems focus on evaluating the content of the user's answers, but are unable to incorporate the user's emotional state into the evaluation. This has led to the problem of being unable to properly manage emotions such as tension and anxiety that users may experience during an actual interview. Furthermore, in the customer service industry, the emotional state of store clerks significantly affects the quality of customer service, so an evaluation system that includes emotion recognition has been in demand.

[0841] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0842] In this invention, the server includes: means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user; means for the terminal to ask the user aloud questions from the list of questions acquired from the server; means for the terminal to convert the user's spoken answers into text data using a voice recognition system; means for transmitting the user's answers to the server, evaluating the content of the answers using the generation AI, generating evaluation results and feedback and transmitting them to the terminal; means for the terminal to transmit recorded voice data to an emotion recognition engine and analyze the user's emotions; and means for transmitting emotional data recognized by the emotion recognition engine to the server and for the generation AI to evaluate the user's answers, including the emotional data. This makes it possible to incorporate not only the content of the user's answers but also their emotional state at the time of answering into the evaluation. This allows users to understand their own emotional state and engage in more practical and comprehensive interview practice and customer service training.

[0843] "User" refers to an individual who uses the system to practice interviews and customer service training.

[0844] An "interview scenario" is a practice scenario that assumes a specific interview situation, and includes different scenarios such as a technical interview and a sales interview.

[0845] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate question lists and assessments.

[0846] A "question list" is a collection of multiple questions generated by generative AI and is constructed based on an interview scenario.

[0847] The "server" refers to a central processing unit that processes user selection information and response data, and generates evaluations and feedback by working with a generative AI and emotion recognition engine.

[0848] A "terminal" is a device operated by a user, including a smartphone or a head-mounted display (HMD).

[0849] "Speech recognition system" refers to technology that converts a user's voice response into text data.

[0850] "Text data" refers to sentence data converted from speech by a speech recognition system.

[0851] An "emotion recognition engine" refers to a system that analyzes a user's emotional state (e.g., tension, anxiety, confidence, etc.) from their voice data.

[0852] "Feedback" refers to advice or evaluation comments provided as a result of evaluating the content of a user's answers or emotional state.

[0853] This invention provides a system for interview practice and customer service training. The system includes a user, a terminal, and a server, and operates and processes as follows.

[0854] Program Overview

[0855] Users log in to the system using a device such as a smartphone or head-mounted display (HMD). The user selects the desired scenario from multiple interview scenarios or customer service scenarios (for example, technical interview, sales interview, product explanation, complaint handling, etc.). This selection information is sent from the device to the server. The server uses this information to have the generation AI generate a list of questions. The generated list of questions is sent to the device, which then asks the user questions by voice. The user then answers by voice, and the answers are converted into text data using the device's voice recognition system, and this text data is sent to the server.

[0856] Data processing and calculation

[0857] The server uses a generative AI to evaluate the text data and analyze it based on criteria such as consistency, specificity, and clarity of the content. The device also sends the recorded voice data to an emotion recognition engine to analyze the user's emotions. This emotion data is also sent to the server. The server then uses the generative AI to perform a comprehensive evaluation, including the emotion data, and generates and sends the evaluation results and feedback to the device. The device then provides this feedback to the user as voice or display.

[0858] Hardware and software used

[0859] Hardware: Smartphone, Head-Mounted Display (HMD)

[0860] Software: Speech recognition system (e.g., Google Speech Recognition), emotion recognition engine (custom or external API), generative AI (for questionnaire generation and evaluation)

[0861] Specific examples

[0862] For example, consider the case where a store clerk launches a customer service training app and selects the "Handling Complaints" scenario. The device retrieves the question from the server, "Please tell us how you would respond if you received a complaint from a customer," and asks the clerk aloud. When the clerk responds, "First, I'll listen to the customer's story to the end...," the content is converted into text data by a voice recognition system. The server receives this text data and evaluates the content and emotional state using a generative AI and an emotion recognition engine. The server then generates feedback such as, "Your opinion is clear, but you seem a little nervous," and the device presents it to the clerk.

[0863] Prompt Sentence Examples

[0864] How do you explain your products to customers?

[0865] What is the most important point when dealing with a complaint?

[0866] By using these concrete examples and prompts, users can understand their own emotional state while conducting practical and comprehensive interview practice and customer service training.

[0867] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0868] Step 1:

[0869] A user logs in to the system from a smartphone or head-mounted display (HMD). The input is the user's authentication information, and the output is authentication that the login was successful. Specifically, a login form is displayed on the terminal screen, and the user enters their ID and password.

[0870] Step 2:

[0871] The user selects the desired scenario from multiple interview or customer service scenarios. The input is the information about the scenario selected by the user, and the output is the ID of the selected scenario. Specifically, a list of scenarios is displayed on the screen, and the user taps to select the scenario they are interested in.

[0872] Step 3:

[0873] The device sends the user's selection information to the server. The input is the selected scenario ID and user ID, and the output is a receipt confirmation from the server. Specifically, the device sends data to the server using an HTTP request.

[0874] Step 4:

[0875] The server makes the generation AI generate a list of questions based on the selected scenario ID. The input is the scenario ID, and the output is the generated list of questions. Specifically, the server calls the generation AI model and inputs a prompt to generate questions based on the scenario.

[0876] Step 5:

[0877] The server sends the generated question list to the device. The input is the question list generated by the generation AI, and the output is the received question list. Specifically, the server sends the question list to the device in an HTTP response.

[0878] Step 6:

[0879] The device receives the list of questions and asks the user aloud. The input is the received list of questions, and the output is the spoken questions to the user. Specifically, the device uses text-to-speech synthesis (TTS) to play back the questions aloud.

[0880] Step 7:

[0881] The user answers the question by voice. The input is the voice question, and the output is the user's voice response. Specifically, the user responds through the device's microphone.

[0882] Step 8:

[0883] The device uses a voice recognition system to convert the user's voice response into text data. The input is the user's voice response, and the output is text data. Specifically, the device uses a voice recognition library to convert voice to text.

[0884] Step 9:

[0885] The device sends text data to the server. The input is text data converted from speech, and the output is confirmation of transmission to the server. Specifically, the device sends the text data to the server via an HTTP request.

[0886] Step 10:

[0887] The server evaluates the text data using generative AI. The input is text data, and the output is the evaluation result. Specifically, the server feeds the text data to the generative AI model and evaluates the consistency, concreteness, and clarity of the content.

[0888] Step 11:

[0889] The device sends recorded voice data to the emotion recognition engine to analyze the user's emotions. The input is recorded voice data and the output is emotion data. Specifically, the device sends the voice data to the emotion recognition engine via an HTTP request.

[0890] Step 12:

[0891] The emotion recognition engine sends the recognized emotion data to the server. The input is the emotion data, and the output is a confirmation of transmission to the server. Specifically, the emotion recognition engine sends the analysis results to the server via an HTTP request.

[0892] Step 13:

[0893] The server uses a generation AI to evaluate the user's answers, including emotional data. The input is text data and emotional data, and the output is a comprehensive evaluation result. Specifically, the server inputs emotional data into the generation AI and performs a comprehensive evaluation.

[0894] Step 14:

[0895] The server generates evaluation results and feedback and sends them to the terminal. The input is the overall evaluation result, and the output is the evaluation result and feedback. Specifically, the server organizes the evaluation results, generates feedback for the user, and sends it to the terminal.

[0896] Step 15:

[0897] The device provides the evaluation results and feedback to the user by voice or display. The input is the evaluation results and feedback, and the output is provided to the user. Specifically, the device uses text-to-speech (TTS) to play back the feedback by voice or displays it on the screen.

[0898] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0899] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0900] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0901] [Third embodiment]

[0902] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0903] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0904] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0905] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0906] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0907] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0908] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0909] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0910] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0911] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0912] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0913] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0914] The present invention is a system that provides practical interview practice via a terminal using a question list generated by a generation AI based on a specific interview scenario selected by the user.

[0915] Program processing

[0916] 1. The user logs in to the system using a terminal and begins the interview practice.

[0917] The user selects the desired scenario from multiple interview scenarios on the terminal screen.

[0918] 2. The terminal sends the user's selection information to the server.

[0919] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0920] 3. The server uses the generation AI to generate a list of questions corresponding to the scenario and sends it to the device.

[0921] The list of questions corresponds to the interview scenario selected by the user (e.g., technical interview or sales interview).

[0922] 4. The device receives the list of questions and uses the ChatBot function to ask the questions to the user via voice.

[0923] The questions are presented to the user one by one in turn by voice, prompting the user to respond.

[0924] 5. The user answers the question by voice.

[0925] The user's response is converted into text data by a voice recognition system via the device's microphone.

[0926] 6. The device sends the converted text data to the server.

[0927] The server has the generation AI evaluate the received text data.

[0928] 7. The server uses the generated AI to evaluate the user's answers.

[0929] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity of the answer.

[0930] 8. The server generates the evaluation results and feedback and sends them to the device.

[0931] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[0932] 9. The device displays or audibly conveys the evaluation results and feedback to the user.

[0933] Users can improve their answers through feedback and incorporate it into their next practice.

[0934] Specific examples

[0935] (Example) For a technical interview:

[0936] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[0937] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[0938] 3. The user responds verbally, "I'm a software engineer..."

[0939] 4. The device converts the voice into text data and sends it to the server.

[0940] 5. The server receives the answer and uses the generation AI to generate an evaluation result such as "The answer was clear and consistent."

[0941] 6. The device will provide the user with audible feedback on the evaluation results and proceed to the next question.

[0942] In this way, the system allows users to practice interviews at high quality, regardless of location or time. This system is expected to increase users' confidence through practical practice and improve their success rate in interviews.

[0943] The processing flow will be explained below.

[0944] Step 1:

[0945] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[0946] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[0947] Step 2:

[0948] The terminal transmits the user's scenario selection information to the server.

[0949] The request sent to the server includes the user ID and the selected scenario type.

[0950] Step 3:

[0951] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[0952] The generative AI generates a list of questions based on past question examples and best practices.

[0953] Step 4:

[0954] The server transmits the generated question list to the terminal.

[0955] The list of questions was tailored to the selected interview scenario.

[0956] Step 5:

[0957] The device receives a list of questions and asks the user aloud using the ChatBot function.

[0958] The questions are presented to the user aloud one by one in turn.

[0959] Step 6:

[0960] The user answers the questions by voice.

[0961] The user's voice response is recorded through the device's microphone.

[0962] Step 7:

[0963] The terminal uses a voice recognition system to convert the user's voice response into text data.

[0964] The converted text data is sent to a server for evaluation.

[0965] Step 8:

[0966] The server receives the user's text data and evaluates the content using a generation AI.

[0967] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[0968] Step 9:

[0969] The server generates evaluation results and feedback and sends them to the device.

[0970] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[0971] Step 10:

[0972] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[0973] Users can review the feedback and use it to improve their answers.

[0974] Example 1

[0975] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0976] Conventional interview practice systems have the problem that it is difficult for users to practice in conditions similar to a real interview environment, and in particular, they do not provide sufficient individual feedback or evaluation, which limits the user's skill improvement.In addition, existing systems often do not provide comprehensive evaluation based on evaluation criteria such as consistency, concreteness, and clarity of answers required in a real interview.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0978] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user verbally the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a speech recognition system, means for evaluating the acquired answers with the generative AI model using prompt sentences created based on the generative AI model, means for transmitting the user's answers to the server, evaluating the contents of the answers with the generative AI model, generating evaluation results and feedback and transmitting them to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to engage in more practical and high-quality interview practice and receive detailed feedback.

[0979] A "user" is an individual or organization that uses the interview practice system to practice for an interview.

[0980] A "terminal" is an electronic device used by a user, such as a computer, smartphone, or tablet, that functions as an interface with the system.

[0981] The "server" is a central computer system that works in conjunction with the terminal to generate a list of questions using a generative AI model, evaluate answers, and provide feedback.

[0982] An "interview scenario" refers to multiple interview situations or themes that a user can select, such as different situations such as a technical interview or a sales interview.

[0983] A "generative AI model" is an artificial intelligence algorithm that generates a list of questions and evaluates answers based on user input data, and primarily uses natural language processing technology.

[0984] A "question list" is a set of questions generated by a generative AI model based on an interview scenario, and is presented to the user.

[0985] A "voice recognition system" is a technology that converts a user's verbal response from voice data into text data.

[0986] A "prompt" is a document that defines the basic instructions and conditions that a generative AI model uses to conduct a question list or evaluation.

[0987] "Feedback" is information that includes the evaluation results of the user's answers and points for improvement, and is provided to the user.

[0988] This invention is a system that uses a generative AI model to generate a list of questions based on a specific interview scenario selected by the user, and provides practical interview practice via a terminal. This system is composed of a server, a terminal, a generative AI model, a speech recognition system, and speech synthesis technology. Detailed embodiments of this system are described below.

[0989] The user logs in to the terminal and starts the interview practice. The user selects the desired scenario from multiple interview scenarios displayed on the terminal. For example, "technical interview" or "sales interview" can be selected. The selected scenario information is sent from the terminal to the server.

[0990] The server sends a request to the generative AI model to generate a list of questions based on the selected scenario. Here, a generative AI model that is strong in natural language processing (e.g., GPT-4) is used. For example, a prompt such as "Please generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10" is sent to the generative AI model. The generative AI model generates a list of questions based on this prompt.

[0991] The server sends the generated question list to the terminal, which then uses voice synthesis technology (e.g., Google Text-to-Speech) to ask the user aloud the questions. The questions are presented to the user one by one in order, prompting them to answer.

[0992] The user answers questions by voice. The user's answers are captured through the device's microphone and converted into text data by a voice recognition system (e.g., Google Cloud Speech-to-Text).

[0993] The device sends the converted text data to a server, which passes the text data to a generative AI model to evaluate the answer. For example, the generative AI model might be provided with a prompt such as, "Please rate the answer based on criteria such as consistency, specificity, and clarity." The generative AI model then analyzes the answer based on the evaluation criteria and generates an evaluation result and feedback.

[0994] The server sends the evaluation results and feedback to the device. The feedback includes the user's strengths and areas for improvement, as well as advice for the next practice. The device then tells the user the evaluation results and feedback by voice or displays them on the screen. This allows the user to improve their answers and reflect them in their next practice.

[0995] As a concrete example, the flow of a technical interview is shown below.

[0996] 1. The user logs in to the terminal and selects "Technical Interview Practice."

[0997] 2. The server sends a prompt to the generative AI model: "Generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10."

[0998] 3. The generative AI model generates a list of questions, which the server sends to the device.

[0999] 4. The device will ask the user aloud, "Please introduce yourself."

[1000] 5. The user responds verbally, "I'm a software engineer..."

[1001] 6. The device converts the voice into text data and sends it to the server.

[1002] 7. The server evaluates the answer using the generative AI model and generates a rating result that says, "The answer was clear and consistent."

[1003] 8. The device provides the user with audio feedback on the evaluation results.

[1004] In this way, the system allows users to conduct high-quality interview practice regardless of location or time.

[1005] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1006] Step 1:

[1007] A user logs in to a terminal

[1008] Input: User ID and password.

[1009] How it works: The user enters their ID and password on the device's login screen and presses the "Login" button.

[1010] Data processing: The device sends the user ID and password to the server, which checks them against the authentication information stored in the database.

[1011] Output: If authentication is successful, the server generates an authentication token and sends it to the device, which then displays the dashboard screen to the user.

[1012] Step 2:

[1013] The user selects an interview scenario

[1014] Input: Interview scenario (e.g. technical interview, sales interview).

[1015] How it works: The user clicks on the desired scenario from the list displayed on the device screen.

[1016] Data processing: The device sends the selected scenario information to the server.

[1017] Output: The server receives the scenario information.

[1018] Step 3:

[1019] The server sends a question list generation request to the generative AI model.

[1020] Input: Selected scenario information.

[1021] Operation: Based on the selected scenario information, the server sends a request to the generative AI model to generate a list of questions.

[1022] Data processing: The generative AI model generates a list of questions based on the prompt.

[1023] Example prompt: "Generate a list of 10 questions for a technical interview at an intermediate level."

[1024] Output: The generative AI model generates a list of questions and returns it to the server.

[1025] Step 4:

[1026] The server sends the question list to the terminal.

[1027] Input: The list of questions received from the generative AI model.

[1028] Operation: The server converts the generated question list into JSON format and sends it to the terminal.

[1029] Data processing: Format conversion and data transmission on the server.

[1030] Output: The terminal receives the list of questions.

[1031] Step 5:

[1032] The device asks the user a list of questions by voice

[1033] Input: Questionnaire.

[1034] How it works: The device uses speech synthesis technology to read the list of questions to the user.

[1035] Text-to-speech example: Google Text-to-Speech.

[1036] Data processing: Text to speech conversion.

[1037] Output: Presents the question to the user by voice, prompting the user for an answer.

[1038] Step 6:

[1039] The user answers the question

[1040] Input: User speaks the answer.

[1041] How it works: The user speaks the answer to a question into the terminal.

[1042] Data processing: The user's voice is recorded through the device's microphone.

[1043] Output: Recorded audio data.

[1044] Step 7:

[1045] The device converts the voice data into text data.

[1046] Input: Recorded audio data.

[1047] How it works: The device uses a voice recognition system to convert voice data into text data.

[1048] Speech recognition example: Google Cloud Speech-to-Text.

[1049] Data processing: Converting audio data into text data.

[1050] Output: The converted text data.

[1051] Step 8:

[1052] The device sends text data to the server.

[1053] Input: Text data.

[1054] Operation: The terminal converts the converted text data into JSON format and sends it to the server.

[1055] Data processing: format conversion and data transmission.

[1056] Output: The server receives the text data.

[1057] Step 9:

[1058] The server evaluates the answers using a generative AI model

[1059] Input: The converted text data.

[1060] How it works: The server passes the received text data to the generative AI model for evaluation.

[1061] Data processing: The generative AI model analyzes the answers based on the prompt sentence and evaluation criteria.

[1062] Sample prompt: "Please rate your answers based on criteria such as coherence, specificity, and clarity."

[1063] Output: The generative AI model generates evaluation results and feedback and returns them to the server.

[1064] Step 10:

[1065] The server sends the evaluation results and feedback to the device.

[1066] Input: Assessment results and feedback.

[1067] Operation: The server formats the evaluation information obtained from the generative AI model and sends it to the device.

[1068] Data processing: Data formatting and transmission.

[1069] Output: The device receives the evaluation results and feedback.

[1070] Step 11:

[1071] The device provides the evaluation results and feedback to the user.

[1072] Input: Assessment results and feedback.

[1073] What it does: The device will read out the assessment results and feedback aloud or display them on the screen.

[1074] Data processing: Using voice synthesis technology.

[1075] Output: User receives evaluation results and feedback.

[1076] (Application example 1)

[1077] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1078] Conventional interview practice systems have difficulty in providing situations similar to those of a real interview, limiting the effectiveness of user practice. Furthermore, in the actual hiring process, there are issues with the heavy burden on interviewers and the inconsistency of evaluations. Given these circumstances, there is a need for a system that can improve the quality of interview practice and ensure efficient and fair hiring of staff for brick-and-mortar stores.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1080] In this invention, the server includes means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user the list of questions acquired from the server by voice or text, means for the terminal to convert the user's voice or text answers into text data using a voice recognition system, means for sending the user's answers to the server, evaluating the answers by the generation AI, generating evaluation results and feedback and sending them to the terminal, and means for the terminal to provide the user with the evaluation results and feedback by voice or display. This makes it possible to provide practical interview practice and enables fair and efficient evaluation even in staff recruitment at physical stores.

[1081] "User" means any individual or organization that uses the System.

[1082] An "interview scenario" is a set of questions designed to fit a specific interview situation.

[1083] "Generative AI" refers to a system that uses artificial intelligence technology to solve problems and generate questions.

[1084] A "question list" refers to a collection of questions provided for a user to answer.

[1085] "Server" refers to the central processing unit responsible for data processing and storage within a system.

[1086] "Terminal" refers to a device (smartphone, tablet, computer, etc.) that is directly operated by a user.

[1087] A "voice recognition system" refers to a technology that converts voice data into text data.

[1088] "Text data" refers to character information converted by a voice recognition system.

[1089] "Evaluation result" refers to the result of the evaluation made by the generation AI on the user's answer.

[1090] "Feedback" refers to advice or suggestions provided to the user based on the evaluation results.

[1091] "Audio or display" refers to the means for providing output information to the user, and includes both audio output and visual display.

[1092] This invention is a system that provides practical interview practice using a question list generated by a generation AI based on an interview scenario selected by the user. A specific embodiment of this system is described below.

[1093] First, a user must log in to a terminal to start the interview practice. Next, the user selects a specific scenario from multiple interview scenarios (e.g., technical interview, sales interview, etc.) through the terminal. This selection information is sent from the terminal to the server.

[1094] The server sends a request to a generative AI to generate a list of questions based on the interview scenario selected by the user, using a generative AI model such as OpenAI's GPT-3, and then sends the generated list of questions to the device.

[1095] The device asks the user questions in order based on the received question list. These questions are provided in the form of voice or text. When the user responds by voice, the device's voice recognition system converts the voice data into text data. Here, Python's speech_recognition library, etc., is used.

[1096] The user's answer, converted into text data, is sent back to the server, where the generation AI evaluates the answer based on criteria such as consistency, specificity, and clarity. The server then uses the generation AI to generate evaluation results and feedback, which are then sent to the device.

[1097] The device provides the user with the evaluation results and feedback, which can be presented as audio or visual, allowing the user to improve their answers and use them for their next practice.

[1098] Specific examples

[1099] For example, in a cashier interview scenario, the following questions might be generated:

[1100] Example prompt sentence:

[1101] Generate interview questions based on the interview scenario "Cashier."

[1102] The resulting questionnaire:

[1103] 1. Please introduce yourself

[1104] 2. Tell us about your experience in treating all customers with kindness.

[1105] 3. How do you manage stress during busy periods?

[1106] In this way, users can conduct high-quality interview practice regardless of location or time. By applying this to staff recruitment in brick-and-mortar stores, it is expected to realize a fair and efficient interview process and reduce the burden on interviewers.

[1107] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1108] Step 1:

[1109] A user logs in to a terminal. The user launches an application on the terminal, enters the required credentials on the login screen, and performs the login operation, completing the authentication. The input is the user's credentials, and the output is an authentication success message.

[1110] Step 2:

[1111] The user selects the desired interview scenario. The user makes a selection from multiple interview scenarios (e.g., technical interview, sales interview, cashier, etc.) displayed on the terminal screen. The input is the user's selection information, and the output is data indicating the selected scenario.

[1112] Step 3:

[1113] The terminal sends the selected scenario information to the server. The terminal converts the user-selected scenario information into a data packet and sends a request to the server. The input is the selected scenario data, and the output is a request to the server.

[1114] Step 4:

[1115] The server requests the generative AI model to generate a list of questions. The server sends a prompt to the generative AI model (e.g., GPT-3) based on the selected scenario, causing it to create a list of questions. The input is the prompt, and the output is the generated list of questions.

[1116] Step 5:

[1117] The server sends the generated question list to the terminal. The server returns the question list obtained from the generative AI model to the terminal. The input is the generated question list, and the output is the data sent to the terminal.

[1118] Step 6:

[1119] The terminal asks the user a list of questions by voice or text. Based on the received list of questions, the terminal uses a speech synthesis system (e.g., gTTS) to present the questions to the user by voice or displays them on the screen as text. The input is the question list data, and the output is voice or text display.

[1120] Step 7:

[1121] The user answers the question. The user answers the question displayed on the terminal by voice or text. The input is the user's voice response to the question, and the output is voice data.

[1122] Step 8:

[1123] The device converts the user's voice into text data. The device's voice recognition system (e.g., speech_recognition) converts the user's voice response into text data. The input is voice data, and the output is text data.

[1124] Step 9:

[1125] The terminal sends the converted text data to the server. The terminal sends the converted text data to the server and makes an evaluation request. The input is the text data, and the output is a request to the server.

[1126] Step 10:

[1127] The server requests the generative AI model to evaluate the answer. The server provides the received text data to the generative AI model and asks for an evaluation. The input is the text data, and the output is the evaluation result and feedback.

[1128] Step 11:

[1129] The server generates evaluation results and feedback and sends them to the terminal. The server evaluates based on evaluation criteria (e.g., consistency, specificity, clarity), creates feedback, and sends it to the terminal. The input is the evaluation result data, and the output is the feedback data.

[1130] Step 12:

[1131] The terminal provides the evaluation results and feedback to the user. The terminal presents the received evaluation results and feedback by voice or displays them on a screen. The input is the feedback data, and the output is voice or text display.

[1132] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1133] This invention is a system that combines an emotion engine and provides practical interview practice via a terminal using a question list generated by a generative AI based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotions into the evaluation.

[1134] Program processing

[1135] 1. The user logs in to the system using a terminal and begins the interview practice.

[1136] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[1137] 2. The terminal sends the user's selection information to the server.

[1138] The request sent to the server includes the user ID and the selected scenario type.

[1139] 3. The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[1140] The generative AI generates a list of questions based on past question examples and best practices.

[1141] 4. The server sends the generated question list to the terminal.

[1142] The list of questions was tailored to the selected interview scenario.

[1143] 5. The device receives the list of questions and asks the user voice questions using the ChatBot function.

[1144] The questions are presented to the user aloud one by one in turn.

[1145] 6. The user answers the question by voice.

[1146] The user's voice response is recorded through the device's microphone.

[1147] 7. The device uses a voice recognition system to convert the user's voice response into text data.

[1148] The converted text data is sent to a server for evaluation.

[1149] 8. The server receives the user's text data and evaluates the content using a generation AI.

[1150] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[1151] 9. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1152] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[1153] 10. The emotion engine sends the recognized emotion data to the device, and the device then sends the data to the server.

[1154] 11. The server uses generated AI to evaluate the user's answers, including emotional data.

[1155] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[1156] 12. The server generates the evaluation results and feedback and sends them to the device.

[1157] Feedback includes user strengths, areas for improvement, and emotional advice.

[1158] 13. The device displays or audibly conveys the evaluation results and feedback to the user.

[1159] Through feedback, users can improve their answers and emotional state and reflect this in their next practice.

[1160] Specific examples

[1161] (Example) For a technical interview:

[1162] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[1163] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[1164] 3. The user responds verbally, "I'm a software engineer..."

[1165] 4. The device converts the voice into text data and sends it to the server.

[1166] 5. The server receives the answer and evaluates it using the generation AI.

[1167] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1168] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[1169] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[1170] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[1171] In this way, users can not only practice for interviews at high quality regardless of time or place, but also understand their emotional state and make comprehensive improvements, including their mental state.

[1172] The processing flow will be explained below.

[1173] Step 1:

[1174] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[1175] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[1176] Step 2:

[1177] The terminal transmits the user's scenario selection information to the server.

[1178] The request sent to the server includes the user ID and the selected scenario type.

[1179] Step 3:

[1180] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[1181] The generative AI generates a list of questions based on past question examples and best practices.

[1182] Step 4:

[1183] The server transmits the generated question list to the terminal.

[1184] The list of questions was tailored to the selected interview scenario.

[1185] Step 5:

[1186] The device receives a list of questions and asks the user aloud using the ChatBot function.

[1187] The questions are presented to the user aloud one by one.

[1188] Step 6:

[1189] The user answers the questions by voice.

[1190] The user's voice response is recorded through the device's microphone.

[1191] Step 7:

[1192] The terminal uses a voice recognition system to convert the user's voice response into text data.

[1193] The converted text data is sent to the server.

[1194] Step 8:

[1195] The server receives the user's text data and evaluates the content using a generation AI.

[1196] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[1197] Step 9:

[1198] The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1199] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[1200] Step 10:

[1201] The emotion engine transmits the recognized emotion data to the terminal, which then transmits the data to the server.

[1202] Emotional data is used as additional information for evaluation.

[1203] Step 11:

[1204] The server uses generated AI to evaluate the user's answers, including emotional data.

[1205] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[1206] Step 12:

[1207] The server generates evaluation results and feedback and sends them to the device.

[1208] Feedback includes specific recommendations based on user strengths, areas for improvement, and feelings.

[1209] Step 13:

[1210] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[1211] Users can review the feedback and use it to improve their answers and emotional state.

[1212] Specific examples

[1213] (Example) For a technical interview:

[1214] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[1215] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[1216] 3. The user responds verbally, "I'm a software engineer..."

[1217] 4. The device converts the voice into text data and sends it to the server.

[1218] 5. The server receives the answer and evaluates it using the generation AI.

[1219] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1220] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[1221] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[1222] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[1223] In this way, users can not only practice for interviews at high quality without being tied down to a specific place or time, but also understand their emotional state and make comprehensive improvements, including their own emotional state.

[1224] Example 2

[1225] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1226] Conventional interview practice systems evaluate only the content of the user's voice responses, making it difficult to provide comprehensive feedback that takes into account the user's emotional state and mental state. This leaves users with no way to effectively overcome feelings of tension and anxiety, making it difficult for them to maximize their performance in an actual interview.

[1227] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1228] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for the terminal to ask the user verbally the questions from the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a voice recognition system, means for transmitting the user's answers to the server, evaluating the content of the answers using the generative AI model, and analyzing the voice data using an emotion engine to recognize the user's emotional state, means for generating and transmitting evaluation results and feedback to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to receive multifaceted evaluation and feedback that takes into account not only the content of their answers but also their emotional state.

[1229] A "generative AI model" is an artificial intelligence that uses machine learning technology to automatically generate a list of questions and answer evaluations based on user input.

[1230] A "terminal" is an electronic device used by a user to practice for an interview, such as a smartphone, tablet, or PC.

[1231] The "server" is a central computer system that processes data for the entire system, generates questionnaires, and executes evaluations.

[1232] An "interview scenario" is a mock interview situation consisting of a number of predetermined questions and situations.

[1233] A "question list" is a set of interview questions generated by a generative AI model.

[1234] A "voice recognition system" is a technology that analyzes a user's voice and converts it into text data.

[1235] "Emotion engine" is an artificial intelligence technology that analyzes a user's voice data to recognize their emotional state.

[1236] The "evaluation results" are a collection of feedback obtained by the generative AI model by analyzing the user's responses and emotional state.

[1237] "Feedback" is information about evaluations and advice regarding the user's interview performance.

[1238] This invention is an interview practice system that combines a generative AI model and an emotion engine to provide practical interview practice based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotional state into the evaluation.

[1239] First, the terminal used by the user is an electronic device such as a smartphone, tablet, or PC, which is used to access and log in to the system. After logging in, the user selects the desired scenario from multiple interview scenarios (e.g., technical interview, sales interview, behavioral interview, etc.). The specific operation of the system is explained below.

[1240] The server requests the generative AI model to generate a list of questions based on the interview scenario selected by the user. For example, if a technical interview scenario is selected, the prompt message is "Please generate a list of questions for the technical interview." The generative AI model references past example questions and best practices to generate a list of questions appropriate to the selected scenario. The generated list of questions is sent from the server to the device in a format such as JSON.

[1241] Based on the received question list, the device uses its ChatBot function to present questions to the user by voice. For example, questions such as "Please introduce yourself" are asked in sequence. When the user responds by voice, the response is recorded through the device's microphone. The device then uses a voice recognition system to convert the voice response into text data in real time and transmits this data to the server.

[1242] The server receives the user's text data and sends the recorded voice data to the emotion engine, which analyzes the voice data and recognizes the user's emotional state (e.g., nervousness, anxiety, confidence, etc.). The recognized emotion data is then sent to the server.

[1243] The server uses a generative AI model to comprehensively evaluate the text data and emotional data, taking into account the consistency, specificity, and clarity of the answers, as well as the user's emotional state. For example, the server may generate an evaluation such as, "Your answers were consistent and specific, but you seemed a little nervous." Once the evaluation results and feedback are generated, they are sent to the device, which then provides them to the user in audio or display format. The user can use this feedback to improve their answers and emotional state and reflect them in their next practice.

[1244] As a concrete example, if a user selects a technical interview scenario, the following prompt is sent to the generative AI model: "Generate a list of questions for a technical interview." A list of questions is generated, and the question "Please introduce yourself" is presented. The user responds, "I'm a software engineer..." and the audio data is recorded and converted into text data. At the same time, the emotion engine generates a recognition result such as "The user is nervous." The generative AI model evaluates the content of the response and the user's emotional state, and generates feedback such as "The response was clear and consistent, but you seem a little nervous."

[1245] As described above, this system combines voice recognition technology and emotion analysis technology to provide users with comprehensive interview practice and feedback.

[1246] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1247] Step 1:

[1248] A user logs into the system using a terminal and selects an interview scenario.

[1249] Specifically, the user enters their ID and password on the login screen and clicks the login button. After logging in, a dashboard is displayed where the user selects the desired interview scenario from among "technical interview" and "sales interview" scenarios. The input is the user ID and password, and the output is information about the selected scenario.

[1250] Step 2:

[1251] The terminal transmits the user's selection information to the server.

[1252] The input is the user ID and the selected scenario information, which the device sends to the server as a request. The specific operation is to send an HTTP request to the server. The output is the request data sent to the server.

[1253] Step 3:

[1254] The server sends a request to the generative AI model to generate a list of questions.

[1255] The input is the user's selection information, and the server sends a prompt to the generative AI model based on this data. For example, the prompt might say, "Please generate a list of questions for a technical interview." Specifically, an API request is sent to the generative AI model. The output is the generated list of questions.

[1256] Step 4:

[1257] The server transmits the generated question list to the terminal.

[1258] The input is a list of questions returned by the generative AI model, and the server sends this list to the device. Specifically, an HTTP response is sent to the device. The output is the list of questions that arrives at the device.

[1259] Step 5:

[1260] The device receives the list of questions and asks the user voice questions using the ChatBot function.

[1261] The input is a list of questions received from the server, and the ChatBot function presents the questions to the user one by one by voice. Specifically, speech synthesis software is used. The output is the voice question presented to the user.

[1262] Step 6:

[1263] The user answers the questions by voice.

[1264] The input is a voice question from the terminal, and the user answers by voice. Specifically, the terminal's microphone records the user's voice. The output is the recorded voice data.

[1265] Step 7:

[1266] The terminal uses a voice recognition system to convert the voice response into text data and transmits it to the server.

[1267] The input is the user's voice data, which the device converts into text data using a voice recognition system. Specifically, voice recognition software is used. The output is the converted text data, which is sent to the server.

[1268] Step 8:

[1269] The server receives the user's text and voice data and analyzes it using a generative AI model and emotion engine.

[1270] The input is the user's text data and voice data, and the server has the generative AI model analyze the text data and the emotion engine analyze the voice data. Specifically, an API request is sent to the generative AI model and the emotion engine. The output is the evaluation result and emotion data.

[1271] Step 9:

[1272] The server generates evaluation results and feedback and sends them to the device.

[1273] The input is the analyzed evaluation results and emotion data, and the server synthesizes these to generate feedback. Specifically, the generative AI model automatically generates the feedback. The output is feedback data, which is sent to the device.

[1274] Step 10:

[1275] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[1276] The input is the feedback data received from the server, which the device displays or speaks to the user. Specific actions include presenting the feedback on a display or through voice synthesis. The output is the feedback provided to the user.

[1277] (Application example 2)

[1278] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1279] Conventional interview practice systems focus on evaluating the content of the user's answers, but are unable to incorporate the user's emotional state into the evaluation. This has led to the problem of being unable to properly manage emotions such as tension and anxiety that users may experience during an actual interview. Furthermore, in the customer service industry, the emotional state of store clerks significantly affects the quality of customer service, so an evaluation system that includes emotion recognition has been in demand.

[1280] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1281] In this invention, the server includes: means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user; means for the terminal to ask the user aloud questions from the list of questions acquired from the server; means for the terminal to convert the user's spoken answers into text data using a voice recognition system; means for transmitting the user's answers to the server, evaluating the content of the answers using the generation AI, generating evaluation results and feedback and transmitting them to the terminal; means for the terminal to transmit recorded voice data to an emotion recognition engine and analyze the user's emotions; and means for transmitting emotional data recognized by the emotion recognition engine to the server and for the generation AI to evaluate the user's answers, including the emotional data. This makes it possible to incorporate not only the content of the user's answers but also their emotional state at the time of answering into the evaluation. This allows users to understand their own emotional state and engage in more practical and comprehensive interview practice and customer service training.

[1282] "User" refers to an individual who uses the system to practice interviews and customer service training.

[1283] An "interview scenario" is a practice scenario that assumes a specific interview situation, and includes different scenarios such as a technical interview and a sales interview.

[1284] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate question lists and assessments.

[1285] A "question list" is a collection of multiple questions generated by generative AI and is constructed based on an interview scenario.

[1286] The "server" refers to a central processing unit that processes user selection information and response data, and generates evaluations and feedback by working with a generative AI and emotion recognition engine.

[1287] A "terminal" is a device operated by a user, including a smartphone or a head-mounted display (HMD).

[1288] "Speech recognition system" refers to technology that converts a user's voice response into text data.

[1289] "Text data" refers to sentence data converted from speech by a speech recognition system.

[1290] An "emotion recognition engine" refers to a system that analyzes a user's emotional state (e.g., tension, anxiety, confidence, etc.) from their voice data.

[1291] "Feedback" refers to advice or evaluation comments provided as a result of evaluating the content of a user's answers or emotional state.

[1292] This invention provides a system for interview practice and customer service training. The system includes a user, a terminal, and a server, and operates and processes as follows.

[1293] Program Overview

[1294] Users log in to the system using a device such as a smartphone or head-mounted display (HMD). The user selects the desired scenario from multiple interview scenarios or customer service scenarios (for example, technical interview, sales interview, product explanation, complaint handling, etc.). This selection information is sent from the device to the server. The server uses this information to have the generation AI generate a list of questions. The generated list of questions is sent to the device, which then asks the user questions by voice. The user then answers by voice, and the answers are converted into text data using the device's voice recognition system, and this text data is sent to the server.

[1295] Data processing and calculation

[1296] The server uses a generative AI to evaluate the text data and analyze it based on criteria such as consistency, specificity, and clarity of the content. The device also sends the recorded voice data to an emotion recognition engine to analyze the user's emotions. This emotion data is also sent to the server. The server then uses the generative AI to perform a comprehensive evaluation, including the emotion data, and generates and sends the evaluation results and feedback to the device. The device then provides this feedback to the user as voice or display.

[1297] Hardware and software used

[1298] Hardware: Smartphone, Head-Mounted Display (HMD)

[1299] Software: Speech recognition system (e.g., Google Speech Recognition), emotion recognition engine (custom or external API), generative AI (for questionnaire generation and evaluation)

[1300] Specific examples

[1301] For example, consider the case where a store clerk launches a customer service training app and selects the "Handling Complaints" scenario. The device retrieves the question from the server, "Please tell us how you would respond if you received a complaint from a customer," and asks the clerk aloud. When the clerk responds, "First, I'll listen to the customer's story to the end...," the content is converted into text data by a voice recognition system. The server receives this text data and evaluates the content and emotional state using a generative AI and an emotion recognition engine. The server then generates feedback such as, "Your opinion is clear, but you seem a little nervous," and the device presents it to the clerk.

[1302] Prompt Sentence Examples

[1303] How do you explain your products to customers?

[1304] What is the most important point when dealing with a complaint?

[1305] By using these concrete examples and prompts, users can understand their own emotional state while conducting practical and comprehensive interview practice and customer service training.

[1306] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1307] Step 1:

[1308] A user logs in to the system from a smartphone or head-mounted display (HMD). The input is the user's authentication information, and the output is authentication that the login was successful. Specifically, a login form is displayed on the terminal screen, and the user enters their ID and password.

[1309] Step 2:

[1310] The user selects the desired scenario from multiple interview or customer service scenarios. The input is the information about the scenario selected by the user, and the output is the ID of the selected scenario. Specifically, a list of scenarios is displayed on the screen, and the user taps to select the scenario they are interested in.

[1311] Step 3:

[1312] The device sends the user's selection information to the server. The input is the selected scenario ID and user ID, and the output is a receipt confirmation from the server. Specifically, the device sends data to the server using an HTTP request.

[1313] Step 4:

[1314] The server makes the generation AI generate a list of questions based on the selected scenario ID. The input is the scenario ID, and the output is the generated list of questions. Specifically, the server calls the generation AI model and inputs a prompt to generate questions based on the scenario.

[1315] Step 5:

[1316] The server sends the generated question list to the device. The input is the question list generated by the generation AI, and the output is the received question list. Specifically, the server sends the question list to the device in an HTTP response.

[1317] Step 6:

[1318] The device receives the list of questions and asks the user aloud. The input is the received list of questions, and the output is the spoken questions to the user. Specifically, the device uses text-to-speech synthesis (TTS) to play back the questions aloud.

[1319] Step 7:

[1320] The user answers the question by voice. The input is the voice question, and the output is the user's voice response. Specifically, the user responds through the device's microphone.

[1321] Step 8:

[1322] The device uses a voice recognition system to convert the user's voice response into text data. The input is the user's voice response, and the output is text data. Specifically, the device uses a voice recognition library to convert voice to text.

[1323] Step 9:

[1324] The device sends text data to the server. The input is text data converted from speech, and the output is confirmation of transmission to the server. Specifically, the device sends the text data to the server via an HTTP request.

[1325] Step 10:

[1326] The server evaluates the text data using generative AI. The input is text data, and the output is the evaluation result. Specifically, the server feeds the text data to the generative AI model and evaluates the consistency, concreteness, and clarity of the content.

[1327] Step 11:

[1328] The device sends recorded voice data to the emotion recognition engine to analyze the user's emotions. The input is recorded voice data and the output is emotion data. Specifically, the device sends the voice data to the emotion recognition engine via an HTTP request.

[1329] Step 12:

[1330] The emotion recognition engine sends the recognized emotion data to the server. The input is the emotion data, and the output is a confirmation of transmission to the server. Specifically, the emotion recognition engine sends the analysis results to the server via an HTTP request.

[1331] Step 13:

[1332] The server uses a generation AI to evaluate the user's answers, including emotional data. The input is text data and emotional data, and the output is a comprehensive evaluation result. Specifically, the server inputs emotional data into the generation AI and performs a comprehensive evaluation.

[1333] Step 14:

[1334] The server generates evaluation results and feedback and sends them to the terminal. The input is the overall evaluation result, and the output is the evaluation result and feedback. Specifically, the server organizes the evaluation results, generates feedback for the user, and sends it to the terminal.

[1335] Step 15:

[1336] The device provides the evaluation results and feedback to the user by voice or display. The input is the evaluation results and feedback, and the output is provided to the user. Specifically, the device uses text-to-speech (TTS) to play back the feedback by voice or displays it on the screen.

[1337] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1338] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1339] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1340] [Fourth embodiment]

[1341] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1342] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1343] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1344] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1345] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1346] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1347] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1348] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1349] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1350] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1351] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1352] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1353] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1354] The present invention is a system that provides practical interview practice via a terminal using a question list generated by a generation AI based on a specific interview scenario selected by the user.

[1355] Program processing

[1356] 1. The user logs in to the system using a terminal and begins the interview practice.

[1357] The user selects the desired scenario from multiple interview scenarios on the terminal screen.

[1358] 2. The terminal sends the user's selection information to the server.

[1359] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[1360] 3. The server uses the generation AI to generate a list of questions corresponding to the scenario and sends it to the device.

[1361] The list of questions corresponds to the interview scenario selected by the user (e.g., technical interview or sales interview).

[1362] 4. The device receives the list of questions and uses the ChatBot function to ask the questions to the user via voice.

[1363] The questions are presented to the user one by one in turn by voice, prompting the user to respond.

[1364] 5. The user answers the question by voice.

[1365] The user's response is converted into text data by a voice recognition system via the device's microphone.

[1366] 6. The device sends the converted text data to the server.

[1367] The server has the generation AI evaluate the received text data.

[1368] 7. The server uses the generated AI to evaluate the user's answers.

[1369] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity of the answer.

[1370] 8. The server generates the evaluation results and feedback and sends them to the device.

[1371] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[1372] 9. The device displays or audibly conveys the evaluation results and feedback to the user.

[1373] Users can improve their answers through feedback and incorporate it into their next practice.

[1374] Specific examples

[1375] (Example) For a technical interview:

[1376] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[1377] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[1378] 3. The user responds verbally, "I'm a software engineer..."

[1379] 4. The device converts the voice into text data and sends it to the server.

[1380] 5. The server receives the answer and uses the generation AI to generate an evaluation result such as "The answer was clear and consistent."

[1381] 6. The device will provide the user with audible feedback on the evaluation results and proceed to the next question.

[1382] In this way, the system allows users to practice interviews at high quality, regardless of location or time. This system is expected to increase users' confidence through practical practice and improve their success rate in interviews.

[1383] The processing flow will be explained below.

[1384] Step 1:

[1385] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[1386] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[1387] Step 2:

[1388] The terminal transmits the user's scenario selection information to the server.

[1389] The request sent to the server includes the user ID and the selected scenario type.

[1390] Step 3:

[1391] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[1392] The generative AI generates a list of questions based on past question examples and best practices.

[1393] Step 4:

[1394] The server transmits the generated question list to the terminal.

[1395] The list of questions was tailored to the selected interview scenario.

[1396] Step 5:

[1397] The device receives a list of questions and asks the user aloud using the ChatBot function.

[1398] The questions are presented to the user aloud one by one in turn.

[1399] Step 6:

[1400] The user answers the questions by voice.

[1401] The user's voice response is recorded through the device's microphone.

[1402] Step 7:

[1403] The terminal uses a voice recognition system to convert the user's voice response into text data.

[1404] The converted text data is sent to a server for evaluation.

[1405] Step 8:

[1406] The server receives the user's text data and evaluates the content using a generation AI.

[1407] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[1408] Step 9:

[1409] The server generates evaluation results and feedback and sends them to the device.

[1410] The feedback includes the user's strengths, areas for improvement, and recommendations for next practice.

[1411] Step 10:

[1412] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[1413] Users can review the feedback and use it to improve their answers.

[1414] Example 1

[1415] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1416] Conventional interview practice systems have the problem that it is difficult for users to practice in conditions similar to a real interview environment, and in particular, they do not provide sufficient individual feedback or evaluation, which limits the user's skill improvement.In addition, existing systems often do not provide comprehensive evaluation based on evaluation criteria such as consistency, concreteness, and clarity of answers required in a real interview.

[1417] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1418] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user verbally the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a speech recognition system, means for evaluating the acquired answers with the generative AI model using prompt sentences created based on the generative AI model, means for transmitting the user's answers to the server, evaluating the contents of the answers with the generative AI model, generating evaluation results and feedback and transmitting them to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to engage in more practical and high-quality interview practice and receive detailed feedback.

[1419] A "user" is an individual or organization that uses the interview practice system to practice for an interview.

[1420] A "terminal" is an electronic device used by a user, such as a computer, smartphone, or tablet, that functions as an interface with the system.

[1421] The "server" is a central computer system that works in conjunction with the terminal to generate a list of questions using a generative AI model, evaluate answers, and provide feedback.

[1422] An "interview scenario" refers to multiple interview situations or themes that a user can select, such as different situations such as a technical interview or a sales interview.

[1423] A "generative AI model" is an artificial intelligence algorithm that generates a list of questions and evaluates answers based on user input data, and primarily uses natural language processing technology.

[1424] A "question list" is a set of questions generated by a generative AI model based on an interview scenario, and is presented to the user.

[1425] A "voice recognition system" is a technology that converts a user's verbal response from voice data into text data.

[1426] A "prompt" is a document that defines the basic instructions and conditions that a generative AI model uses to conduct a question list or evaluation.

[1427] "Feedback" is information that includes the evaluation results of the user's answers and points for improvement, and is provided to the user.

[1428] This invention is a system that uses a generative AI model to generate a list of questions based on a specific interview scenario selected by the user, and provides practical interview practice via a terminal. This system is composed of a server, a terminal, a generative AI model, a speech recognition system, and speech synthesis technology. Detailed embodiments of this system are described below.

[1429] The user logs in to the terminal and starts the interview practice. The user selects the desired scenario from multiple interview scenarios displayed on the terminal. For example, "technical interview" or "sales interview" can be selected. The selected scenario information is sent from the terminal to the server.

[1430] The server sends a request to the generative AI model to generate a list of questions based on the selected scenario. Here, a generative AI model that is strong in natural language processing (e.g., GPT-4) is used. For example, a prompt such as "Please generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10" is sent to the generative AI model. The generative AI model generates a list of questions based on this prompt.

[1431] The server sends the generated question list to the terminal, which then uses voice synthesis technology (e.g., Google Text-to-Speech) to ask the user aloud the questions. The questions are presented to the user one by one in order, prompting them to answer.

[1432] The user answers questions by voice. The user's answers are captured through the device's microphone and converted into text data by a voice recognition system (e.g., Google Cloud Speech-to-Text).

[1433] The device sends the converted text data to a server, which passes the text data to a generative AI model to evaluate the answer. For example, the generative AI model might be provided with a prompt such as, "Please rate the answer based on criteria such as consistency, specificity, and clarity." The generative AI model then analyzes the answer based on the evaluation criteria and generates an evaluation result and feedback.

[1434] The server sends the evaluation results and feedback to the device. The feedback includes the user's strengths and areas for improvement, as well as advice for the next practice. The device then tells the user the evaluation results and feedback by voice or displays them on the screen. This allows the user to improve their answers and reflect them in their next practice.

[1435] As a concrete example, the flow of a technical interview is shown below.

[1436] 1. The user logs in to the terminal and selects "Technical Interview Practice."

[1437] 2. The server sends a prompt to the generative AI model: "Generate a list of questions for a technical interview. The level is intermediate, and the number of questions is 10."

[1438] 3. The generative AI model generates a list of questions, which the server sends to the device.

[1439] 4. The device will ask the user aloud, "Please introduce yourself."

[1440] 5. The user responds verbally, "I'm a software engineer..."

[1441] 6. The device converts the voice into text data and sends it to the server.

[1442] 7. The server evaluates the answer using the generative AI model and generates a rating result that says, "The answer was clear and consistent."

[1443] 8. The device provides the user with audio feedback on the evaluation results.

[1444] In this way, the system allows users to conduct high-quality interview practice regardless of location or time.

[1445] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1446] Step 1:

[1447] A user logs in to a terminal

[1448] Input: User ID and password.

[1449] How it works: The user enters their ID and password on the device's login screen and presses the "Login" button.

[1450] Data processing: The device sends the user ID and password to the server, which checks them against the authentication information stored in the database.

[1451] Output: If authentication is successful, the server generates an authentication token and sends it to the device, which then displays the dashboard screen to the user.

[1452] Step 2:

[1453] The user selects an interview scenario

[1454] Input: Interview scenario (e.g. technical interview, sales interview).

[1455] How it works: The user clicks on the desired scenario from the list displayed on the device screen.

[1456] Data processing: The device sends the selected scenario information to the server.

[1457] Output: The server receives the scenario information.

[1458] Step 3:

[1459] The server sends a question list generation request to the generative AI model.

[1460] Input: Selected scenario information.

[1461] Operation: Based on the selected scenario information, the server sends a request to the generative AI model to generate a list of questions.

[1462] Data processing: The generative AI model generates a list of questions based on the prompt.

[1463] Example prompt: "Generate a list of 10 questions for a technical interview at an intermediate level."

[1464] Output: The generative AI model generates a list of questions and returns it to the server.

[1465] Step 4:

[1466] The server sends the question list to the terminal.

[1467] Input: The list of questions received from the generative AI model.

[1468] Operation: The server converts the generated question list into JSON format and sends it to the terminal.

[1469] Data processing: Format conversion and data transmission on the server.

[1470] Output: The terminal receives the list of questions.

[1471] Step 5:

[1472] The device asks the user a list of questions by voice

[1473] Input: Questionnaire.

[1474] How it works: The device uses speech synthesis technology to read the list of questions to the user.

[1475] Text-to-speech example: Google Text-to-Speech.

[1476] Data processing: Text to speech conversion.

[1477] Output: Presents the question to the user by voice, prompting the user for an answer.

[1478] Step 6:

[1479] The user answers the question

[1480] Input: User speaks the answer.

[1481] How it works: The user speaks the answer to a question into the terminal.

[1482] Data processing: The user's voice is recorded through the device's microphone.

[1483] Output: Recorded audio data.

[1484] Step 7:

[1485] The device converts the voice data into text data.

[1486] Input: Recorded audio data.

[1487] How it works: The device uses a voice recognition system to convert voice data into text data.

[1488] Speech recognition example: Google Cloud Speech-to-Text.

[1489] Data processing: Converting audio data into text data.

[1490] Output: The converted text data.

[1491] Step 8:

[1492] The device sends text data to the server.

[1493] Input: Text data.

[1494] Operation: The terminal converts the converted text data into JSON format and sends it to the server.

[1495] Data processing: format conversion and data transmission.

[1496] Output: The server receives the text data.

[1497] Step 9:

[1498] The server evaluates the answers using a generative AI model

[1499] Input: The converted text data.

[1500] How it works: The server passes the received text data to the generative AI model for evaluation.

[1501] Data processing: The generative AI model analyzes the answers based on the prompt sentence and evaluation criteria.

[1502] Sample prompt: "Please rate your answers based on criteria such as coherence, specificity, and clarity."

[1503] Output: The generative AI model generates evaluation results and feedback and returns them to the server.

[1504] Step 10:

[1505] The server sends the evaluation results and feedback to the device.

[1506] Input: Assessment results and feedback.

[1507] Operation: The server formats the evaluation information obtained from the generative AI model and sends it to the device.

[1508] Data processing: Data formatting and transmission.

[1509] Output: The device receives the evaluation results and feedback.

[1510] Step 11:

[1511] The device provides the evaluation results and feedback to the user.

[1512] Input: Assessment results and feedback.

[1513] What it does: The device will read out the assessment results and feedback aloud or display them on the screen.

[1514] Data processing: Using voice synthesis technology.

[1515] Output: User receives evaluation results and feedback.

[1516] (Application example 1)

[1517] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1518] Conventional interview practice systems have difficulty in providing situations similar to those of a real interview, limiting the effectiveness of user practice. Furthermore, in the actual hiring process, there are issues with the heavy burden on interviewers and the inconsistency of evaluations. Given these circumstances, there is a need for a system that can improve the quality of interview practice and ensure efficient and fair hiring of staff for brick-and-mortar stores.

[1519] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1520] In this invention, the server includes means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for a terminal to ask the user the list of questions acquired from the server by voice or text, means for the terminal to convert the user's voice or text answers into text data using a voice recognition system, means for sending the user's answers to the server, evaluating the answers by the generation AI, generating evaluation results and feedback and sending them to the terminal, and means for the terminal to provide the user with the evaluation results and feedback by voice or display. This makes it possible to provide practical interview practice and enables fair and efficient evaluation even in staff recruitment at physical stores.

[1521] "User" means any individual or organization that uses the System.

[1522] An "interview scenario" is a set of questions designed to fit a specific interview situation.

[1523] "Generative AI" refers to a system that uses artificial intelligence technology to solve problems and generate questions.

[1524] A "question list" refers to a collection of questions provided for a user to answer.

[1525] "Server" refers to the central processing unit responsible for data processing and storage within a system.

[1526] "Terminal" refers to a device (smartphone, tablet, computer, etc.) that is directly operated by a user.

[1527] A "voice recognition system" refers to a technology that converts voice data into text data.

[1528] "Text data" refers to character information converted by a voice recognition system.

[1529] "Evaluation result" refers to the result of the evaluation made by the generation AI on the user's answer.

[1530] "Feedback" refers to advice or suggestions provided to the user based on the evaluation results.

[1531] "Audio or display" refers to the means for providing output information to the user, and includes both audio output and visual display.

[1532] This invention is a system that provides practical interview practice using a question list generated by a generation AI based on an interview scenario selected by the user. A specific embodiment of this system is described below.

[1533] First, a user must log in to a terminal to start the interview practice. Next, the user selects a specific scenario from multiple interview scenarios (e.g., technical interview, sales interview, etc.) through the terminal. This selection information is sent from the terminal to the server.

[1534] The server sends a request to a generative AI to generate a list of questions based on the interview scenario selected by the user, using a generative AI model such as OpenAI's GPT-3, and then sends the generated list of questions to the device.

[1535] The device asks the user questions in order based on the received question list. These questions are provided in the form of voice or text. When the user responds by voice, the device's voice recognition system converts the voice data into text data. Here, Python's speech_recognition library, etc., is used.

[1536] The user's answer, converted into text data, is sent back to the server, where the generation AI evaluates the answer based on criteria such as consistency, specificity, and clarity. The server then uses the generation AI to generate evaluation results and feedback, which are then sent to the device.

[1537] The device provides the user with the evaluation results and feedback, which can be presented as audio or visual, allowing the user to improve their answers and use them for their next practice.

[1538] Specific examples

[1539] For example, in a cashier interview scenario, the following questions might be generated:

[1540] Example prompt sentence:

[1541] Generate interview questions based on the interview scenario "Cashier."

[1542] The resulting questionnaire:

[1543] 1. Please introduce yourself

[1544] 2. Tell us about your experience in treating all customers with kindness.

[1545] 3. How do you manage stress during busy periods?

[1546] In this way, users can conduct high-quality interview practice regardless of location or time. By applying this to staff recruitment in brick-and-mortar stores, it is expected to realize a fair and efficient interview process and reduce the burden on interviewers.

[1547] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1548] Step 1:

[1549] A user logs in to a terminal. The user launches an application on the terminal, enters the required credentials on the login screen, and performs the login operation, completing the authentication. The input is the user's credentials, and the output is an authentication success message.

[1550] Step 2:

[1551] The user selects the desired interview scenario. The user makes a selection from multiple interview scenarios (e.g., technical interview, sales interview, cashier, etc.) displayed on the terminal screen. The input is the user's selection information, and the output is data indicating the selected scenario.

[1552] Step 3:

[1553] The terminal sends the selected scenario information to the server. The terminal converts the user-selected scenario information into a data packet and sends a request to the server. The input is the selected scenario data, and the output is a request to the server.

[1554] Step 4:

[1555] The server requests the generative AI model to generate a list of questions. The server sends a prompt to the generative AI model (e.g., GPT-3) based on the selected scenario, causing it to create a list of questions. The input is the prompt, and the output is the generated list of questions.

[1556] Step 5:

[1557] The server sends the generated question list to the terminal. The server returns the question list obtained from the generative AI model to the terminal. The input is the generated question list, and the output is the data sent to the terminal.

[1558] Step 6:

[1559] The terminal asks the user a list of questions by voice or text. Based on the received list of questions, the terminal uses a speech synthesis system (e.g., gTTS) to present the questions to the user by voice or displays them on the screen as text. The input is the question list data, and the output is voice or text display.

[1560] Step 7:

[1561] The user answers the question. The user answers the question displayed on the terminal by voice or text. The input is the user's voice response to the question, and the output is voice data.

[1562] Step 8:

[1563] The device converts the user's voice into text data. The device's voice recognition system (e.g., speech_recognition) converts the user's voice response into text data. The input is voice data, and the output is text data.

[1564] Step 9:

[1565] The terminal sends the converted text data to the server. The terminal sends the converted text data to the server and makes an evaluation request. The input is the text data, and the output is a request to the server.

[1566] Step 10:

[1567] The server requests the generative AI model to evaluate the answer. The server provides the received text data to the generative AI model and asks for an evaluation. The input is the text data, and the output is the evaluation result and feedback.

[1568] Step 11:

[1569] The server generates evaluation results and feedback and sends them to the terminal. The server evaluates based on evaluation criteria (e.g., consistency, specificity, clarity), creates feedback, and sends it to the terminal. The input is the evaluation result data, and the output is the feedback data.

[1570] Step 12:

[1571] The terminal provides the evaluation results and feedback to the user. The terminal presents the received evaluation results and feedback by voice or displays them on a screen. The input is the feedback data, and the output is voice or text display.

[1572] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1573] This invention is a system that combines an emotion engine and provides practical interview practice via a terminal using a question list generated by a generative AI based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotions into the evaluation.

[1574] Program processing

[1575] 1. The user logs in to the system using a terminal and begins the interview practice.

[1576] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[1577] 2. The terminal sends the user's selection information to the server.

[1578] The request sent to the server includes the user ID and the selected scenario type.

[1579] 3. The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[1580] The generative AI generates a list of questions based on past question examples and best practices.

[1581] 4. The server sends the generated question list to the terminal.

[1582] The list of questions was tailored to the selected interview scenario.

[1583] 5. The device receives the list of questions and asks the user voice questions using the ChatBot function.

[1584] The questions are presented to the user aloud one by one in turn.

[1585] 6. The user answers the question by voice.

[1586] The user's voice response is recorded through the device's microphone.

[1587] 7. The device uses a voice recognition system to convert the user's voice response into text data.

[1588] The converted text data is sent to a server for evaluation.

[1589] 8. The server receives the user's text data and evaluates the content using a generation AI.

[1590] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[1591] 9. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1592] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[1593] 10. The emotion engine sends the recognized emotion data to the device, and the device then sends the data to the server.

[1594] 11. The server uses generated AI to evaluate the user's answers, including emotional data.

[1595] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[1596] 12. The server generates the evaluation results and feedback and sends them to the device.

[1597] Feedback includes user strengths, areas for improvement, and emotional advice.

[1598] 13. The device displays or audibly conveys the evaluation results and feedback to the user.

[1599] Through feedback, users can improve their answers and emotional state and reflect this in their next practice.

[1600] Specific examples

[1601] (Example) For a technical interview:

[1602] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[1603] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[1604] 3. The user responds verbally, "I'm a software engineer..."

[1605] 4. The device converts the voice into text data and sends it to the server.

[1606] 5. The server receives the answer and evaluates it using the generation AI.

[1607] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1608] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[1609] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[1610] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[1611] In this way, users can not only practice for interviews at high quality regardless of time or place, but also understand their emotional state and make comprehensive improvements, including their mental state.

[1612] The processing flow will be explained below.

[1613] Step 1:

[1614] The user logs in to the system using a terminal and accesses a screen to start the interview practice.

[1615] The user selects the desired scenario from multiple interview scenarios (technical interview, sales interview, etc.).

[1616] Step 2:

[1617] The terminal transmits the user's scenario selection information to the server.

[1618] The request sent to the server includes the user ID and the selected scenario type.

[1619] Step 3:

[1620] The server sends a request to the generation AI to generate a list of questions based on the selected scenario.

[1621] The generative AI generates a list of questions based on past question examples and best practices.

[1622] Step 4:

[1623] The server transmits the generated question list to the terminal.

[1624] The list of questions was tailored to the selected interview scenario.

[1625] Step 5:

[1626] The device receives a list of questions and asks the user aloud using the ChatBot function.

[1627] The questions are presented to the user aloud one by one.

[1628] Step 6:

[1629] The user answers the questions by voice.

[1630] The user's voice response is recorded through the device's microphone.

[1631] Step 7:

[1632] The terminal uses a voice recognition system to convert the user's voice response into text data.

[1633] The converted text data is sent to the server.

[1634] Step 8:

[1635] The server receives the user's text data and evaluates the content using a generation AI.

[1636] The generative AI analyzes answers based on criteria such as consistency, specificity, and clarity.

[1637] Step 9:

[1638] The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1639] The emotion engine analyzes the user's emotional state (e.g., tension, anxiety, confidence, etc.) from the voice data.

[1640] Step 10:

[1641] The emotion engine transmits the recognized emotion data to the terminal, which then transmits the data to the server.

[1642] Emotional data is used as additional information for evaluation.

[1643] Step 11:

[1644] The server uses generated AI to evaluate the user's answers, including emotional data.

[1645] The generative AI takes into account emotional data and performs a multifaceted evaluation, including the user's feelings and interview attitude.

[1646] Step 12:

[1647] The server generates evaluation results and feedback and sends them to the device.

[1648] Feedback includes specific recommendations based on user strengths, areas for improvement, and feelings.

[1649] Step 13:

[1650] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[1651] Users can review the feedback and use it to improve their answers and emotional state.

[1652] Specific examples

[1653] (Example) For a technical interview:

[1654] 1. The user selects "Technical Interview Practice" and logs in to the terminal to start the scenario.

[1655] 2. The device retrieves a list of questions from the server, such as "Please introduce yourself," and asks the user verbally.

[1656] 3. The user responds verbally, "I'm a software engineer..."

[1657] 4. The device converts the voice into text data and sends it to the server.

[1658] 5. The server receives the answer and evaluates it using the generation AI.

[1659] 6. The device sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1660] 7. The emotion engine outputs the result "The user is nervous" and sends it to the device.

[1661] 8. The device sends the emotional data to the server, and the AI ​​generates an evaluation result such as, "Your answers were clear and consistent, but you seem a little nervous."

[1662] 9. The server sends the evaluation results to the device, which then provides audio feedback to the user.

[1663] In this way, users can not only practice for interviews at high quality without being tied down to a specific place or time, but also understand their emotional state and make comprehensive improvements, including their own emotional state.

[1664] Example 2

[1665] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1666] Conventional interview practice systems evaluate only the content of the user's voice responses, making it difficult to provide comprehensive feedback that takes into account the user's emotional state and mental state. This leaves users with no way to effectively overcome feelings of tension and anxiety, making it difficult for them to maximize their performance in an actual interview.

[1667] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1668] In this invention, the server includes means for acquiring from the server a list of questions generated by a generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by the user, means for the terminal to ask the user verbally the questions from the list of questions acquired from the server, means for the terminal to convert the user's verbal answers into text data using a voice recognition system, means for transmitting the user's answers to the server, evaluating the content of the answers using the generative AI model, and analyzing the voice data using an emotion engine to recognize the user's emotional state, means for generating and transmitting evaluation results and feedback to the terminal, and means for the terminal to provide the evaluation results and feedback to the user as voice or display. This enables the user to receive multifaceted evaluation and feedback that takes into account not only the content of their answers but also their emotional state.

[1669] A "generative AI model" is an artificial intelligence that uses machine learning technology to automatically generate a list of questions and answer evaluations based on user input.

[1670] A "terminal" is an electronic device used by a user to practice for an interview, such as a smartphone, tablet, or PC.

[1671] The "server" is a central computer system that processes data for the entire system, generates questionnaires, and executes evaluations.

[1672] An "interview scenario" is a mock interview situation consisting of a number of predetermined questions and situations.

[1673] A "question list" is a set of interview questions generated by a generative AI model.

[1674] A "voice recognition system" is a technology that analyzes a user's voice and converts it into text data.

[1675] "Emotion engine" is an artificial intelligence technology that analyzes a user's voice data to recognize their emotional state.

[1676] The "evaluation results" are a collection of feedback obtained by the generative AI model by analyzing the user's responses and emotional state.

[1677] "Feedback" is information about evaluations and advice regarding the user's interview performance.

[1678] This invention is an interview practice system that combines a generative AI model and an emotion engine to provide practical interview practice based on a specific interview scenario selected by the user. The system aims to provide more advanced feedback by incorporating not only the content of the user's voice responses but also their emotional state into the evaluation.

[1679] First, the terminal used by the user is an electronic device such as a smartphone, tablet, or PC, which is used to access and log in to the system. After logging in, the user selects the desired scenario from multiple interview scenarios (e.g., technical interview, sales interview, behavioral interview, etc.). The specific operation of the system is explained below.

[1680] The server requests the generative AI model to generate a list of questions based on the interview scenario selected by the user. For example, if a technical interview scenario is selected, the prompt message is "Please generate a list of questions for the technical interview." The generative AI model references past example questions and best practices to generate a list of questions appropriate to the selected scenario. The generated list of questions is sent from the server to the device in a format such as JSON.

[1681] Based on the received question list, the device uses its ChatBot function to present questions to the user by voice. For example, questions such as "Please introduce yourself" are asked in sequence. When the user responds by voice, the response is recorded through the device's microphone. The device then uses a voice recognition system to convert the voice response into text data in real time and transmits this data to the server.

[1682] The server receives the user's text data and sends the recorded voice data to the emotion engine, which analyzes the voice data and recognizes the user's emotional state (e.g., nervousness, anxiety, confidence, etc.). The recognized emotion data is then sent to the server.

[1683] The server uses a generative AI model to comprehensively evaluate the text data and emotional data, taking into account the consistency, specificity, and clarity of the answers, as well as the user's emotional state. For example, the server may generate an evaluation such as, "Your answers were consistent and specific, but you seemed a little nervous." Once the evaluation results and feedback are generated, they are sent to the device, which then provides them to the user in audio or display format. The user can use this feedback to improve their answers and emotional state and reflect them in their next practice.

[1684] As a concrete example, if a user selects a technical interview scenario, the following prompt is sent to the generative AI model: "Generate a list of questions for a technical interview." A list of questions is generated, and the question "Please introduce yourself" is presented. The user responds, "I'm a software engineer..." and the audio data is recorded and converted into text data. At the same time, the emotion engine generates a recognition result such as "The user is nervous." The generative AI model evaluates the content of the response and the user's emotional state, and generates feedback such as "The response was clear and consistent, but you seem a little nervous."

[1685] As described above, this system combines voice recognition technology and emotion analysis technology to provide users with comprehensive interview practice and feedback.

[1686] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1687] Step 1:

[1688] A user logs into the system using a terminal and selects an interview scenario.

[1689] Specifically, the user enters their ID and password on the login screen and clicks the login button. After logging in, a dashboard is displayed where the user selects the desired interview scenario from among "technical interview" and "sales interview" scenarios. The input is the user ID and password, and the output is information about the selected scenario.

[1690] Step 2:

[1691] The terminal transmits the user's selection information to the server.

[1692] The input is the user ID and the selected scenario information, which the device sends to the server as a request. The specific operation is to send an HTTP request to the server. The output is the request data sent to the server.

[1693] Step 3:

[1694] The server sends a request to the generative AI model to generate a list of questions.

[1695] The input is the user's selection information, and the server sends a prompt to the generative AI model based on this data. For example, the prompt might say, "Please generate a list of questions for a technical interview." Specifically, an API request is sent to the generative AI model. The output is the generated list of questions.

[1696] Step 4:

[1697] The server transmits the generated question list to the terminal.

[1698] The input is a list of questions returned by the generative AI model, and the server sends this list to the device. Specifically, an HTTP response is sent to the device. The output is the list of questions that arrives at the device.

[1699] Step 5:

[1700] The device receives the list of questions and asks the user voice questions using the ChatBot function.

[1701] The input is a list of questions received from the server, and the ChatBot function presents the questions to the user one by one by voice. Specifically, speech synthesis software is used. The output is the voice question presented to the user.

[1702] Step 6:

[1703] The user answers the questions by voice.

[1704] The input is a voice question from the terminal, and the user answers by voice. Specifically, the terminal's microphone records the user's voice. The output is the recorded voice data.

[1705] Step 7:

[1706] The terminal uses a voice recognition system to convert the voice response into text data and transmits it to the server.

[1707] The input is the user's voice data, which the device converts into text data using a voice recognition system. Specifically, voice recognition software is used. The output is the converted text data, which is sent to the server.

[1708] Step 8:

[1709] The server receives the user's text and voice data and analyzes it using a generative AI model and emotion engine.

[1710] The input is the user's text data and voice data, and the server has the generative AI model analyze the text data and the emotion engine analyze the voice data. Specifically, an API request is sent to the generative AI model and the emotion engine. The output is the evaluation result and emotion data.

[1711] Step 9:

[1712] The server generates evaluation results and feedback and sends them to the device.

[1713] The input is the analyzed evaluation results and emotion data, and the server synthesizes these to generate feedback. Specifically, the generative AI model automatically generates the feedback. The output is feedback data, which is sent to the device.

[1714] Step 10:

[1715] The terminal displays or audibly conveys the evaluation results and feedback to the user.

[1716] The input is the feedback data received from the server, which the device displays or speaks to the user. Specific actions include presenting the feedback on a display or through voice synthesis. The output is the feedback provided to the user.

[1717] (Application example 2)

[1718] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1719] Conventional interview practice systems focus on evaluating the content of the user's answers, but are unable to incorporate the user's emotional state into the evaluation. This has led to the problem of being unable to properly manage emotions such as tension and anxiety that users may experience during an actual interview. Furthermore, in the customer service industry, the emotional state of store clerks significantly affects the quality of customer service, so an evaluation system that includes emotion recognition has been in demand.

[1720] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1721] In this invention, the server includes: means for acquiring from the server a list of questions generated by a generation AI based on a specific interview scenario selected from multiple interview scenarios selected by the user; means for the terminal to ask the user aloud questions from the list of questions acquired from the server; means for the terminal to convert the user's spoken answers into text data using a voice recognition system; means for transmitting the user's answers to the server, evaluating the content of the answers using the generation AI, generating evaluation results and feedback and transmitting them to the terminal; means for the terminal to transmit recorded voice data to an emotion recognition engine and analyze the user's emotions; and means for transmitting emotional data recognized by the emotion recognition engine to the server and for the generation AI to evaluate the user's answers, including the emotional data. This makes it possible to incorporate not only the content of the user's answers but also their emotional state at the time of answering into the evaluation. This allows users to understand their own emotional state and engage in more practical and comprehensive interview practice and customer service training.

[1722] "User" refers to an individual who uses the system to practice interviews and customer service training.

[1723] An "interview scenario" is a practice scenario that assumes a specific interview situation, and includes different scenarios such as a technical interview and a sales interview.

[1724] "Generative AI" refers to a system that uses artificial intelligence technology to automatically generate question lists and assessments.

[1725] A "question list" is a collection of multiple questions generated by generative AI and is constructed based on an interview scenario.

[1726] The "server" refers to a central processing unit that processes user selection information and response data, and generates evaluations and feedback by working with a generative AI and emotion recognition engine.

[1727] A "terminal" is a device operated by a user, including a smartphone or a head-mounted display (HMD).

[1728] "Speech recognition system" refers to technology that converts a user's voice response into text data.

[1729] "Text data" refers to sentence data converted from speech by a speech recognition system.

[1730] An "emotion recognition engine" refers to a system that analyzes a user's emotional state (e.g., tension, anxiety, confidence, etc.) from their voice data.

[1731] "Feedback" refers to advice or evaluation comments provided as a result of evaluating the content of a user's answers or emotional state.

[1732] This invention provides a system for interview practice and customer service training. The system includes a user, a terminal, and a server, and operates and processes as follows.

[1733] Program Overview

[1734] Users log in to the system using a device such as a smartphone or head-mounted display (HMD). The user selects the desired scenario from multiple interview scenarios or customer service scenarios (for example, technical interview, sales interview, product explanation, complaint handling, etc.). This selection information is sent from the device to the server. The server uses this information to have the generation AI generate a list of questions. The generated list of questions is sent to the device, which then asks the user questions by voice. The user then answers by voice, and the answers are converted into text data using the device's voice recognition system, and this text data is sent to the server.

[1735] Data processing and calculation

[1736] The server uses a generative AI to evaluate the text data and analyze it based on criteria such as consistency, specificity, and clarity of the content. The device also sends the recorded voice data to an emotion recognition engine to analyze the user's emotions. This emotion data is also sent to the server. The server then uses the generative AI to perform a comprehensive evaluation, including the emotion data, and generates and sends the evaluation results and feedback to the device. The device then provides this feedback to the user as voice or display.

[1737] Hardware and software used

[1738] Hardware: Smartphone, Head-Mounted Display (HMD)

[1739] Software: Speech recognition system (e.g., Google Speech Recognition), emotion recognition engine (custom or external API), generative AI (for questionnaire generation and evaluation)

[1740] Specific examples

[1741] For example, consider the case where a store clerk launches a customer service training app and selects the "Handling Complaints" scenario. The device retrieves the question from the server, "Please tell us how you would respond if you received a complaint from a customer," and asks the clerk aloud. When the clerk responds, "First, I'll listen to the customer's story to the end...," the content is converted into text data by a voice recognition system. The server receives this text data and evaluates the content and emotional state using a generative AI and an emotion recognition engine. The server then generates feedback such as, "Your opinion is clear, but you seem a little nervous," and the device presents it to the clerk.

[1742] Prompt Sentence Examples

[1743] How do you explain your products to customers?

[1744] What is the most important point when dealing with a complaint?

[1745] By using these concrete examples and prompts, users can understand their own emotional state while conducting practical and comprehensive interview practice and customer service training.

[1746] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1747] Step 1:

[1748] A user logs in to the system from a smartphone or head-mounted display (HMD). The input is the user's authentication information, and the output is authentication that the login was successful. Specifically, a login form is displayed on the terminal screen, and the user enters their ID and password.

[1749] Step 2:

[1750] The user selects the desired scenario from multiple interview or customer service scenarios. The input is the information about the scenario selected by the user, and the output is the ID of the selected scenario. Specifically, a list of scenarios is displayed on the screen, and the user taps to select the scenario they are interested in.

[1751] Step 3:

[1752] The device sends the user's selection information to the server. The input is the selected scenario ID and user ID, and the output is a receipt confirmation from the server. Specifically, the device sends data to the server using an HTTP request.

[1753] Step 4:

[1754] The server makes the generation AI generate a list of questions based on the selected scenario ID. The input is the scenario ID, and the output is the generated list of questions. Specifically, the server calls the generation AI model and inputs a prompt to generate questions based on the scenario.

[1755] Step 5:

[1756] The server sends the generated question list to the device. The input is the question list generated by the generation AI, and the output is the received question list. Specifically, the server sends the question list to the device in an HTTP response.

[1757] Step 6:

[1758] The device receives the list of questions and asks the user aloud. The input is the received list of questions, and the output is the spoken questions to the user. Specifically, the device uses text-to-speech synthesis (TTS) to play back the questions aloud.

[1759] Step 7:

[1760] The user answers the question by voice. The input is the voice question, and the output is the user's voice response. Specifically, the user responds through the device's microphone.

[1761] Step 8:

[1762] The device uses a voice recognition system to convert the user's voice response into text data. The input is the user's voice response, and the output is text data. Specifically, the device uses a voice recognition library to convert voice to text.

[1763] Step 9:

[1764] The device sends text data to the server. The input is text data converted from speech, and the output is confirmation of transmission to the server. Specifically, the device sends the text data to the server via an HTTP request.

[1765] Step 10:

[1766] The server evaluates the text data using generative AI. The input is text data, and the output is the evaluation result. Specifically, the server feeds the text data to the generative AI model and evaluates the consistency, concreteness, and clarity of the content.

[1767] Step 11:

[1768] The device sends recorded voice data to the emotion recognition engine to analyze the user's emotions. The input is recorded voice data and the output is emotion data. Specifically, the device sends the voice data to the emotion recognition engine via an HTTP request.

[1769] Step 12:

[1770] The emotion recognition engine sends the recognized emotion data to the server. The input is the emotion data, and the output is a confirmation of transmission to the server. Specifically, the emotion recognition engine sends the analysis results to the server via an HTTP request.

[1771] Step 13:

[1772] The server uses a generation AI to evaluate the user's answers, including emotional data. The input is text data and emotional data, and the output is a comprehensive evaluation result. Specifically, the server inputs emotional data into the generation AI and performs a comprehensive evaluation.

[1773] Step 14:

[1774] The server generates evaluation results and feedback and sends them to the terminal. The input is the overall evaluation result, and the output is the evaluation result and feedback. Specifically, the server organizes the evaluation results, generates feedback for the user, and sends it to the terminal.

[1775] Step 15:

[1776] The device provides the evaluation results and feedback to the user by voice or display. The input is the evaluation results and feedback, and the output is provided to the user. Specifically, the device uses text-to-speech (TTS) to play back the feedback by voice or displays it on the screen.

[1777] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1778] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1779] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1780] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1781] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1782] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1783] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1784] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1785] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1786] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1787] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1788] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1789] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1790] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1791] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1792] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1793] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1794] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1795] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1796] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1797] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1798] The following is further disclosed regarding the above embodiment.

[1799] (Claim 1)

[1800] A means for acquiring from a server a question list generated by the generation AI based on a specific interview scenario selected from multiple interview scenarios selected by a user;

[1801] a means for the terminal to ask the user a question by voice from a list of questions acquired from the server;

[1802] A means for converting the contents of a user's voice response into text data using a voice recognition system in the terminal;

[1803] A means for transmitting the user's answer to a server, evaluating the content of the answer using a generation AI, generating an evaluation result and feedback, and transmitting the evaluation result and feedback to the terminal;

[1804] a means for the terminal to provide the user with the evaluation result and feedback as voice or display;

[1805] A system including:

[1806] (Claim 2)

[1807] The system of claim 1, further comprising a means for the terminal to transmit text data including the user's answer to the server, and the server to evaluate the answer using the generating AI based on evaluation criteria such as consistency, specificity, and clarity.

[1808] (Claim 3)

[1809] The system according to claim 1, further comprising means for the server to generate a list of questions from multiple interview scenarios using a generation AI and transmit the list to the terminal.

[1810] "Example 1"

[1811] (Claim 1)

[1812] A means for acquiring from a server a list of questions generated by the generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by a user;

[1813] a means for the terminal to ask the user a question by voice from a list of questions acquired from the server;

[1814] A means for converting the contents of a user's voice response into text data using a voice recognition system in the terminal;

[1815] A means for evaluating the obtained answer by the generative AI model using a prompt sentence created based on the generative AI model;

[1816] A means for transmitting the user's answer to a server, evaluating the content of the answer using a generative AI model, generating an evaluation result and feedback, and transmitting the evaluation result and feedback to the terminal;

[1817] a means for the terminal to provide the user with the evaluation result and feedback as voice or display;

[1818] A system including:

[1819] (Claim 2)

[1820] The system of claim 1, further comprising means for the terminal to transmit text data including the user's answer to the server, and for the server to evaluate the answer based on evaluation criteria such as consistency, specificity, and clarity using a generative AI model.

[1821] (Claim 3)

[1822] The system of claim 1, wherein the server includes means for generating a list of questions from multiple interview scenarios using a generative AI model and transmitting the list to the terminal.

[1823] "Application Example 1"

[1824] (Claim 1)

[1825] A means for acquiring from a server a question list generated by the generation AI based on a specific interview scenario selected from multiple interview scenarios selected by a user;

[1826] a means for the terminal to ask the user a list of questions acquired from the server by voice or text;

[1827] A means for converting the contents of a user's voice or text response into text data using a voice recognition system in the terminal;

[1828] A means for transmitting the user's answer to a server, evaluating the content of the answer using a generation AI, generating an evaluation result and feedback, and transmitting the evaluation result and feedback to the terminal;

[1829] a means for the terminal to provide the user with the evaluation result and feedback as voice or display;

[1830] A system including:

[1831] (Claim 2)

[1832] The system of claim 1, further comprising a means for the terminal to transmit text data including the user's answer to the server, and the server to evaluate the answer using the generating AI based on evaluation criteria such as consistency, specificity, and clarity.

[1833] (Claim 3)

[1834] The system according to claim 1, further comprising means for the server to generate a list of questions from multiple interview scenarios using a generation AI and transmit the list to the terminal.

[1835] "Example 2: Combining Emotion Engines"

[1836] (Claim 1)

[1837] A means for acquiring from a server a list of questions generated by the generative AI model based on a specific interview scenario selected from multiple interview scenarios selected by a user;

[1838] a means for the terminal to ask the user a question by voice from a list of questions acquired from the server;

[1839] A means for converting the contents of a user's voice response into text data using a voice recognition system in the terminal;

[1840] a means for transmitting the user's response to a server, evaluating the content of the response using a generative AI model, and analyzing the voice data using an emotion engine to recognize the user's emotional state;

[1841] means for generating and transmitting evaluation results and feedback to the terminal;

[1842] a means for the terminal to provide the user with the evaluation result and feedback as voice or display;

[1843] A system including:

[1844] (Claim 2)

[1845] The system of claim 1, further comprising means for the terminal to transmit text data and voice data including the user's answer to the server, and the server to evaluate the answer based on evaluation criteria such as consistency, specificity, and clarity using a generative AI model, and further to analyze the user's emotional state using an emotion engine.

[1846] (Claim 3)

[1847] The system of claim 1, wherein the server includes means for generating a list of questions from multiple interview scenarios using a generative AI model and transmitting the list to the terminal.

[1848] "Application example 2 when combining emotion engines"

[1849] (Claim 1)

[1850] A means for acquiring from a server a question list generated by the generation AI based on a specific interview scenario selected from multiple interview scenarios selected by a user;

[1851] a means for the terminal to ask the user a question by voice from a list of questions acquired from the server;

[1852] A means for converting the contents of a user's voice response into text data using a voice recognition system in the terminal;

[1853] A means for transmitting the user's answer to a server, evaluating the content of the answer using a generation AI, generating an evaluation result and feedback, and transmitting the evaluation result and feedback to the terminal;

[1854] a means for the terminal to provide the user with the evaluation result and feedback as voice or display;

[1855] a means for transmitting the recorded voice data to an emotion recognition engine by the terminal to analyze the user's emotion;

[1856] The emotion recognition engine sends the recognized emotion data to the server, and the generation AI evaluates the user's response, including the emotion data.

[1857] A system including:

[1858] (Claim 2)

[1859] The system of claim 1, further comprising means for the terminal to transmit text data including the user's answer to the server, and the server to evaluate the answer based on consistency, specificity, clarity and emotional data using the generation AI.

[1860] (Claim 3)

[1861] The system according to claim 1, further comprising means for the server to generate a list of questions from multiple interview scenarios using a generation AI and transmit the list to the terminal. [Explanation of symbols]

[1862] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring from a server a question list generated by the generation AI based on a specific interview scenario selected from multiple interview scenarios selected by a user; a means for the terminal to ask the user a question by voice from a list of questions acquired from the server; A means for converting the contents of a user's voice response into text data using a voice recognition system in the terminal; A means for transmitting the user's answer to a server, evaluating the content of the answer using a generation AI, generating an evaluation result and feedback, and transmitting the evaluation result and feedback to the terminal; a means for the terminal to provide the user with the evaluation result and feedback as voice or display; A system including:

2. The system of claim 1, further comprising a means for the terminal to transmit text data including the user's answer to the server, and for the server to evaluate the answer using the generating AI based on evaluation criteria such as consistency, specificity, and clarity.

3. The system according to claim 1, further comprising means for generating a list of questions from a plurality of interview scenarios using a generation AI, and transmitting the list of questions to the terminal.

Images