System

Abstract

A system is provided.SOLUTION: The system includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting a question to the user by voice on the basis of the generated scenario, a voice input means for acquiring the answer of the user in a voice format, and an evaluation means for analyzing the acquired answer and generating the evaluation and feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When preparing for interviews, such as for entrance exams or job hunting, practicing alone has its limitations, and it is difficult to conduct high-quality interview practice in an environment that closely resembles the real thing. Furthermore, there is a lack of means to obtain objective evaluations or specific feedback on one's answers, making it difficult to practice effectively. In this situation, there is a need to provide a system that allows users to conduct high-quality interview practice alone and receive specific feedback. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means: a system including a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question-and-answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, and an evaluation means for analyzing the acquired answers and generating an evaluation and feedback for them. This system allows a user to practice for a high-quality interview alone in an environment that is close to a real interview, and further allows the user to receive an evaluation of their answers and specific feedback.

[0006] "User Information" refers to personal information and related data entered by the user, such as name, desired occupation, purpose of interview, etc.

[0007] The "user interface means" refers to an interface for a user to input information, and includes, for example, a web form or an application input screen.

[0008] "Generation means" refers to an algorithm or program for analyzing user information and generating an appropriate question and answer scenario.

[0009] A "question and answer scenario" refers to a scenario that compiles pairs of interview questions and their expected answers.

[0010] "Voice output means" refers to a speaker or a voice synthesis engine for outputting questions to the user by voice based on the generated scenario.

[0011] "Voice input means" refers to a microphone or voice recording device for obtaining the user's response in voice format.

[0012] The "evaluation means" refers to an algorithm or program for analyzing the acquired voice response and generating evaluation scores and feedback.

[0013] "Feedback" refers to information including evaluation of the user's answer and suggestions for improvement.

[0014] "Display means" refers to a display or audio output device for providing the generated feedback to the user in audio or text form. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[0037] 1. Initial settings and user information entry

[0038] A user first uses a user interface to access the system and enter their profile information, including basic information such as their name, desired career, and purpose of the interview. The terminal receives this information and transmits it to the server.

[0039] 2. Generating interview scenarios

[0040] The server uses a generation means to generate an appropriate Q&A scenario based on the received user information. This generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of a combination of questions and expected answers.

[0041] 3. Start practicing for the interview

[0042] Based on the received scenario, the device uses the voice output means to ask the user a question by voice. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer by using the voice input means. In this way, the question and answer process progresses.

[0043] 4. Sending and analyzing response data

[0044] The device sends the recorded audio data to a server, which then uses an evaluation tool to analyze the responses, including the appropriateness of the content, language, attitude, and pronunciation. Each element is scored and feedback is generated.

[0045] 5. Providing Feedback

[0046] The server sends the generated evaluation score and feedback to the device. The device receives the evaluation score and feedback and provides the user with feedback in the form of voice or text. For example, the device may display feedback such as, "Your self-introduction is not specific enough. Please add more specific examples."

[0047] Specific examples

[0048] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question may be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[0049] This system allows users to practice interviews in a realistic environment and also provides objective evaluations and specific areas for improvement, allowing users to fully prepare for the actual interview.

[0050] The processing flow will be explained below.

[0051] Step 1:

[0052] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal receives this information, converts it into data packets, and sends them to the server.

[0053] Step 2:

[0054] The server receives and analyzes the received user information. It then activates a generation means to generate an appropriate question and answer scenario based on the user's desired career and the purpose of the interview. The generation means selects questions using a past database and machine learning algorithms and generates a scenario. The generated scenario consists of pairs of questions and their expected answers. The server converts this scenario into data packets and sends them back to the terminal.

[0055] Step 3:

[0056] The terminal analyzes the received scenario data and outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the user's answer using the voice input means.

[0057] Step 4:

[0058] The terminal converts the recorded voice data into data packets and sends them to the server, where the user's voice response reaches the server.

[0059] Step 5:

[0060] The server launches an evaluation means to analyze the received voice data. The evaluation means uses a speech recognition algorithm to convert the voice data into text, and analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. The evaluation means assigns an evaluation score as a result of the analysis, and also generates feedback on the user's weaknesses and areas for improvement. This information is compiled as feedback data.

[0061] Step 6:

[0062] The server converts the generated evaluation score and feedback data into a data packet and sends it to the terminal. The terminal receives this data and provides the user with feedback in the form of voice or text. For example, the server might say, "Your self-introduction is not specific enough. Please add more specific examples."

[0063] Step 7:

[0064] The user can refine their answer based on the provided feedback, practice again using the refined answer, and the device will record a new answer and repeat the process from step 4 onwards.

[0065] Through these steps, users can engage in high-quality interview practice in a realistic environment and improve their skills through objective evaluation and specific feedback.

[0066] Example 1

[0067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0068] Traditional interview practice requires the time and effort of others to obtain third-party evaluation, making it difficult to conduct high-quality practice alone. Furthermore, there is a lack of a system for receiving immediate, objective, and specific feedback. This makes it difficult to maintain motivation for independent interview practice and achieve effective self-improvement.

[0069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0070] In this invention, the server includes a generation means that analyzes user information and generates an appropriate Q&A scenario using a generative AI model, an evaluation means that analyzes the acquired answers and generates an evaluation and feedback thereof, and a display means that provides the generated feedback to the user, thereby enabling the user to practice for a high-quality interview on their own and receive immediate, objective, and specific feedback.

[0071] "User interface means" refers to an interface that allows a user to input information into a system. Examples include web forms and application input screens.

[0072] "Communication Methods" refers to the ability to send and receive data between a device and a server, including HTTP requests over the Internet and other digital communication protocols.

[0073] "Generation means" refers to the function of analyzing user information and generating appropriate Q&A scenarios using a generative AI model, which uses past data and specific algorithms.

[0074] "Voice output means" refers to a function that outputs questions to the user by voice based on the generated scenario. Specific examples include speakers and voice synthesis technology.

[0075] "Voice input means" refers to a function for acquiring and recording a user's response in voice format. A specific example is a microphone.

[0076] "Evaluation means" refers to the function that analyzes the answers obtained and generates evaluations and feedback, using natural language processing and voice analysis technologies.

[0077] The "display means" refers to a function for providing the generated feedback to the user. Specific examples include a method of displaying text on a display or a method of playing back the feedback by voice.

[0078] The present invention is a system that allows users to practice interviews on their own and receive specific feedback. This system is implemented mainly using a terminal and a server.

[0079] 1. Initial settings and user information entry

[0080] First, a user accesses the system using a terminal. Using a user interface, the user inputs his / her profile information (e.g., name, desired career, purpose of interview, etc.). This information is sent from the terminal to the server.

[0081] 2. Generating interview scenarios

[0082] The server analyzes the received user information and generates an appropriate question and answer scenario using a generative AI model. The generative AI model operates based on past data and specific algorithms to select questions that match the user's desired career and the purpose of the interview. This scenario consists of a combination of questions and expected answers, and the generated scenario is sent to the device.

[0083] 3. Start practicing for the interview

[0084] The terminal uses the voice output means to ask the user a question based on the received scenario. For example, the question "Please introduce yourself" is output as a voice. The user answers this question by voice, and this answer is recorded by the voice input means.

[0085] 4. Sending and analyzing response data

[0086] The recorded voice data is sent from the device to a server. The server receives this data and uses evaluation tools to analyze the answers from various angles. Analysis criteria include appropriateness of content, language, attitude, pronunciation, etc. The server generates feedback based on these criteria.

[0087] 5. Providing Feedback

[0088] The server sends the generated feedback to the device, which then receives it and provides the user with feedback in the form of voice or text. For example, the feedback might say, "Your self-introduction is not specific enough. Please add more specific examples."

[0089] Specific examples

[0090] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" in their profile information, which is then sent to the server. Based on this information, the server generates a scenario containing questions related to sales positions and expected answers to those questions. For example, the server generates a question such as "Tell us about your work experience to date."

[0091] The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server, which analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[0092] Prompt Sentence Examples

[0093] Below are some example prompts using a generative AI model:

[0094] "Generate an interview scenario for a sales position. User profile information is as follows: name, desired career is sales, purpose of interview is job hunting."

[0095] This system allows users to practice interviews in a realistic environment and receive objective and specific feedback, helping them to fully prepare for the actual interview.

[0096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0097] Step 1:

[0098] A user accesses the system and uses the user interface to input their profile information (name, desired career, purpose of the interview), which is then acquired by the terminal and transmitted to the server via the communication means.

[0099] Input: User profile information

[0100] Data processing and calculation: Convert information into JSON format

[0101] Output: User information sent to the server

[0102] Step 2:

[0103] The server analyzes the received user information and generates an appropriate Q&A scenario using a generative AI model. The generative AI model uses past data and algorithms to create a set of questions and answers that are tailored to the user's desired career and interview objectives. The generated scenario is sent to the device via communication means.

[0104] Input: User information sent to the server

[0105] Data processing and calculation: Creating Q&A scenarios using generative AI models

[0106] Output: Sends the Q&A scenario (JSON format) to the terminal.

[0107] Step 3:

[0108] Based on the received scenario, the terminal uses the audio output means to ask the user a question by voice. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the answer by using the audio input means.

[0109] Input: Question and Answer scenario received by terminal

[0110] Data processing and calculation: Voice output of questions and answers, recording of user voice responses

[0111] Output: Recorded user voice response

[0112] Step 4:

[0113] The terminal encodes the recorded voice data using BASE64 and transmits it to the server via a communication means.

[0114] Input: Recorded user voice response

[0115] Data processing and calculation: BASE64 encoding of audio data

[0116] Output: The encoded audio data sent to the server.

[0117] Step 5:

[0118] The server decodes the received voice data and converts it into text using speech recognition technology. It then uses natural language processing (NLP) technology to evaluate the answers and generate feedback. Evaluation criteria include appropriateness of content, language, attitude, pronunciation, etc.

[0119] Input: Received encoded audio data

[0120] Data processing and calculation: Decoding voice data, speech recognition, analysis using NLP

[0121] Output: Generated feedback (in text format)

[0122] Step 6:

[0123] The server sends the generated feedback to the terminal. The terminal receives the feedback and provides it to the user in the form of voice or text via a display means. For example, a message may be played over the speaker saying, "Your self-introduction is not specific enough. Please add more specific examples."

[0124] Input: Feedback sent by the server

[0125] Data processing and calculation: Text-based feedback is displayed as voice or text

[0126] Output: Feedback provided to the user (audio or text)

[0127] This series of processes allows users to conduct high-quality interview practice on their own and receive objective and specific feedback.

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] With conventional interview practice systems, it was difficult for users to conduct high-quality interview practice on their own, and it was also difficult to obtain specific feedback. In particular, when store staff practiced customer service interviews, there was a lack of a means to objectively evaluate the results of the practice and provide specific areas for improvement.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice synthesis means for outputting the generated scenario as voice, a voice recognition means for acquiring the user's answers in voice format, recording them and converting them into text format, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback, and a display means for providing the evaluation and feedback generated by the evaluation means to the user in voice or text format. This enables a user to conduct high-quality interview practice even by themselves and to receive specific feedback even in customer service training at a physical store.

[0133] "User interface means" refers to an interface through which a user inputs their information into the system.

[0134] The "generation means" is a means for analyzing user information and generating an appropriate question and answer scenario based on that information.

[0135] The "audio output means" is a means for outputting the generated scenario to the user by audio.

[0136] The "voice input means" is a means for obtaining the user's response in voice form.

[0137] The "evaluation means" is a means for analyzing the acquired voice data and generating an evaluation or feedback.

[0138] The "display means" is a means for providing the evaluation and feedback generated by the evaluation means to the user in audio or text format.

[0139] The "voice synthesis means" is a means for converting a text-format scenario into voice and outputting it.

[0140] A "voice recognition means" is a means for recording a user's response in voice form and converting it into text form.

[0141]

[0142] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[0143] User Interface Means

[0144] The user first enters their profile information using the system's user interface means. At this stage, the user enters basic information such as their name, desired career, and purpose of the interview. This information is received by the terminal and transmitted to the server.

[0145] Scenario generation method

[0146] The server uses a generation means to generate an appropriate question and answer scenario based on the received user information. This generation means selects questions that are in line with the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario includes a combination of questions and expected answers.

[0147] Voice output means and voice synthesis means

[0148] The terminal uses the voice output means based on the generated scenario and utilizes the voice synthesis means to ask the user a question by voice, for example, "Please introduce yourself."

[0149] Voice input and voice recognition methods

[0150] The user answers the questions by voice, and the terminal records the answers using a voice input means. The recorded voice data is converted into text format by a voice recognition means.

[0151] Analysis of response data

[0152] The device sends the recorded and transcribed voice data to the server, which then uses an evaluation tool to analyze the answers, including the appropriateness of the content, the language used, the attitude, and the pronunciation, and assigns a score for each element to generate feedback.

[0153] Providing feedback

[0154] The server transmits the generated evaluation score and feedback to the terminal. The terminal receives the evaluation score and feedback and provides the feedback to the user in the form of voice or text using a display means. For example, feedback such as "Your self-introduction is not specific enough. Please add more specific examples" is displayed as voice or text.

[0155] Specific examples

[0156] Let's say the user is aiming to become a store staff member, and the purpose of the interview is job hunting. The user enters "store staff member" and "job hunting" as their profile information. The server generates a scenario containing questions related to store staff members and expected answers. For example, a question might be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud with "I worked as a store staff member for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides this to the user via the device.

[0157] Prompt Sentence Examples

[0158] Example prompt: "Generate an interview scenario for a store associate."

[0159] This allows users to practice interviews in an environment that is close to the real thing, and also provides objective evaluations and specific areas for improvement.The system utilizes natural language processing and speech recognition technology to provide an advanced interview practice environment.

[0160] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0161] Step 1:

[0162] The user inputs his / her profile information using the user interface means. At this stage, the user inputs basic information such as his / her name, desired career, and purpose of the interview. The input data is transmitted to the terminal.

[0163] Step 2:

[0164] The device sends the received user information to the server, which analyzes the user information and uses a generation method to generate an appropriate question and answer scenario. In doing so, the server utilizes past data and algorithms to select questions that are in line with the user's desired career and the purpose of the interview.

[0165] Step 3:

[0166] The scenario generated by the server is sent to the terminal. The terminal uses a speech synthesis means to convert the generated scenario into a voice format. An audio file is generated, and the questions in the scenario are output to the user by voice.

[0167] Step 4:

[0168] The user answers questions by voice, and the terminal records the user's answers using a voice input means. This voice data is converted into text format by a voice recognition means.

[0169] Step 5:

[0170] The device then transmits the converted response data to the server, which then uses an evaluation tool to analyze the response. The analysis includes the appropriateness of the response, the language used, the attitude, and the pronunciation, and assigns a score to each element.

[0171] Step 6:

[0172] The server generates an evaluation and feedback based on the analysis results, and the generated feedback and evaluation points are sent to the device.

[0173] Step 7:

[0174] The terminal uses the display means to provide the user with evaluation and feedback in the form of voice or text, for example, by displaying specific advice such as "Your answer is not specific enough. Please add more specific examples."

[0175] Step 8:

[0176] Users can receive feedback and review their practice interviews, providing specific improvements to incorporate into their next practice.

[0177] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0178] This system allows users to practice interviews independently and receive specific feedback. In particular, it combines an emotion engine that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, enabling a more comprehensive interview evaluation. This system is comprised of the following components:

[0179] 1. Initial settings and user information entry

[0180] A user first uses a user interface means to access the system and enter their profile information. The user enters basic information such as their name, desired career, and purpose of the interview. The terminal receives this information, converts it into data packets, and sends them to the server.

[0181] 2. Generating interview scenarios

[0182] Based on the received user information, the server uses a generation means to generate an appropriate question and answer scenario. The generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of pairs of questions and expected answers. The server converts this scenario into data packets and sends them to the terminal.

[0183] 3. Start practicing for the interview

[0184] Based on the received scenario, the device outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer using the voice input means. This is how the question and answer process progresses.

[0185] 4. Sending and analyzing response data

[0186] The device converts the recorded voice data into data packets and sends them to the server. The server receives this data and analyzes the answers using an evaluation tool. At the same time, an emotion engine is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice responses.

[0187] 5. Evaluation and feedback generation

[0188] The server analyzes the voice data through the evaluation means and evaluates the appropriateness of the response, pronunciation, wording, attitude, etc. Furthermore, it generates an evaluation score taking into account the user's emotional state recognized by the emotion engine. This allows for a more comprehensive evaluation. The generated evaluation score and feedback are packaged as a data packet and sent to the terminal.

[0189] 6. Providing Feedback

[0190] The device analyzes the received evaluation scores and feedback data and provides the user with feedback in the form of voice or text. For example, in addition to feedback such as "Your self-introduction is not specific enough. Please add more specific examples," it also provides emotion-based advice such as "You seem a little nervous. Try to relax."

[0191] Specific examples

[0192] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question might be generated: "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud: "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and further provides advice based on the analysis results of the emotion engine: "You seem tense in your voice. Take a deep breath and try to relax."

[0193] This system allows users to practice interviews in a realistic environment and receive comprehensive evaluations, specific suggestions for improvement, and emotional support, helping them to fully prepare for the actual interview.

[0194] The processing flow will be explained below.

[0195] Step 1:

[0196] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal collects this information, converts it into data packets, and sends them to the server.

[0197] Step 2:

[0198] The server receives and analyzes the received user information. Based on the analyzed information, it activates a generation means to generate an appropriate question and answer scenario. The generation means uses algorithms and pattern recognition technology to select questions that are in line with the user's desired career and the purpose of the interview, and generates a scenario. The generated scenario is sent to the terminal as a data packet.

[0199] Step 3:

[0200] The terminal analyzes the received scenario data and asks the user questions aloud using the voice output means. For example, a question such as "Please tell us about your work experience to date" is played aloud. The user answers the questions aloud, and the terminal records the answers using the voice input means.

[0201] Step 4:

[0202] The terminal converts the recorded user's answer data into a data packet and sends it to the server, whereupon the user's voice answer reaches the server.

[0203] Step 5:

[0204] The server activates an evaluation means to analyze the received voice data, which converts the voice data into text using a speech recognition algorithm and analyzes the appropriateness of the content, pronunciation, wording, attitude, etc.

[0205] Step 6:

[0206] The server activates an emotion engine to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. The emotion engine analyzes the tone, speed, and pitch of the voice to identify the emotion.

[0207] Step 7:

[0208] The evaluation means generates a comprehensive evaluation score based on the analysis results of the obtained answers and the emotional state recognized by the emotion engine, and simultaneously generates specific feedback and emotion-based advice based on the user's evaluation.

[0209] Step 8:

[0210] The server converts the generated evaluation score and feedback data into a data packet and transmits it to the terminal.

[0211] Step 9:

[0212] The device analyzes the evaluation scores and feedback data it receives and provides the user with feedback in the form of voice or text. For example, it may provide feedback such as, "It would be good to mention your role and specific achievements," or emotional advice such as, "You sound tense. Take a deep breath and relax."

[0213] Step 10:

[0214] The user can use the provided feedback and advice to improve their answer next time and practice again. The device will record a new voice answer and repeat the process from step 4 onwards.

[0215] Through this series of steps, users can practice interviews in a realistic environment and receive comprehensive evaluations and feedback that take into account their emotional state, helping them to fully prepare for the actual interview.

[0216] Example 2

[0217] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] In modern interview practice, it is difficult to achieve high-quality training, especially when done alone. Furthermore, conventional interview practice systems often do not provide specific feedback on the user's answers or evaluations that reflect the user's emotional state. As a result, users are unable to accurately identify their own problems and are unable to make effective improvements.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0220] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for recognizing emotions from the user's voice and generating an evaluation and feedback based on the emotions. This allows the user to effectively practice for an advanced interview even on their own, and to receive a comprehensive evaluation that reflects specific feedback and their emotional state.

[0221] "User interface means" means a device or software that allows a user to access the system and input required information.

[0222] The "generator" is a device or software that creates an appropriate question and answer scenario based on information provided by the user.

[0223] The "audio output means" is a device or software for outputting questions to the user by voice based on the generated scenario.

[0224] "Voice input means" refers to a device or software for obtaining a user's voice response.

[0225] The "evaluation means" is a device or software that analyzes the user's answers obtained, evaluates their appropriateness, pronunciation, wording, attitude, etc., and generates feedback.

[0226] The "emotion analysis means" is a device or software for recognizing emotions from the user's voice and generating evaluations and feedback based on those emotions.

[0227] A "display means" is a device or software for providing the feedback generated by the evaluation means to the user in audio or text form.

[0228] This system allows a single user to practice interviews with high quality and receive specific feedback. In particular, by combining it with an emotion analysis tool that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, a more comprehensive interview evaluation can be achieved. This system is composed of the following elements:

[0229] Initial settings and user information entry

[0230] First, a user accesses the system using a terminal connected to the Internet. They enter their profile information (such as their name, desired occupation, and purpose of the interview) through a dedicated user interface. For example, they enter information such as "I'm looking for a sales position, and the purpose of the interview is job hunting." The entered information is converted into data packets by the terminal and sent to the server.

[0231] Interview scenario generation

[0232] The server analyzes the user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario that matches the user's desired career and the purpose of the interview. For example, a question such as "Please tell us about your work experience to date" is generated as a question about a sales position. The generated scenario is converted into a data packet and sent to the device.

[0233] Interview practice begins

[0234] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the audio output means, it outputs questions based on the scenario by voice. For example, a question such as "Please introduce yourself" is presented to the user by voice. The user answers this question by voice, and the terminal records the answer using the audio input means.

[0235] Sending and analyzing response data

[0236] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data, converts the voice content into text using a natural language processing algorithm, and analyzes the response. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (tension, joy, confidence, etc.) from the voice response.

[0237] Generate ratings and feedback

[0238] The server analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. using evaluation means. Furthermore, the user's emotional state, as recognized by emotion analysis means, is also included in the evaluation. This generates an overall evaluation score and specific feedback. For example, the server may provide feedback such as "It would be good to add a specific anecdote" or emotion-based advice such as "Your voice sounds tense, so try relaxing."

[0239] Providing feedback

[0240] The server converts the generated evaluation score and feedback into a data packet and sends it to the device. The device analyzes the received feedback data and provides the user with feedback in voice or text format. For example, the device might provide advice based on emotions, such as, "Your self-introduction lacks specificity. Please add more specific examples," or, "You seem a little nervous. Take a deep breath and try to relax."

[0241] Specific examples

[0242] For example, let's take a specific example where a user is aiming for a sales position. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers to those questions. The generated scenario includes the question, "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as, "It would be good to mention specific roles and achievements." Furthermore, based on the analysis results of the emotion engine, advice is provided such as, "Your voice sounds tense. Take a deep breath and try to relax."

[0243] As described above, the present invention allows users to effectively practice for interviews on their own and receive comprehensive evaluations, specific feedback, and emotional support, thereby enabling users to fully prepare for the actual interview.

[0244] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0245] Step 1:

[0246] Initial settings and user information entry

[0247] Users access the system using a terminal connected to the Internet. Through a dedicated user interface, they input profile information such as their name, desired occupation, and purpose of the interview (e.g., sales, job hunting). The terminal converts the input information into data packets and sends them to the server, which then receives the user's profile information.

[0248] Input: User profile information (name, occupation, purpose of interview)

[0249] Output: Data packet (including user information)

[0250] Step 2:

[0251] Interview scenario generation

[0252] The server receives and analyzes user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario tailored to the user's desired career and the purpose of the interview (e.g., questions about sales positions). The generated scenario is converted into a data packet and sent to the device. The server selects questions based on past data and algorithms, and also creates sample answers.

[0253] Input: Data packet (user information)

[0254] Output: Data packet (generated scenario)

[0255] Step 3:

[0256] Interview practice begins

[0257] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the voice output means, it outputs a question based on the scenario to the user by voice (e.g., "Please introduce yourself"). The user answers this question by voice, and the terminal records the answer using the voice input means. As a result, the user's speech is saved in the system.

[0258] Input: Data packet (scenario)

[0259] Output: Audio data (user's answer)

[0260] Step 4:

[0261] Sending and analyzing response data

[0262] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data and converts the voice content into text using a natural language processing algorithm, which then analyzes it. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. This allows both the content of the response and the user's emotional state to be analyzed.

[0263] Input: Voice data (user response)

[0264] Output: Text data (analyzed answers), emotion data

[0265] Step 5:

[0266] Generate ratings and feedback

[0267] The server uses an evaluation tool to analyze the appropriateness of the response, pronunciation, wording, and attitude based on the text data. Furthermore, the server also incorporates the user's emotional state, as recognized by the emotion analysis tool, into the evaluation. This generates an overall evaluation score and specific feedback (e.g., adding a specific anecdote would be helpful). The generated evaluation score and feedback are converted into a data packet and sent to the device.

[0268] Input: Text data (answer content), emotion data

[0269] Output: Data packet (ratings and feedback)

[0270] Step 6:

[0271] Providing feedback

[0272] The device analyzes the received data packets and provides feedback to the user in voice or text format (e.g., your self-introduction is not specific enough). In addition to specific areas for improvement, emotion-based advice is also provided (e.g., take a deep breath to relax). This allows users to receive practical feedback.

[0273] Input: Data packet (ratings and feedback)

[0274] Output: Feedback (audio or text)

[0275] (Application example 2)

[0276] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0277] Conventional interview practice systems have difficulty understanding the user's own emotions and state of tension, and providing appropriate feedback and advice. Furthermore, the training process for new staff at logistics centers can sometimes delay the acquisition of skills that will enable them to be immediately effective. This can lead to issues such as reduced efficiency throughout the center.

[0278] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0279] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for analyzing the user's emotional state and providing an evaluation and feedback based on the analysis results. This makes it possible to provide specific advice based on the user's emotional state, allowing new staff at a logistics center to effectively hone their skills and become immediately effective.

[0280] 1. "User interface means" refers to an interface through which a user inputs or obtains information into a system.

[0281] 2. "Generation means" refers to the means for analyzing input user information and creating an appropriate question and answer scenario based on that information.

[0282] 3. "Voice output means" refers to a means for outputting questions to the user by voice based on the generated scenario.

[0283] 4. "Voice input means" refers to the means for obtaining a user's response in voice form and inputting it into the system.

[0284] 5. "Evaluation means" refers to the means for analyzing a user's voice response, evaluating its content and appropriateness, and generating feedback.

[0285] 6. "Emotion analysis means" refers to a means for analyzing the emotional state of a user from their voice response and providing evaluation and feedback based on the analysis results.

[0286] 7. "Feedback" refers to information including evaluation of the user's answers, suggestions for improvement, and advice.

[0287] 8. "Logistics center" refers to a facility where goods and materials are stored, sorted, and prepared for delivery.

[0288] 9. "New staff" refers to employees newly hired at the logistics center.

[0289] 10. "Immediately effective" refers to having techniques and skills that can be immediately used in actual work.

[0290] A system for realizing this application example will be described in detail below.

[0291] System configuration

[0292] First, the interview training system for new staff at the logistics center consists of the following main components:

[0293] 1. User Interface Methods

[0294] 2. Generation means

[0295] 3. Audio output means

[0296] 4. Voice input method

[0297] 5. Evaluation Methods

[0298] 6. Emotion analysis method

[0299] User Interface Means

[0300] The user interface means is an interface that allows staff to access the system and input basic information such as their name, the work they will be responsible for, past experience, etc. For example, a smartphone or PC application can be used.

[0301] generation means

[0302] The generation method analyzes the input staff information and creates appropriate Q&A scenarios based on that information. This method uses past data and algorithms. Specifically, it uses natural language generation technology to automatically generate questions related to the staff member's work.

[0303] Audio output means

[0304] The voice output means outputs a question to the staff member by voice based on the generated scenario. For example, the voice output function of a smartphone can be used to voice the question, "Do you have experience in inventory management?"

[0305] Voice input means

[0306] The voice input method receives the staff's response in voice format and inputs it into the system. The voice data is recorded as is and sent to the server. For example, the voice is converted into text using the Google Speech-to-Text API.

[0307] Evaluation methods

[0308] The evaluation means analyzes the answer data acquired on the server side and evaluates its content and appropriateness. The content, pronunciation, and wording of the answer are evaluated here. Feedback is also generated based on the evaluation results.

[0309] Emotion analysis means

[0310] The emotion analysis means analyzes the emotional state (e.g., nervousness, confidence, etc.) of staff members' voice responses and provides evaluation and feedback based on the analysis results. For example, IBM Watson Tone Analyzer API is used to recognize emotions from voice data.

[0311] Program processing

[0312] In this system, a staff member first inputs basic information using a smartphone or PC. The server receives that information, and a generation means generates an appropriate question-and-answer scenario. Based on the generated scenario, a voice output means outputs questions to the staff member by voice, and the staff member answers by voice. A voice input means acquires the answers and sends them to the server. An evaluation means then analyzes the answers, and emotions are also analyzed by an emotion analysis means. Finally, feedback is generated based on the results of the evaluation and emotion analysis and provided to the staff member.

[0313] Specific examples

[0314] For example, if a new staff member at a distribution center is in charge of "inventory management," the following questions might be generated:

[0315] 1. Enter staff information

[0316] Name: Taro

[0317] Responsibilities: Inventory management

[0318] Experience: 2 years of inventory management experience

[0319] 2. Generated Questions

[0320] "Explain the main steps in inventory management."

[0321] "Tell us about a difficult situation you've had with inventory management in the past and how you dealt with it."

[0322] 3. Staff responses and feedback

[0323] Staff members answer questions and the answers are sent to the server.

[0324] The server analyzes the answers and generates feedback such as "Please explain the specific steps in more detail" and emotion-based advice such as "Your voice sounds tense. Relax."

[0325] Prompt Sentence Examples

[0326] Example prompts to be input to the generative AI model:

[0327] "Encourage candidates to describe in detail their inventory management experience. See if they use specific examples to explain the procedures and their role. Also, assess their emotional state (tension, confidence, etc.) and provide feedback accordingly."

[0328] In this way, the present invention allows new staff at a logistics center to effectively hone their skills so that they can be immediately put to work through practical interview training.

[0329] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0330] Step 1:

[0331] A user uses a smartphone or a PC to input basic information (such as name, work to be performed, past experience, etc.) through a user interface. The input information is converted into a data packet and sent to the server.

[0332] Input: Basic information such as name, expected duties, past experience, etc.

[0333] Output: Basic information about the data packet format.

[0334] Specific behavior: A user accesses the application, enters information into the designated fields, and then presses the "Submit" button to send the information to the server.

[0335] Step 2:

[0336] The server receives the basic information sent by the user and generates an appropriate question and answer scenario using a generation means. The generated scenario consists of a list of questions and answers, which are converted into data packets again and sent to the terminal.

[0337] Input: Basic information in data packet format.

[0338] Output: Question and answer scenario in data packet format.

[0339] Specific operation: The generation means uses natural language generation technology to automatically select relevant questions based on the input information and compile them into a scenario.

[0340] Step 3:

[0341] The terminal receives the scenario and outputs a question to the user by voice using the voice output means. For example, a question such as "Do you have experience in inventory management?" is presented by voice.

[0342] Input: A question and answer scenario in the form of a data packet.

[0343] Output: Question in audio format.

[0344] Specific operation: The generated question is played aloud using the device's voice output function, and the user listens to it and prepares to answer the question.

[0345] Step 4:

[0346] The user answers questions by voice, and the voice input means captures the answers, records them, converts them into data packets, and sends them to the server.

[0347] Input: The user's spoken response.

[0348] Output: Voice response in the form of data packets.

[0349] Specific operation: The user answers questions using a microphone, and the answers are recorded and sent to the server in the form of data packets.

[0350] Step 5:

[0351] The server analyzes the received voice data and converts it into text using the Google Speech-to-Text API. It also uses the IBM Watson Tone Analyzer API to analyze the emotional state of staff. The analysis results are compiled as evaluation data, including the content of responses and emotional state.

[0352] Input: Voice response in the form of a data packet.

[0353] Output: Text response data and sentiment analysis data.

[0354] Specific operation: The server converts the voice data into text and then uses a sentiment analysis engine to analyze the user's emotional state. Evaluation data is generated as the analysis result.

[0355] Step 6:

[0356] The server generates a rating and feedback based on the analysis, including specific advice on the appropriateness of the answer, pronunciation, wording, and appropriate responses. The rating and feedback are converted into a data packet and sent to the device.

[0357] Input: Text response data and sentiment analysis data.

[0358] Output: Evaluation and feedback in the form of a data packet.

[0359] Specific operation: The server's evaluation means creates specific feedback based on the analysis results and sends it to the terminal in the form of a data packet.

[0360] Step 7:

[0361] The device provides the user with the rating and feedback it receives, either as audio output or text, and includes specific advice based on emotions.

[0362] Input: Ratings and feedback in the form of data packets.

[0363] Output: Feedback in the form of audio or text.

[0364] Specific operation: The device provides feedback to the user by playing the feedback aloud or displaying it as text on the screen.

[0365] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0366] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0367] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0368] [Second embodiment]

[0369] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0370] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0371] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0372] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0373] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0374] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0375] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0376] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0377] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0378] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0379] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0380] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0381] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[0382] 1. Initial settings and user information entry

[0383] A user first uses a user interface to access the system and enter their profile information, including basic information such as their name, desired career, and purpose of the interview. The terminal receives this information and transmits it to the server.

[0384] 2. Generating interview scenarios

[0385] The server uses a generation means to generate an appropriate Q&A scenario based on the received user information. This generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of a combination of questions and expected answers.

[0386] 3. Start practicing for the interview

[0387] Based on the received scenario, the device uses the voice output means to ask the user a question by voice. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer by using the voice input means. In this way, the question and answer process progresses.

[0388] 4. Sending and analyzing response data

[0389] The device sends the recorded audio data to a server, which then uses an evaluation tool to analyze the responses, including the appropriateness of the content, language, attitude, and pronunciation. Each element is scored and feedback is generated.

[0390] 5. Providing Feedback

[0391] The server sends the generated evaluation score and feedback to the device. The device receives the evaluation score and feedback and provides the user with feedback in the form of voice or text. For example, the device may display feedback such as, "Your self-introduction is not specific enough. Please add more specific examples."

[0392] Specific examples

[0393] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question may be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[0394] This system allows users to practice interviews in a realistic environment and also provides objective evaluations and specific areas for improvement, allowing users to fully prepare for the actual interview.

[0395] The processing flow will be explained below.

[0396] Step 1:

[0397] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal receives this information, converts it into data packets, and sends them to the server.

[0398] Step 2:

[0399] The server receives and analyzes the received user information. It then activates a generation means to generate an appropriate question and answer scenario based on the user's desired career and the purpose of the interview. The generation means selects questions using a past database and machine learning algorithms and generates a scenario. The generated scenario consists of pairs of questions and their expected answers. The server converts this scenario into data packets and sends them back to the terminal.

[0400] Step 3:

[0401] The terminal analyzes the received scenario data and outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the user's answer using the voice input means.

[0402] Step 4:

[0403] The terminal converts the recorded voice data into data packets and sends them to the server, where the user's voice response reaches the server.

[0404] Step 5:

[0405] The server launches an evaluation means to analyze the received voice data. The evaluation means uses a speech recognition algorithm to convert the voice data into text, and analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. The evaluation means assigns an evaluation score as a result of the analysis, and also generates feedback on the user's weaknesses and areas for improvement. This information is compiled as feedback data.

[0406] Step 6:

[0407] The server converts the generated evaluation score and feedback data into a data packet and sends it to the terminal. The terminal receives this data and provides the user with feedback in the form of voice or text. For example, the server might say, "Your self-introduction is not specific enough. Please add more specific examples."

[0408] Step 7:

[0409] The user can refine their answer based on the provided feedback, practice again using the refined answer, and the device will record a new answer and repeat the process from step 4 onwards.

[0410] Through these steps, users can engage in high-quality interview practice in a realistic environment and improve their skills through objective evaluation and specific feedback.

[0411] Example 1

[0412] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0413] Traditional interview practice requires the time and effort of others to obtain third-party evaluation, making it difficult to conduct high-quality practice alone. Furthermore, there is a lack of a system for receiving immediate, objective, and specific feedback. This makes it difficult to maintain motivation for independent interview practice and achieve effective self-improvement.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0415] In this invention, the server includes a generation means that analyzes user information and generates an appropriate Q&A scenario using a generative AI model, an evaluation means that analyzes the acquired answers and generates an evaluation and feedback thereof, and a display means that provides the generated feedback to the user, thereby enabling the user to practice for a high-quality interview on their own and receive immediate, objective, and specific feedback.

[0416] "User interface means" refers to an interface that allows a user to input information into a system. Examples include web forms and application input screens.

[0417] "Communication Methods" refers to the ability to send and receive data between a device and a server, including HTTP requests over the Internet and other digital communication protocols.

[0418] "Generation means" refers to the function of analyzing user information and generating appropriate Q&A scenarios using a generative AI model, which uses past data and specific algorithms.

[0419] "Voice output means" refers to a function that outputs questions to the user by voice based on the generated scenario. Specific examples include speakers and voice synthesis technology.

[0420] "Voice input means" refers to a function for acquiring and recording a user's response in voice format. A specific example is a microphone.

[0421] "Evaluation means" refers to the function that analyzes the answers obtained and generates evaluations and feedback, using natural language processing and voice analysis technologies.

[0422] The "display means" refers to a function for providing the generated feedback to the user. Specific examples include a method of displaying text on a display or a method of playing back the feedback by voice.

[0423] The present invention is a system that allows users to practice interviews on their own and receive specific feedback. This system is implemented mainly using a terminal and a server.

[0424] 1. Initial settings and user information entry

[0425] First, a user accesses the system using a terminal. Using a user interface, the user inputs his / her profile information (e.g., name, desired career, purpose of interview, etc.). This information is sent from the terminal to the server.

[0426] 2. Generating interview scenarios

[0427] The server analyzes the received user information and generates an appropriate question and answer scenario using a generative AI model. The generative AI model operates based on past data and specific algorithms to select questions that match the user's desired career and the purpose of the interview. This scenario consists of a combination of questions and expected answers, and the generated scenario is sent to the device.

[0428] 3. Start practicing for the interview

[0429] The terminal uses the voice output means to ask the user a question based on the received scenario. For example, the question "Please introduce yourself" is output as a voice. The user answers this question by voice, and this answer is recorded by the voice input means.

[0430] 4. Sending and analyzing response data

[0431] The recorded voice data is sent from the device to a server. The server receives this data and uses evaluation tools to analyze the answers from various angles. Analysis criteria include appropriateness of content, language, attitude, pronunciation, etc. The server generates feedback based on these criteria.

[0432] 5. Providing Feedback

[0433] The server sends the generated feedback to the device, which then receives it and provides the user with feedback in the form of voice or text. For example, the feedback might say, "Your self-introduction is not specific enough. Please add more specific examples."

[0434] Specific examples

[0435] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" in their profile information, which is then sent to the server. Based on this information, the server generates a scenario containing questions related to sales positions and expected answers to those questions. For example, the server generates a question such as "Tell us about your work experience to date."

[0436] The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server, which analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[0437] Prompt Sentence Examples

[0438] Below are some example prompts using a generative AI model:

[0439] "Generate an interview scenario for a sales position. User profile information is as follows: name, desired career is sales, purpose of interview is job hunting."

[0440] This system allows users to practice interviews in a realistic environment and receive objective and specific feedback, helping them to fully prepare for the actual interview.

[0441] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0442] Step 1:

[0443] A user accesses the system and uses the user interface to input their profile information (name, desired career, purpose of the interview), which is then acquired by the terminal and transmitted to the server via the communication means.

[0444] Input: User profile information

[0445] Data processing and calculation: Convert information into JSON format

[0446] Output: User information sent to the server

[0447] Step 2:

[0448] The server analyzes the received user information and generates an appropriate Q&A scenario using a generative AI model. The generative AI model uses past data and algorithms to create a set of questions and answers that are tailored to the user's desired career and interview objectives. The generated scenario is sent to the device via communication means.

[0449] Input: User information sent to the server

[0450] Data processing and calculation: Creating Q&A scenarios using generative AI models

[0451] Output: Sends the Q&A scenario (JSON format) to the terminal.

[0452] Step 3:

[0453] Based on the received scenario, the terminal uses the audio output means to ask the user a question by voice. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the answer by using the audio input means.

[0454] Input: Question and Answer scenario received by terminal

[0455] Data processing and calculation: Voice output of questions and answers, recording of user voice responses

[0456] Output: Recorded user voice response

[0457] Step 4:

[0458] The terminal encodes the recorded voice data using BASE64 and transmits it to the server via a communication means.

[0459] Input: Recorded user voice response

[0460] Data processing and calculation: BASE64 encoding of audio data

[0461] Output: The encoded audio data sent to the server.

[0462] Step 5:

[0463] The server decodes the received voice data and converts it into text using speech recognition technology. It then uses natural language processing (NLP) technology to evaluate the answers and generate feedback. Evaluation criteria include appropriateness of content, language, attitude, pronunciation, etc.

[0464] Input: Received encoded audio data

[0465] Data processing and calculation: Decoding voice data, speech recognition, analysis using NLP

[0466] Output: Generated feedback (in text format)

[0467] Step 6:

[0468] The server sends the generated feedback to the terminal. The terminal receives the feedback and provides it to the user in the form of voice or text via a display means. For example, a message may be played over the speaker saying, "Your self-introduction is not specific enough. Please add more specific examples."

[0469] Input: Feedback sent by the server

[0470] Data processing and calculation: Text-based feedback is displayed as voice or text

[0471] Output: Feedback provided to the user (audio or text)

[0472] This series of processes allows users to conduct high-quality interview practice on their own and receive objective and specific feedback.

[0473] (Application example 1)

[0474] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0475] With conventional interview practice systems, it was difficult for users to conduct high-quality interview practice on their own, and it was also difficult to obtain specific feedback. In particular, when store staff practiced customer service interviews, there was a lack of a means to objectively evaluate the results of the practice and provide specific areas for improvement.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0477] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice synthesis means for outputting the generated scenario as voice, a voice recognition means for acquiring the user's answers in voice format, recording them and converting them into text format, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback, and a display means for providing the evaluation and feedback generated by the evaluation means to the user in voice or text format. This enables a user to conduct high-quality interview practice even by themselves and to receive specific feedback even in customer service training at a physical store.

[0478] "User interface means" refers to an interface through which a user inputs their information into the system.

[0479] The "generation means" is a means for analyzing user information and generating an appropriate question and answer scenario based on that information.

[0480] The "audio output means" is a means for outputting the generated scenario to the user by audio.

[0481] The "voice input means" is a means for obtaining the user's response in voice form.

[0482] The "evaluation means" is a means for analyzing the acquired voice data and generating an evaluation or feedback.

[0483] The "display means" is a means for providing the evaluation and feedback generated by the evaluation means to the user in audio or text format.

[0484] The "voice synthesis means" is a means for converting a text-format scenario into voice and outputting it.

[0485] A "voice recognition means" is a means for recording a user's response in voice form and converting it into text form.

[0486]

[0487] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[0488] User Interface Means

[0489] The user first enters their profile information using the system's user interface means. At this stage, the user enters basic information such as their name, desired career, and purpose of the interview. This information is received by the terminal and transmitted to the server.

[0490] Scenario generation method

[0491] The server uses a generation means to generate an appropriate question and answer scenario based on the received user information. This generation means selects questions that are in line with the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario includes a combination of questions and expected answers.

[0492] Voice output means and voice synthesis means

[0493] The terminal uses the voice output means based on the generated scenario and utilizes the voice synthesis means to ask the user a question by voice, for example, "Please introduce yourself."

[0494] Voice input and voice recognition methods

[0495] The user answers the questions by voice, and the terminal records the answers using a voice input means. The recorded voice data is converted into text format by a voice recognition means.

[0496] Analysis of response data

[0497] The device sends the recorded and transcribed voice data to the server, which then uses an evaluation tool to analyze the answers, including the appropriateness of the content, the language used, the attitude, and the pronunciation, and assigns a score for each element to generate feedback.

[0498] Providing feedback

[0499] The server transmits the generated evaluation score and feedback to the terminal. The terminal receives the evaluation score and feedback and provides the feedback to the user in the form of voice or text using a display means. For example, feedback such as "Your self-introduction is not specific enough. Please add more specific examples" is displayed as voice or text.

[0500] Specific examples

[0501] Let's say the user is aiming to become a store staff member, and the purpose of the interview is job hunting. The user enters "store staff member" and "job hunting" as their profile information. The server generates a scenario containing questions related to store staff members and expected answers. For example, a question might be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud with "I worked as a store staff member for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides this to the user via the device.

[0502] Prompt Sentence Examples

[0503] Example prompt: "Generate an interview scenario for a store associate."

[0504] This allows users to practice interviews in an environment that is close to the real thing, and also provides objective evaluations and specific areas for improvement.The system utilizes natural language processing and speech recognition technology to provide an advanced interview practice environment.

[0505] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0506] Step 1:

[0507] The user inputs his / her profile information using the user interface means. At this stage, the user inputs basic information such as his / her name, desired career, and purpose of the interview. The input data is transmitted to the terminal.

[0508] Step 2:

[0509] The device sends the received user information to the server, which analyzes the user information and uses a generation method to generate an appropriate question and answer scenario. In doing so, the server utilizes past data and algorithms to select questions that are in line with the user's desired career and the purpose of the interview.

[0510] Step 3:

[0511] The scenario generated by the server is sent to the terminal. The terminal uses a speech synthesis means to convert the generated scenario into a voice format. An audio file is generated, and the questions in the scenario are output to the user by voice.

[0512] Step 4:

[0513] The user answers questions by voice, and the terminal records the user's answers using a voice input means. This voice data is converted into text format by a voice recognition means.

[0514] Step 5:

[0515] The device then transmits the converted response data to the server, which then uses an evaluation tool to analyze the response. The analysis includes the appropriateness of the response, the language used, the attitude, and the pronunciation, and assigns a score to each element.

[0516] Step 6:

[0517] The server generates an evaluation and feedback based on the analysis results, and the generated feedback and evaluation points are sent to the device.

[0518] Step 7:

[0519] The terminal uses the display means to provide the user with evaluation and feedback in the form of voice or text, for example, by displaying specific advice such as "Your answer is not specific enough. Please add more specific examples."

[0520] Step 8:

[0521] Users can receive feedback and review their practice interviews, providing specific improvements to incorporate into their next practice.

[0522] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0523] This system allows users to practice interviews independently and receive specific feedback. In particular, it combines an emotion engine that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, enabling a more comprehensive interview evaluation. This system is comprised of the following components:

[0524] 1. Initial settings and user information entry

[0525] A user first uses a user interface means to access the system and enter their profile information. The user enters basic information such as their name, desired career, and purpose of the interview. The terminal receives this information, converts it into data packets, and sends them to the server.

[0526] 2. Generating interview scenarios

[0527] Based on the received user information, the server uses a generation means to generate an appropriate question and answer scenario. The generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of pairs of questions and expected answers. The server converts this scenario into data packets and sends them to the terminal.

[0528] 3. Start practicing for the interview

[0529] Based on the received scenario, the device outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer using the voice input means. This is how the question and answer process progresses.

[0530] 4. Sending and analyzing response data

[0531] The device converts the recorded voice data into data packets and sends them to the server. The server receives this data and analyzes the answers using an evaluation tool. At the same time, an emotion engine is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice responses.

[0532] 5. Evaluation and feedback generation

[0533] The server analyzes the voice data through the evaluation means and evaluates the appropriateness of the response, pronunciation, wording, attitude, etc. Furthermore, it generates an evaluation score taking into account the user's emotional state recognized by the emotion engine. This allows for a more comprehensive evaluation. The generated evaluation score and feedback are packaged as a data packet and sent to the terminal.

[0534] 6. Providing Feedback

[0535] The device analyzes the received evaluation scores and feedback data and provides the user with feedback in the form of voice or text. For example, in addition to feedback such as "Your self-introduction is not specific enough. Please add more specific examples," it also provides emotion-based advice such as "You seem a little nervous. Try to relax."

[0536] Specific examples

[0537] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question might be generated: "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud: "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and further provides advice based on the analysis results of the emotion engine: "You seem tense in your voice. Take a deep breath and try to relax."

[0538] This system allows users to practice interviews in a realistic environment and receive comprehensive evaluations, specific suggestions for improvement, and emotional support, helping them to fully prepare for the actual interview.

[0539] The processing flow will be explained below.

[0540] Step 1:

[0541] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal collects this information, converts it into data packets, and sends them to the server.

[0542] Step 2:

[0543] The server receives and analyzes the received user information. Based on the analyzed information, it activates a generation means to generate an appropriate question and answer scenario. The generation means uses algorithms and pattern recognition technology to select questions that are in line with the user's desired career and the purpose of the interview, and generates a scenario. The generated scenario is sent to the terminal as a data packet.

[0544] Step 3:

[0545] The terminal analyzes the received scenario data and asks the user questions aloud using the voice output means. For example, a question such as "Please tell us about your work experience to date" is played aloud. The user answers the questions aloud, and the terminal records the answers using the voice input means.

[0546] Step 4:

[0547] The terminal converts the recorded user's answer data into a data packet and sends it to the server, whereupon the user's voice answer reaches the server.

[0548] Step 5:

[0549] The server activates an evaluation means to analyze the received voice data, which converts the voice data into text using a speech recognition algorithm and analyzes the appropriateness of the content, pronunciation, wording, attitude, etc.

[0550] Step 6:

[0551] The server activates an emotion engine to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. The emotion engine analyzes the tone, speed, and pitch of the voice to identify the emotion.

[0552] Step 7:

[0553] The evaluation means generates a comprehensive evaluation score based on the analysis results of the obtained answers and the emotional state recognized by the emotion engine, and simultaneously generates specific feedback and emotion-based advice based on the user's evaluation.

[0554] Step 8:

[0555] The server converts the generated evaluation score and feedback data into a data packet and transmits it to the terminal.

[0556] Step 9:

[0557] The device analyzes the evaluation scores and feedback data it receives and provides the user with feedback in the form of voice or text. For example, it may provide feedback such as, "It would be good to mention your role and specific achievements," or emotional advice such as, "You sound tense. Take a deep breath and relax."

[0558] Step 10:

[0559] The user can use the provided feedback and advice to improve their answer next time and practice again. The device will record a new voice answer and repeat the process from step 4 onwards.

[0560] Through this series of steps, users can practice interviews in a realistic environment and receive comprehensive evaluations and feedback that take into account their emotional state, helping them to fully prepare for the actual interview.

[0561] Example 2

[0562] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0563] In modern interview practice, it is difficult to achieve high-quality training, especially when done alone. Furthermore, conventional interview practice systems often do not provide specific feedback on the user's answers or evaluations that reflect the user's emotional state. As a result, users are unable to accurately identify their own problems and are unable to make effective improvements.

[0564] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0565] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for recognizing emotions from the user's voice and generating an evaluation and feedback based on the emotions. This allows the user to effectively practice for an advanced interview even on their own, and to receive a comprehensive evaluation that reflects specific feedback and their emotional state.

[0566] "User interface means" means a device or software that allows a user to access the system and input required information.

[0567] The "generator" is a device or software that creates an appropriate question and answer scenario based on information provided by the user.

[0568] The "audio output means" is a device or software for outputting questions to the user by voice based on the generated scenario.

[0569] "Voice input means" refers to a device or software for obtaining a user's voice response.

[0570] The "evaluation means" is a device or software that analyzes the user's answers obtained, evaluates their appropriateness, pronunciation, wording, attitude, etc., and generates feedback.

[0571] The "emotion analysis means" is a device or software for recognizing emotions from the user's voice and generating evaluations and feedback based on those emotions.

[0572] A "display means" is a device or software for providing the feedback generated by the evaluation means to the user in audio or text form.

[0573] This system allows a single user to practice interviews with high quality and receive specific feedback. In particular, by combining it with an emotion analysis tool that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, a more comprehensive interview evaluation can be achieved. This system is composed of the following elements:

[0574] Initial settings and user information entry

[0575] First, a user accesses the system using a terminal connected to the Internet. They enter their profile information (such as their name, desired occupation, and purpose of the interview) through a dedicated user interface. For example, they enter information such as "I'm looking for a sales position, and the purpose of the interview is job hunting." The entered information is converted into data packets by the terminal and sent to the server.

[0576] Interview scenario generation

[0577] The server analyzes the user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario that matches the user's desired career and the purpose of the interview. For example, a question such as "Please tell us about your work experience to date" is generated as a question about a sales position. The generated scenario is converted into a data packet and sent to the device.

[0578] Interview practice begins

[0579] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the audio output means, it outputs questions based on the scenario by voice. For example, a question such as "Please introduce yourself" is presented to the user by voice. The user answers this question by voice, and the terminal records the answer using the audio input means.

[0580] Sending and analyzing response data

[0581] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data, converts the voice content into text using a natural language processing algorithm, and analyzes the response. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (tension, joy, confidence, etc.) from the voice response.

[0582] Generate ratings and feedback

[0583] The server analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. using evaluation means. Furthermore, the user's emotional state, as recognized by emotion analysis means, is also included in the evaluation. This generates an overall evaluation score and specific feedback. For example, the server may provide feedback such as "It would be good to add a specific anecdote" or emotion-based advice such as "Your voice sounds tense, so try relaxing."

[0584] Providing feedback

[0585] The server converts the generated evaluation score and feedback into a data packet and sends it to the device. The device analyzes the received feedback data and provides the user with feedback in voice or text format. For example, the device might provide advice based on emotions, such as, "Your self-introduction lacks specificity. Please add more specific examples," or, "You seem a little nervous. Take a deep breath and try to relax."

[0586] Specific examples

[0587] For example, let's take a specific example where a user is aiming for a sales position. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers to those questions. The generated scenario includes the question, "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as, "It would be good to mention specific roles and achievements." Furthermore, based on the analysis results of the emotion engine, advice is provided such as, "Your voice sounds tense. Take a deep breath and try to relax."

[0588] As described above, the present invention allows users to effectively practice for interviews on their own and receive comprehensive evaluations, specific feedback, and emotional support, thereby enabling users to fully prepare for the actual interview.

[0589] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0590] Step 1:

[0591] Initial settings and user information entry

[0592] Users access the system using a terminal connected to the Internet. Through a dedicated user interface, they input profile information such as their name, desired occupation, and purpose of the interview (e.g., sales, job hunting). The terminal converts the input information into data packets and sends them to the server, which then receives the user's profile information.

[0593] Input: User profile information (name, occupation, purpose of interview)

[0594] Output: Data packet (including user information)

[0595] Step 2:

[0596] Interview scenario generation

[0597] The server receives and analyzes user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario tailored to the user's desired career and the purpose of the interview (e.g., questions about sales positions). The generated scenario is converted into a data packet and sent to the device. The server selects questions based on past data and algorithms, and also creates sample answers.

[0598] Input: Data packet (user information)

[0599] Output: Data packet (generated scenario)

[0600] Step 3:

[0601] Interview practice begins

[0602] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the voice output means, it outputs a question based on the scenario to the user by voice (e.g., "Please introduce yourself"). The user answers this question by voice, and the terminal records the answer using the voice input means. As a result, the user's speech is saved in the system.

[0603] Input: Data packet (scenario)

[0604] Output: Audio data (user's answer)

[0605] Step 4:

[0606] Sending and analyzing response data

[0607] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data and converts the voice content into text using a natural language processing algorithm, which then analyzes it. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. This allows both the content of the response and the user's emotional state to be analyzed.

[0608] Input: Voice data (user response)

[0609] Output: Text data (analyzed answers), emotion data

[0610] Step 5:

[0611] Generate ratings and feedback

[0612] The server uses an evaluation tool to analyze the appropriateness of the response, pronunciation, wording, and attitude based on the text data. Furthermore, the server also incorporates the user's emotional state, as recognized by the emotion analysis tool, into the evaluation. This generates an overall evaluation score and specific feedback (e.g., adding a specific anecdote would be helpful). The generated evaluation score and feedback are converted into a data packet and sent to the device.

[0613] Input: Text data (answer content), emotion data

[0614] Output: Data packet (ratings and feedback)

[0615] Step 6:

[0616] Providing feedback

[0617] The device analyzes the received data packets and provides feedback to the user in voice or text format (e.g., your self-introduction is not specific enough). In addition to specific areas for improvement, emotion-based advice is also provided (e.g., take a deep breath to relax). This allows users to receive practical feedback.

[0618] Input: Data packet (ratings and feedback)

[0619] Output: Feedback (audio or text)

[0620] (Application example 2)

[0621] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0622] Conventional interview practice systems have difficulty understanding the user's own emotions and state of tension, and providing appropriate feedback and advice. Furthermore, the training process for new staff at logistics centers can sometimes delay the acquisition of skills that will enable them to be immediately effective. This can lead to issues such as reduced efficiency throughout the center.

[0623] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0624] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for analyzing the user's emotional state and providing an evaluation and feedback based on the analysis results. This makes it possible to provide specific advice based on the user's emotional state, allowing new staff at a logistics center to effectively hone their skills and become immediately effective.

[0625] 1. "User interface means" refers to an interface through which a user inputs or obtains information into a system.

[0626] 2. "Generation means" refers to the means for analyzing input user information and creating an appropriate question and answer scenario based on that information.

[0627] 3. "Voice output means" refers to a means for outputting questions to the user by voice based on the generated scenario.

[0628] 4. "Voice input means" refers to the means for obtaining a user's response in voice form and inputting it into the system.

[0629] 5. "Evaluation means" refers to the means for analyzing a user's voice response, evaluating its content and appropriateness, and generating feedback.

[0630] 6. "Emotion analysis means" refers to a means for analyzing the emotional state of a user from their voice response and providing evaluation and feedback based on the analysis results.

[0631] 7. "Feedback" refers to information including evaluation of the user's answers, suggestions for improvement, and advice.

[0632] 8. "Logistics center" refers to a facility where goods and materials are stored, sorted, and prepared for delivery.

[0633] 9. "New staff" refers to employees newly hired at the logistics center.

[0634] 10. "Immediately effective" refers to having techniques and skills that can be immediately used in actual work.

[0635] A system for realizing this application example will be described in detail below.

[0636] System configuration

[0637] First, the interview training system for new staff at the logistics center consists of the following main components:

[0638] 1. User Interface Methods

[0639] 2. Generation means

[0640] 3. Audio output means

[0641] 4. Voice input method

[0642] 5. Evaluation Methods

[0643] 6. Emotion analysis method

[0644] User Interface Means

[0645] The user interface means is an interface that allows staff to access the system and input basic information such as their name, the work they will be responsible for, past experience, etc. For example, a smartphone or PC application can be used.

[0646] generation means

[0647] The generation method analyzes the input staff information and creates appropriate Q&A scenarios based on that information. This method uses past data and algorithms. Specifically, it uses natural language generation technology to automatically generate questions related to the staff member's work.

[0648] Audio output means

[0649] The voice output means outputs a question to the staff member by voice based on the generated scenario. For example, the voice output function of a smartphone can be used to voice the question, "Do you have experience in inventory management?"

[0650] Voice input means

[0651] The voice input method receives the staff's response in voice format and inputs it into the system. The voice data is recorded as is and sent to the server. For example, the voice is converted into text using the Google Speech-to-Text API.

[0652] Evaluation methods

[0653] The evaluation means analyzes the answer data acquired on the server side and evaluates its content and appropriateness. The content, pronunciation, and wording of the answer are evaluated here. Feedback is also generated based on the evaluation results.

[0654] Emotion analysis means

[0655] The emotion analysis means analyzes the emotional state (e.g., nervousness, confidence, etc.) of staff members' voice responses and provides evaluation and feedback based on the analysis results. For example, IBM Watson Tone Analyzer API is used to recognize emotions from voice data.

[0656] Program processing

[0657] In this system, a staff member first inputs basic information using a smartphone or PC. The server receives that information, and a generation means generates an appropriate question-and-answer scenario. Based on the generated scenario, a voice output means outputs questions to the staff member by voice, and the staff member answers by voice. A voice input means acquires the answers and sends them to the server. An evaluation means then analyzes the answers, and emotions are also analyzed by an emotion analysis means. Finally, feedback is generated based on the results of the evaluation and emotion analysis and provided to the staff member.

[0658] Specific examples

[0659] For example, if a new staff member at a distribution center is in charge of "inventory management," the following questions might be generated:

[0660] 1. Enter staff information

[0661] Name: Taro

[0662] Responsibilities: Inventory management

[0663] Experience: 2 years of inventory management experience

[0664] 2. Generated Questions

[0665] "Explain the main steps in inventory management."

[0666] "Tell us about a difficult situation you've had with inventory management in the past and how you dealt with it."

[0667] 3. Staff responses and feedback

[0668] Staff members answer questions and the answers are sent to the server.

[0669] The server analyzes the answers and generates feedback such as "Please explain the specific steps in more detail" and emotion-based advice such as "Your voice sounds tense. Relax."

[0670] Prompt Sentence Examples

[0671] Example prompts to be input to the generative AI model:

[0672] "Encourage candidates to describe in detail their inventory management experience. See if they use specific examples to explain the procedures and their role. Also, assess their emotional state (tension, confidence, etc.) and provide feedback accordingly."

[0673] In this way, the present invention allows new staff at a logistics center to effectively hone their skills so that they can be immediately put to work through practical interview training.

[0674] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0675] Step 1:

[0676] A user uses a smartphone or a PC to input basic information (such as name, work to be performed, past experience, etc.) through a user interface. The input information is converted into a data packet and sent to the server.

[0677] Input: Basic information such as name, expected duties, past experience, etc.

[0678] Output: Basic information about the data packet format.

[0679] Specific behavior: A user accesses the application, enters information into the designated fields, and then presses the "Submit" button to send the information to the server.

[0680] Step 2:

[0681] The server receives the basic information sent by the user and generates an appropriate question and answer scenario using a generation means. The generated scenario consists of a list of questions and answers, which are converted into data packets again and sent to the terminal.

[0682] Input: Basic information in data packet format.

[0683] Output: Question and answer scenario in data packet format.

[0684] Specific operation: The generation means uses natural language generation technology to automatically select relevant questions based on the input information and compile them into a scenario.

[0685] Step 3:

[0686] The terminal receives the scenario and outputs a question to the user by voice using the voice output means. For example, a question such as "Do you have experience in inventory management?" is presented by voice.

[0687] Input: A question and answer scenario in the form of a data packet.

[0688] Output: Question in audio format.

[0689] Specific operation: The generated question is played aloud using the device's voice output function, and the user listens to it and prepares to answer the question.

[0690] Step 4:

[0691] The user answers questions by voice, and the voice input means captures the answers, records them, converts them into data packets, and sends them to the server.

[0692] Input: The user's spoken response.

[0693] Output: Voice response in the form of data packets.

[0694] Specific operation: The user answers questions using a microphone, and the answers are recorded and sent to the server in the form of data packets.

[0695] Step 5:

[0696] The server analyzes the received voice data and converts it into text using the Google Speech-to-Text API. It also uses the IBM Watson Tone Analyzer API to analyze the emotional state of staff. The analysis results are compiled as evaluation data, including the content of responses and emotional state.

[0697] Input: Voice response in the form of a data packet.

[0698] Output: Text response data and sentiment analysis data.

[0699] Specific operation: The server converts the voice data into text and then uses a sentiment analysis engine to analyze the user's emotional state. Evaluation data is generated as the analysis result.

[0700] Step 6:

[0701] The server generates a rating and feedback based on the analysis, including specific advice on the appropriateness of the answer, pronunciation, wording, and appropriate responses. The rating and feedback are converted into a data packet and sent to the device.

[0702] Input: Text response data and sentiment analysis data.

[0703] Output: Evaluation and feedback in the form of a data packet.

[0704] Specific operation: The server's evaluation means creates specific feedback based on the analysis results and sends it to the terminal in the form of a data packet.

[0705] Step 7:

[0706] The device provides the user with the rating and feedback it receives, either as audio output or text, and includes specific advice based on emotions.

[0707] Input: Ratings and feedback in the form of data packets.

[0708] Output: Feedback in the form of audio or text.

[0709] Specific operation: The device provides feedback to the user by playing the feedback aloud or displaying it as text on the screen.

[0710] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0711] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0712] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0713] [Third embodiment]

[0714] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0715] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0716] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0717] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0718] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0719] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0720] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0721] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0722] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0723] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0724] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0725] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0726] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[0727] 1. Initial settings and user information entry

[0728] A user first uses a user interface to access the system and enter their profile information, including basic information such as their name, desired career, and purpose of the interview. The terminal receives this information and transmits it to the server.

[0729] 2. Generating interview scenarios

[0730] The server uses a generation means to generate an appropriate Q&A scenario based on the received user information. This generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of a combination of questions and expected answers.

[0731] 3. Start practicing for the interview

[0732] Based on the received scenario, the device uses the voice output means to ask the user a question by voice. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer by using the voice input means. In this way, the question and answer process progresses.

[0733] 4. Sending and analyzing response data

[0734] The device sends the recorded audio data to a server, which then uses an evaluation tool to analyze the responses, including the appropriateness of the content, language, attitude, and pronunciation. Each element is scored and feedback is generated.

[0735] 5. Providing Feedback

[0736] The server sends the generated evaluation score and feedback to the device. The device receives the evaluation score and feedback and provides the user with feedback in the form of voice or text. For example, the device may display feedback such as, "Your self-introduction is not specific enough. Please add more specific examples."

[0737] Specific examples

[0738] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question may be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[0739] This system allows users to practice interviews in a realistic environment and also provides objective evaluations and specific areas for improvement, allowing users to fully prepare for the actual interview.

[0740] The processing flow will be explained below.

[0741] Step 1:

[0742] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal receives this information, converts it into data packets, and sends them to the server.

[0743] Step 2:

[0744] The server receives and analyzes the received user information. It then activates a generation means to generate an appropriate question and answer scenario based on the user's desired career and the purpose of the interview. The generation means selects questions using a past database and machine learning algorithms and generates a scenario. The generated scenario consists of pairs of questions and their expected answers. The server converts this scenario into data packets and sends them back to the terminal.

[0745] Step 3:

[0746] The terminal analyzes the received scenario data and outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the user's answer using the voice input means.

[0747] Step 4:

[0748] The terminal converts the recorded voice data into data packets and sends them to the server, where the user's voice response reaches the server.

[0749] Step 5:

[0750] The server launches an evaluation means to analyze the received voice data. The evaluation means uses a speech recognition algorithm to convert the voice data into text, and analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. The evaluation means assigns an evaluation score as a result of the analysis, and also generates feedback on the user's weaknesses and areas for improvement. This information is compiled as feedback data.

[0751] Step 6:

[0752] The server converts the generated evaluation score and feedback data into a data packet and sends it to the terminal. The terminal receives this data and provides the user with feedback in the form of voice or text. For example, the server might say, "Your self-introduction is not specific enough. Please add more specific examples."

[0753] Step 7:

[0754] The user can refine their answer based on the provided feedback, practice again using the refined answer, and the device will record a new answer and repeat the process from step 4 onwards.

[0755] Through these steps, users can engage in high-quality interview practice in a realistic environment and improve their skills through objective evaluation and specific feedback.

[0756] Example 1

[0757] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0758] Traditional interview practice requires the time and effort of others to obtain third-party evaluation, making it difficult to conduct high-quality practice alone. Furthermore, there is a lack of a system for receiving immediate, objective, and specific feedback. This makes it difficult to maintain motivation for independent interview practice and achieve effective self-improvement.

[0759] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0760] In this invention, the server includes a generation means that analyzes user information and generates an appropriate Q&A scenario using a generative AI model, an evaluation means that analyzes the acquired answers and generates an evaluation and feedback thereof, and a display means that provides the generated feedback to the user, thereby enabling the user to practice for a high-quality interview on their own and receive immediate, objective, and specific feedback.

[0761] "User interface means" refers to an interface that allows a user to input information into a system. Examples include web forms and application input screens.

[0762] "Communication Methods" refers to the ability to send and receive data between a device and a server, including HTTP requests over the Internet and other digital communication protocols.

[0763] "Generation means" refers to the function of analyzing user information and generating appropriate Q&A scenarios using a generative AI model, which uses past data and specific algorithms.

[0764] "Voice output means" refers to a function that outputs questions to the user by voice based on the generated scenario. Specific examples include speakers and voice synthesis technology.

[0765] "Voice input means" refers to a function for acquiring and recording a user's response in voice format. A specific example is a microphone.

[0766] "Evaluation means" refers to the function that analyzes the answers obtained and generates evaluations and feedback, using natural language processing and voice analysis technologies.

[0767] The "display means" refers to a function for providing the generated feedback to the user. Specific examples include a method of displaying text on a display or a method of playing back the feedback by voice.

[0768] The present invention is a system that allows users to practice interviews on their own and receive specific feedback. This system is implemented mainly using a terminal and a server.

[0769] 1. Initial settings and user information entry

[0770] First, a user accesses the system using a terminal. Using a user interface, the user inputs his / her profile information (e.g., name, desired career, purpose of interview, etc.). This information is sent from the terminal to the server.

[0771] 2. Generating interview scenarios

[0772] The server analyzes the received user information and generates an appropriate question and answer scenario using a generative AI model. The generative AI model operates based on past data and specific algorithms to select questions that match the user's desired career and the purpose of the interview. This scenario consists of a combination of questions and expected answers, and the generated scenario is sent to the device.

[0773] 3. Start practicing for the interview

[0774] The terminal uses the voice output means to ask the user a question based on the received scenario. For example, the question "Please introduce yourself" is output as a voice. The user answers this question by voice, and this answer is recorded by the voice input means.

[0775] 4. Sending and analyzing response data

[0776] The recorded voice data is sent from the device to a server. The server receives this data and uses evaluation tools to analyze the answers from various angles. Analysis criteria include appropriateness of content, language, attitude, pronunciation, etc. The server generates feedback based on these criteria.

[0777] 5. Providing Feedback

[0778] The server sends the generated feedback to the device, which then receives it and provides the user with feedback in the form of voice or text. For example, the feedback might say, "Your self-introduction is not specific enough. Please add more specific examples."

[0779] Specific examples

[0780] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" in their profile information, which is then sent to the server. Based on this information, the server generates a scenario containing questions related to sales positions and expected answers to those questions. For example, the server generates a question such as "Tell us about your work experience to date."

[0781] The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server, which analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[0782] Prompt Sentence Examples

[0783] Below are some example prompts using a generative AI model:

[0784] "Generate an interview scenario for a sales position. User profile information is as follows: name, desired career is sales, purpose of interview is job hunting."

[0785] This system allows users to practice interviews in a realistic environment and receive objective and specific feedback, helping them to fully prepare for the actual interview.

[0786] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0787] Step 1:

[0788] A user accesses the system and uses the user interface to input their profile information (name, desired career, purpose of the interview), which is then acquired by the terminal and transmitted to the server via the communication means.

[0789] Input: User profile information

[0790] Data processing and calculation: Convert information into JSON format

[0791] Output: User information sent to the server

[0792] Step 2:

[0793] The server analyzes the received user information and generates an appropriate Q&A scenario using a generative AI model. The generative AI model uses past data and algorithms to create a set of questions and answers that are tailored to the user's desired career and interview objectives. The generated scenario is sent to the device via communication means.

[0794] Input: User information sent to the server

[0795] Data processing and calculation: Creating Q&A scenarios using generative AI models

[0796] Output: Sends the Q&A scenario (JSON format) to the terminal.

[0797] Step 3:

[0798] Based on the received scenario, the terminal uses the audio output means to ask the user a question by voice. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the answer by using the audio input means.

[0799] Input: Question and Answer scenario received by terminal

[0800] Data processing and calculation: Voice output of questions and answers, recording of user voice responses

[0801] Output: Recorded user voice response

[0802] Step 4:

[0803] The terminal encodes the recorded voice data using BASE64 and transmits it to the server via a communication means.

[0804] Input: Recorded user voice response

[0805] Data processing and calculation: BASE64 encoding of audio data

[0806] Output: The encoded audio data sent to the server.

[0807] Step 5:

[0808] The server decodes the received voice data and converts it into text using speech recognition technology. It then uses natural language processing (NLP) technology to evaluate the answers and generate feedback. Evaluation criteria include appropriateness of content, language, attitude, pronunciation, etc.

[0809] Input: Received encoded audio data

[0810] Data processing and calculation: Decoding voice data, speech recognition, analysis using NLP

[0811] Output: Generated feedback (in text format)

[0812] Step 6:

[0813] The server sends the generated feedback to the terminal. The terminal receives the feedback and provides it to the user in the form of voice or text via a display means. For example, a message may be played over the speaker saying, "Your self-introduction is not specific enough. Please add more specific examples."

[0814] Input: Feedback sent by the server

[0815] Data processing and calculation: Text-based feedback is displayed as voice or text

[0816] Output: Feedback provided to the user (audio or text)

[0817] This series of processes allows users to conduct high-quality interview practice on their own and receive objective and specific feedback.

[0818] (Application example 1)

[0819] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0820] With conventional interview practice systems, it was difficult for users to conduct high-quality interview practice on their own, and it was also difficult to obtain specific feedback. In particular, when store staff practiced customer service interviews, there was a lack of a means to objectively evaluate the results of the practice and provide specific areas for improvement.

[0821] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0822] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice synthesis means for outputting the generated scenario as voice, a voice recognition means for acquiring the user's answers in voice format, recording them and converting them into text format, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback, and a display means for providing the evaluation and feedback generated by the evaluation means to the user in voice or text format. This enables a user to conduct high-quality interview practice even by themselves and to receive specific feedback even in customer service training at a physical store.

[0823] "User interface means" refers to an interface through which a user inputs their information into the system.

[0824] The "generation means" is a means for analyzing user information and generating an appropriate question and answer scenario based on that information.

[0825] The "audio output means" is a means for outputting the generated scenario to the user by audio.

[0826] The "voice input means" is a means for obtaining the user's response in voice form.

[0827] The "evaluation means" is a means for analyzing the acquired voice data and generating an evaluation or feedback.

[0828] The "display means" is a means for providing the evaluation and feedback generated by the evaluation means to the user in audio or text format.

[0829] The "voice synthesis means" is a means for converting a text-format scenario into voice and outputting it.

[0830] A "voice recognition means" is a means for recording a user's response in voice form and converting it into text form.

[0831]

[0832] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[0833] User Interface Means

[0834] The user first enters their profile information using the system's user interface means. At this stage, the user enters basic information such as their name, desired career, and purpose of the interview. This information is received by the terminal and transmitted to the server.

[0835] Scenario generation method

[0836] The server uses a generation means to generate an appropriate question and answer scenario based on the received user information. This generation means selects questions that are in line with the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario includes a combination of questions and expected answers.

[0837] Voice output means and voice synthesis means

[0838] The terminal uses the voice output means based on the generated scenario and utilizes the voice synthesis means to ask the user a question by voice, for example, "Please introduce yourself."

[0839] Voice input and voice recognition methods

[0840] The user answers the questions by voice, and the terminal records the answers using a voice input means. The recorded voice data is converted into text format by a voice recognition means.

[0841] Analysis of response data

[0842] The device sends the recorded and transcribed voice data to the server, which then uses an evaluation tool to analyze the answers, including the appropriateness of the content, the language used, the attitude, and the pronunciation, and assigns a score for each element to generate feedback.

[0843] Providing feedback

[0844] The server transmits the generated evaluation score and feedback to the terminal. The terminal receives the evaluation score and feedback and provides the feedback to the user in the form of voice or text using a display means. For example, feedback such as "Your self-introduction is not specific enough. Please add more specific examples" is displayed as voice or text.

[0845] Specific examples

[0846] Let's say the user is aiming to become a store staff member, and the purpose of the interview is job hunting. The user enters "store staff member" and "job hunting" as their profile information. The server generates a scenario containing questions related to store staff members and expected answers. For example, a question might be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud with "I worked as a store staff member for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides this to the user via the device.

[0847] Prompt Sentence Examples

[0848] Example prompt: "Generate an interview scenario for a store associate."

[0849] This allows users to practice interviews in an environment that is close to the real thing, and also provides objective evaluations and specific areas for improvement.The system utilizes natural language processing and speech recognition technology to provide an advanced interview practice environment.

[0850] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0851] Step 1:

[0852] The user inputs his / her profile information using the user interface means. At this stage, the user inputs basic information such as his / her name, desired career, and purpose of the interview. The input data is transmitted to the terminal.

[0853] Step 2:

[0854] The device sends the received user information to the server, which analyzes the user information and uses a generation method to generate an appropriate question and answer scenario. In doing so, the server utilizes past data and algorithms to select questions that are in line with the user's desired career and the purpose of the interview.

[0855] Step 3:

[0856] The scenario generated by the server is sent to the terminal. The terminal uses a speech synthesis means to convert the generated scenario into a voice format. An audio file is generated, and the questions in the scenario are output to the user by voice.

[0857] Step 4:

[0858] The user answers questions by voice, and the terminal records the user's answers using a voice input means. This voice data is converted into text format by a voice recognition means.

[0859] Step 5:

[0860] The device then transmits the converted response data to the server, which then uses an evaluation tool to analyze the response. The analysis includes the appropriateness of the response, the language used, the attitude, and the pronunciation, and assigns a score to each element.

[0861] Step 6:

[0862] The server generates an evaluation and feedback based on the analysis results, and the generated feedback and evaluation points are sent to the device.

[0863] Step 7:

[0864] The terminal uses the display means to provide the user with evaluation and feedback in the form of voice or text, for example, by displaying specific advice such as "Your answer is not specific enough. Please add more specific examples."

[0865] Step 8:

[0866] Users can receive feedback and review their practice interviews, providing specific improvements to incorporate into their next practice.

[0867] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0868] This system allows users to practice interviews independently and receive specific feedback. In particular, it combines an emotion engine that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, enabling a more comprehensive interview evaluation. This system is comprised of the following components:

[0869] 1. Initial settings and user information entry

[0870] A user first uses a user interface means to access the system and enter their profile information. The user enters basic information such as their name, desired career, and purpose of the interview. The terminal receives this information, converts it into data packets, and sends them to the server.

[0871] 2. Generating interview scenarios

[0872] Based on the received user information, the server uses a generation means to generate an appropriate question and answer scenario. The generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of pairs of questions and expected answers. The server converts this scenario into data packets and sends them to the terminal.

[0873] 3. Start practicing for the interview

[0874] Based on the received scenario, the device outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer using the voice input means. This is how the question and answer process progresses.

[0875] 4. Sending and analyzing response data

[0876] The device converts the recorded voice data into data packets and sends them to the server. The server receives this data and analyzes the answers using an evaluation tool. At the same time, an emotion engine is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice responses.

[0877] 5. Evaluation and feedback generation

[0878] The server analyzes the voice data through the evaluation means and evaluates the appropriateness of the response, pronunciation, wording, attitude, etc. Furthermore, it generates an evaluation score taking into account the user's emotional state recognized by the emotion engine. This allows for a more comprehensive evaluation. The generated evaluation score and feedback are packaged as a data packet and sent to the terminal.

[0879] 6. Providing Feedback

[0880] The device analyzes the received evaluation scores and feedback data and provides the user with feedback in the form of voice or text. For example, in addition to feedback such as "Your self-introduction is not specific enough. Please add more specific examples," it also provides emotion-based advice such as "You seem a little nervous. Try to relax."

[0881] Specific examples

[0882] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question might be generated: "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud: "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and further provides advice based on the analysis results of the emotion engine: "You seem tense in your voice. Take a deep breath and try to relax."

[0883] This system allows users to practice interviews in a realistic environment and receive comprehensive evaluations, specific suggestions for improvement, and emotional support, helping them to fully prepare for the actual interview.

[0884] The processing flow will be explained below.

[0885] Step 1:

[0886] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal collects this information, converts it into data packets, and sends them to the server.

[0887] Step 2:

[0888] The server receives and analyzes the received user information. Based on the analyzed information, it activates a generation means to generate an appropriate question and answer scenario. The generation means uses algorithms and pattern recognition technology to select questions that are in line with the user's desired career and the purpose of the interview, and generates a scenario. The generated scenario is sent to the terminal as a data packet.

[0889] Step 3:

[0890] The terminal analyzes the received scenario data and asks the user questions aloud using the voice output means. For example, a question such as "Please tell us about your work experience to date" is played aloud. The user answers the questions aloud, and the terminal records the answers using the voice input means.

[0891] Step 4:

[0892] The terminal converts the recorded user's answer data into a data packet and sends it to the server, whereupon the user's voice answer reaches the server.

[0893] Step 5:

[0894] The server activates an evaluation means to analyze the received voice data, which converts the voice data into text using a speech recognition algorithm and analyzes the appropriateness of the content, pronunciation, wording, attitude, etc.

[0895] Step 6:

[0896] The server activates an emotion engine to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. The emotion engine analyzes the tone, speed, and pitch of the voice to identify the emotion.

[0897] Step 7:

[0898] The evaluation means generates a comprehensive evaluation score based on the analysis results of the obtained answers and the emotional state recognized by the emotion engine, and simultaneously generates specific feedback and emotion-based advice based on the user's evaluation.

[0899] Step 8:

[0900] The server converts the generated evaluation score and feedback data into a data packet and transmits it to the terminal.

[0901] Step 9:

[0902] The device analyzes the evaluation scores and feedback data it receives and provides the user with feedback in the form of voice or text. For example, it may provide feedback such as, "It would be good to mention your role and specific achievements," or emotional advice such as, "You sound tense. Take a deep breath and relax."

[0903] Step 10:

[0904] The user can use the provided feedback and advice to improve their answer next time and practice again. The device will record a new voice answer and repeat the process from step 4 onwards.

[0905] Through this series of steps, users can practice interviews in a realistic environment and receive comprehensive evaluations and feedback that take into account their emotional state, helping them to fully prepare for the actual interview.

[0906] Example 2

[0907] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0908] In modern interview practice, it is difficult to achieve high-quality training, especially when done alone. Furthermore, conventional interview practice systems often do not provide specific feedback on the user's answers or evaluations that reflect the user's emotional state. As a result, users are unable to accurately identify their own problems and are unable to make effective improvements.

[0909] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0910] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for recognizing emotions from the user's voice and generating an evaluation and feedback based on the emotions. This allows the user to effectively practice for an advanced interview even on their own, and to receive a comprehensive evaluation that reflects specific feedback and their emotional state.

[0911] "User interface means" means a device or software that allows a user to access the system and input required information.

[0912] The "generator" is a device or software that creates an appropriate question and answer scenario based on information provided by the user.

[0913] The "audio output means" is a device or software for outputting questions to the user by voice based on the generated scenario.

[0914] "Voice input means" refers to a device or software for obtaining a user's voice response.

[0915] The "evaluation means" is a device or software that analyzes the user's answers obtained, evaluates their appropriateness, pronunciation, wording, attitude, etc., and generates feedback.

[0916] The "emotion analysis means" is a device or software for recognizing emotions from the user's voice and generating evaluations and feedback based on those emotions.

[0917] A "display means" is a device or software for providing the feedback generated by the evaluation means to the user in audio or text form.

[0918] This system allows a single user to practice interviews with high quality and receive specific feedback. In particular, by combining it with an emotion analysis tool that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, a more comprehensive interview evaluation can be achieved. This system is composed of the following elements:

[0919] Initial settings and user information entry

[0920] First, a user accesses the system using a terminal connected to the Internet. They enter their profile information (such as their name, desired occupation, and purpose of the interview) through a dedicated user interface. For example, they enter information such as "I'm looking for a sales position, and the purpose of the interview is job hunting." The entered information is converted into data packets by the terminal and sent to the server.

[0921] Interview scenario generation

[0922] The server analyzes the user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario that matches the user's desired career and the purpose of the interview. For example, a question such as "Please tell us about your work experience to date" is generated as a question about a sales position. The generated scenario is converted into a data packet and sent to the device.

[0923] Interview practice begins

[0924] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the audio output means, it outputs questions based on the scenario by voice. For example, a question such as "Please introduce yourself" is presented to the user by voice. The user answers this question by voice, and the terminal records the answer using the audio input means.

[0925] Sending and analyzing response data

[0926] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data, converts the voice content into text using a natural language processing algorithm, and analyzes the response. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (tension, joy, confidence, etc.) from the voice response.

[0927] Generate ratings and feedback

[0928] The server analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. using evaluation means. Furthermore, the user's emotional state, as recognized by emotion analysis means, is also included in the evaluation. This generates an overall evaluation score and specific feedback. For example, the server may provide feedback such as "It would be good to add a specific anecdote" or emotion-based advice such as "Your voice sounds tense, so try relaxing."

[0929] Providing feedback

[0930] The server converts the generated evaluation score and feedback into a data packet and sends it to the device. The device analyzes the received feedback data and provides the user with feedback in voice or text format. For example, the device might provide advice based on emotions, such as, "Your self-introduction lacks specificity. Please add more specific examples," or, "You seem a little nervous. Take a deep breath and try to relax."

[0931] Specific examples

[0932] For example, let's take a specific example where a user is aiming for a sales position. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers to those questions. The generated scenario includes the question, "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as, "It would be good to mention specific roles and achievements." Furthermore, based on the analysis results of the emotion engine, advice is provided such as, "Your voice sounds tense. Take a deep breath and try to relax."

[0933] As described above, the present invention allows users to effectively practice for interviews on their own and receive comprehensive evaluations, specific feedback, and emotional support, thereby enabling users to fully prepare for the actual interview.

[0934] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0935] Step 1:

[0936] Initial settings and user information entry

[0937] Users access the system using a terminal connected to the Internet. Through a dedicated user interface, they input profile information such as their name, desired occupation, and purpose of the interview (e.g., sales, job hunting). The terminal converts the input information into data packets and sends them to the server, which then receives the user's profile information.

[0938] Input: User profile information (name, occupation, purpose of interview)

[0939] Output: Data packet (including user information)

[0940] Step 2:

[0941] Interview scenario generation

[0942] The server receives and analyzes user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario tailored to the user's desired career and the purpose of the interview (e.g., questions about sales positions). The generated scenario is converted into a data packet and sent to the device. The server selects questions based on past data and algorithms, and also creates sample answers.

[0943] Input: Data packet (user information)

[0944] Output: Data packet (generated scenario)

[0945] Step 3:

[0946] Interview practice begins

[0947] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the voice output means, it outputs a question based on the scenario to the user by voice (e.g., "Please introduce yourself"). The user answers this question by voice, and the terminal records the answer using the voice input means. As a result, the user's speech is saved in the system.

[0948] Input: Data packet (scenario)

[0949] Output: Audio data (user's answer)

[0950] Step 4:

[0951] Sending and analyzing response data

[0952] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data and converts the voice content into text using a natural language processing algorithm, which then analyzes it. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. This allows both the content of the response and the user's emotional state to be analyzed.

[0953] Input: Voice data (user response)

[0954] Output: Text data (analyzed answers), emotion data

[0955] Step 5:

[0956] Generate ratings and feedback

[0957] The server uses an evaluation tool to analyze the appropriateness of the response, pronunciation, wording, and attitude based on the text data. Furthermore, the server also incorporates the user's emotional state, as recognized by the emotion analysis tool, into the evaluation. This generates an overall evaluation score and specific feedback (e.g., adding a specific anecdote would be helpful). The generated evaluation score and feedback are converted into a data packet and sent to the device.

[0958] Input: Text data (answer content), emotion data

[0959] Output: Data packet (ratings and feedback)

[0960] Step 6:

[0961] Providing feedback

[0962] The device analyzes the received data packets and provides feedback to the user in voice or text format (e.g., your self-introduction is not specific enough). In addition to specific areas for improvement, emotion-based advice is also provided (e.g., take a deep breath to relax). This allows users to receive practical feedback.

[0963] Input: Data packet (ratings and feedback)

[0964] Output: Feedback (audio or text)

[0965] (Application example 2)

[0966] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0967] Conventional interview practice systems have difficulty understanding the user's own emotions and state of tension, and providing appropriate feedback and advice. Furthermore, the training process for new staff at logistics centers can sometimes delay the acquisition of skills that will enable them to be immediately effective. This can lead to issues such as reduced efficiency throughout the center.

[0968] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0969] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for analyzing the user's emotional state and providing an evaluation and feedback based on the analysis results. This makes it possible to provide specific advice based on the user's emotional state, allowing new staff at a logistics center to effectively hone their skills and become immediately effective.

[0970] 1. "User interface means" refers to an interface through which a user inputs or obtains information into a system.

[0971] 2. "Generation means" refers to the means for analyzing input user information and creating an appropriate question and answer scenario based on that information.

[0972] 3. "Voice output means" refers to a means for outputting questions to the user by voice based on the generated scenario.

[0973] 4. "Voice input means" refers to the means for obtaining a user's response in voice form and inputting it into the system.

[0974] 5. "Evaluation means" refers to the means for analyzing a user's voice response, evaluating its content and appropriateness, and generating feedback.

[0975] 6. "Emotion analysis means" refers to a means for analyzing the emotional state of a user from their voice response and providing evaluation and feedback based on the analysis results.

[0976] 7. "Feedback" refers to information including evaluation of the user's answers, suggestions for improvement, and advice.

[0977] 8. "Logistics center" refers to a facility where goods and materials are stored, sorted, and prepared for delivery.

[0978] 9. "New staff" refers to employees newly hired at the logistics center.

[0979] 10. "Immediately effective" refers to having techniques and skills that can be immediately used in actual work.

[0980] A system for realizing this application example will be described in detail below.

[0981] System configuration

[0982] First, the interview training system for new staff at the logistics center consists of the following main components:

[0983] 1. User Interface Methods

[0984] 2. Generation means

[0985] 3. Audio output means

[0986] 4. Voice input method

[0987] 5. Evaluation Methods

[0988] 6. Emotion analysis method

[0989] User Interface Means

[0990] The user interface means is an interface that allows staff to access the system and input basic information such as their name, the work they will be responsible for, past experience, etc. For example, a smartphone or PC application can be used.

[0991] generation means

[0992] The generation method analyzes the input staff information and creates appropriate Q&A scenarios based on that information. This method uses past data and algorithms. Specifically, it uses natural language generation technology to automatically generate questions related to the staff member's work.

[0993] Audio output means

[0994] The voice output means outputs a question to the staff member by voice based on the generated scenario. For example, the voice output function of a smartphone can be used to voice the question, "Do you have experience in inventory management?"

[0995] Voice input means

[0996] The voice input method receives the staff's response in voice format and inputs it into the system. The voice data is recorded as is and sent to the server. For example, the voice is converted into text using the Google Speech-to-Text API.

[0997] Evaluation methods

[0998] The evaluation means analyzes the answer data acquired on the server side and evaluates its content and appropriateness. The content, pronunciation, and wording of the answer are evaluated here. Feedback is also generated based on the evaluation results.

[0999] Emotion analysis means

[1000] The emotion analysis means analyzes the emotional state (e.g., nervousness, confidence, etc.) of staff members' voice responses and provides evaluation and feedback based on the analysis results. For example, IBM Watson Tone Analyzer API is used to recognize emotions from voice data.

[1001] Program processing

[1002] In this system, a staff member first inputs basic information using a smartphone or PC. The server receives that information, and a generation means generates an appropriate question-and-answer scenario. Based on the generated scenario, a voice output means outputs questions to the staff member by voice, and the staff member answers by voice. A voice input means acquires the answers and sends them to the server. An evaluation means then analyzes the answers, and emotions are also analyzed by an emotion analysis means. Finally, feedback is generated based on the results of the evaluation and emotion analysis and provided to the staff member.

[1003] Specific examples

[1004] For example, if a new staff member at a distribution center is in charge of "inventory management," the following questions might be generated:

[1005] 1. Enter staff information

[1006] Name: Taro

[1007] Responsibilities: Inventory management

[1008] Experience: 2 years of inventory management experience

[1009] 2. Generated Questions

[1010] "Explain the main steps in inventory management."

[1011] "Tell us about a difficult situation you've had with inventory management in the past and how you dealt with it."

[1012] 3. Staff responses and feedback

[1013] Staff members answer questions and the answers are sent to the server.

[1014] The server analyzes the answers and generates feedback such as "Please explain the specific steps in more detail" and emotion-based advice such as "Your voice sounds tense. Relax."

[1015] Prompt Sentence Examples

[1016] Example prompts to be input to the generative AI model:

[1017] "Encourage candidates to describe in detail their inventory management experience. See if they use specific examples to explain the procedures and their role. Also, assess their emotional state (tension, confidence, etc.) and provide feedback accordingly."

[1018] In this way, the present invention allows new staff at a logistics center to effectively hone their skills so that they can be immediately put to work through practical interview training.

[1019] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1020] Step 1:

[1021] A user uses a smartphone or a PC to input basic information (such as name, work to be performed, past experience, etc.) through a user interface. The input information is converted into a data packet and sent to the server.

[1022] Input: Basic information such as name, expected duties, past experience, etc.

[1023] Output: Basic information about the data packet format.

[1024] Specific behavior: A user accesses the application, enters information into the designated fields, and then presses the "Submit" button to send the information to the server.

[1025] Step 2:

[1026] The server receives the basic information sent by the user and generates an appropriate question and answer scenario using a generation means. The generated scenario consists of a list of questions and answers, which are converted into data packets again and sent to the terminal.

[1027] Input: Basic information in data packet format.

[1028] Output: Question and answer scenario in data packet format.

[1029] Specific operation: The generation means uses natural language generation technology to automatically select relevant questions based on the input information and compile them into a scenario.

[1030] Step 3:

[1031] The terminal receives the scenario and outputs a question to the user by voice using the voice output means. For example, a question such as "Do you have experience in inventory management?" is presented by voice.

[1032] Input: A question and answer scenario in the form of a data packet.

[1033] Output: Question in audio format.

[1034] Specific operation: The generated question is played aloud using the device's voice output function, and the user listens to it and prepares to answer the question.

[1035] Step 4:

[1036] The user answers questions by voice, and the voice input means captures the answers, records them, converts them into data packets, and sends them to the server.

[1037] Input: The user's spoken response.

[1038] Output: Voice response in the form of data packets.

[1039] Specific operation: The user answers questions using a microphone, and the answers are recorded and sent to the server in the form of data packets.

[1040] Step 5:

[1041] The server analyzes the received voice data and converts it into text using the Google Speech-to-Text API. It also uses the IBM Watson Tone Analyzer API to analyze the emotional state of staff. The analysis results are compiled as evaluation data, including the content of responses and emotional state.

[1042] Input: Voice response in the form of a data packet.

[1043] Output: Text response data and sentiment analysis data.

[1044] Specific operation: The server converts the voice data into text and then uses a sentiment analysis engine to analyze the user's emotional state. Evaluation data is generated as the analysis result.

[1045] Step 6:

[1046] The server generates a rating and feedback based on the analysis, including specific advice on the appropriateness of the answer, pronunciation, wording, and appropriate responses. The rating and feedback are converted into a data packet and sent to the device.

[1047] Input: Text response data and sentiment analysis data.

[1048] Output: Evaluation and feedback in the form of a data packet.

[1049] Specific operation: The server's evaluation means creates specific feedback based on the analysis results and sends it to the terminal in the form of a data packet.

[1050] Step 7:

[1051] The device provides the user with the rating and feedback it receives, either as audio output or text, and includes specific advice based on emotions.

[1052] Input: Ratings and feedback in the form of data packets.

[1053] Output: Feedback in the form of audio or text.

[1054] Specific operation: The device provides feedback to the user by playing the feedback aloud or displaying it as text on the screen.

[1055] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1056] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1057] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1058] [Fourth embodiment]

[1059] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1060] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1061] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1062] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1063] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1064] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1065] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1066] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1067] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1068] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1069] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1070] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1071] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1072] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[1073] 1. Initial settings and user information entry

[1074] A user first uses a user interface to access the system and enter their profile information, including basic information such as their name, desired career, and purpose of the interview. The terminal receives this information and transmits it to the server.

[1075] 2. Generating interview scenarios

[1076] The server uses a generation means to generate an appropriate Q&A scenario based on the received user information. This generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of a combination of questions and expected answers.

[1077] 3. Start practicing for the interview

[1078] Based on the received scenario, the device uses the voice output means to ask the user a question by voice. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer by using the voice input means. In this way, the question and answer process progresses.

[1079] 4. Sending and analyzing response data

[1080] The device sends the recorded audio data to a server, which then uses an evaluation tool to analyze the responses, including the appropriateness of the content, language, attitude, and pronunciation. Each element is scored and feedback is generated.

[1081] 5. Providing Feedback

[1082] The server sends the generated evaluation score and feedback to the device. The device receives the evaluation score and feedback and provides the user with feedback in the form of voice or text. For example, the device may display feedback such as, "Your self-introduction is not specific enough. Please add more specific examples."

[1083] Specific examples

[1084] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question may be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[1085] This system allows users to practice interviews in a realistic environment and also provides objective evaluations and specific areas for improvement, allowing users to fully prepare for the actual interview.

[1086] The processing flow will be explained below.

[1087] Step 1:

[1088] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal receives this information, converts it into data packets, and sends them to the server.

[1089] Step 2:

[1090] The server receives and analyzes the received user information. It then activates a generation means to generate an appropriate question and answer scenario based on the user's desired career and the purpose of the interview. The generation means selects questions using a past database and machine learning algorithms and generates a scenario. The generated scenario consists of pairs of questions and their expected answers. The server converts this scenario into data packets and sends them back to the terminal.

[1091] Step 3:

[1092] The terminal analyzes the received scenario data and outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the user's answer using the voice input means.

[1093] Step 4:

[1094] The terminal converts the recorded voice data into data packets and sends them to the server, where the user's voice response reaches the server.

[1095] Step 5:

[1096] The server launches an evaluation means to analyze the received voice data. The evaluation means uses a speech recognition algorithm to convert the voice data into text, and analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. The evaluation means assigns an evaluation score as a result of the analysis, and also generates feedback on the user's weaknesses and areas for improvement. This information is compiled as feedback data.

[1097] Step 6:

[1098] The server converts the generated evaluation score and feedback data into a data packet and sends it to the terminal. The terminal receives this data and provides the user with feedback in the form of voice or text. For example, the server might say, "Your self-introduction is not specific enough. Please add more specific examples."

[1099] Step 7:

[1100] The user can refine their answer based on the provided feedback, practice again using the refined answer, and the device will record a new answer and repeat the process from step 4 onwards.

[1101] Through these steps, users can engage in high-quality interview practice in a realistic environment and improve their skills through objective evaluation and specific feedback.

[1102] Example 1

[1103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1104] Traditional interview practice requires the time and effort of others to obtain third-party evaluation, making it difficult to conduct high-quality practice alone. Furthermore, there is a lack of a system for receiving immediate, objective, and specific feedback. This makes it difficult to maintain motivation for independent interview practice and achieve effective self-improvement.

[1105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1106] In this invention, the server includes a generation means that analyzes user information and generates an appropriate Q&A scenario using a generative AI model, an evaluation means that analyzes the acquired answers and generates an evaluation and feedback thereof, and a display means that provides the generated feedback to the user, thereby enabling the user to practice for a high-quality interview on their own and receive immediate, objective, and specific feedback.

[1107] "User interface means" refers to an interface that allows a user to input information into a system. Examples include web forms and application input screens.

[1108] "Communication Methods" refers to the ability to send and receive data between a device and a server, including HTTP requests over the Internet and other digital communication protocols.

[1109] "Generation means" refers to the function of analyzing user information and generating appropriate Q&A scenarios using a generative AI model, which uses past data and specific algorithms.

[1110] "Voice output means" refers to a function that outputs questions to the user by voice based on the generated scenario. Specific examples include speakers and voice synthesis technology.

[1111] "Voice input means" refers to a function for acquiring and recording a user's response in voice format. A specific example is a microphone.

[1112] "Evaluation means" refers to the function that analyzes the answers obtained and generates evaluations and feedback, using natural language processing and voice analysis technologies.

[1113] The "display means" refers to a function for providing the generated feedback to the user. Specific examples include a method of displaying text on a display or a method of playing back the feedback by voice.

[1114] The present invention is a system that allows users to practice interviews on their own and receive specific feedback. This system is implemented mainly using a terminal and a server.

[1115] 1. Initial settings and user information entry

[1116] First, a user accesses the system using a terminal. Using a user interface, the user inputs his / her profile information (e.g., name, desired career, purpose of interview, etc.). This information is sent from the terminal to the server.

[1117] 2. Generating interview scenarios

[1118] The server analyzes the received user information and generates an appropriate question and answer scenario using a generative AI model. The generative AI model operates based on past data and specific algorithms to select questions that match the user's desired career and the purpose of the interview. This scenario consists of a combination of questions and expected answers, and the generated scenario is sent to the device.

[1119] 3. Start practicing for the interview

[1120] The terminal uses the voice output means to ask the user a question based on the received scenario. For example, the question "Please introduce yourself" is output as a voice. The user answers this question by voice, and this answer is recorded by the voice input means.

[1121] 4. Sending and analyzing response data

[1122] The recorded voice data is sent from the device to a server. The server receives this data and uses evaluation tools to analyze the answers from various angles. Analysis criteria include appropriateness of content, language, attitude, pronunciation, etc. The server generates feedback based on these criteria.

[1123] 5. Providing Feedback

[1124] The server sends the generated feedback to the device, which then receives it and provides the user with feedback in the form of voice or text. For example, the feedback might say, "Your self-introduction is not specific enough. Please add more specific examples."

[1125] Specific examples

[1126] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" in their profile information, which is then sent to the server. Based on this information, the server generates a scenario containing questions related to sales positions and expected answers to those questions. For example, the server generates a question such as "Tell us about your work experience to date."

[1127] The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server, which analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides it to the user via the device.

[1128] Prompt Sentence Examples

[1129] Below are some example prompts using a generative AI model:

[1130] "Generate an interview scenario for a sales position. User profile information is as follows: name, desired career is sales, purpose of interview is job hunting."

[1131] This system allows users to practice interviews in a realistic environment and receive objective and specific feedback, helping them to fully prepare for the actual interview.

[1132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1133] Step 1:

[1134] A user accesses the system and uses the user interface to input their profile information (name, desired career, purpose of the interview), which is then acquired by the terminal and transmitted to the server via the communication means.

[1135] Input: User profile information

[1136] Data processing and calculation: Convert information into JSON format

[1137] Output: User information sent to the server

[1138] Step 2:

[1139] The server analyzes the received user information and generates an appropriate Q&A scenario using a generative AI model. The generative AI model uses past data and algorithms to create a set of questions and answers that are tailored to the user's desired career and interview objectives. The generated scenario is sent to the device via communication means.

[1140] Input: User information sent to the server

[1141] Data processing and calculation: Creating Q&A scenarios using generative AI models

[1142] Output: Sends the Q&A scenario (JSON format) to the terminal.

[1143] Step 3:

[1144] Based on the received scenario, the terminal uses the audio output means to ask the user a question by voice. For example, the question "Please introduce yourself" is played aloud. The user answers this question by voice, and the terminal records the answer by using the audio input means.

[1145] Input: Question and Answer scenario received by terminal

[1146] Data processing and calculation: Voice output of questions and answers, recording of user voice responses

[1147] Output: Recorded user voice response

[1148] Step 4:

[1149] The terminal encodes the recorded voice data using BASE64 and transmits it to the server via a communication means.

[1150] Input: Recorded user voice response

[1151] Data processing and calculation: BASE64 encoding of audio data

[1152] Output: The encoded audio data sent to the server.

[1153] Step 5:

[1154] The server decodes the received voice data and converts it into text using speech recognition technology. It then uses natural language processing (NLP) technology to evaluate the answers and generate feedback. Evaluation criteria include appropriateness of content, language, attitude, pronunciation, etc.

[1155] Input: Received encoded audio data

[1156] Data processing and calculation: Decoding voice data, speech recognition, analysis using NLP

[1157] Output: Generated feedback (in text format)

[1158] Step 6:

[1159] The server sends the generated feedback to the terminal. The terminal receives the feedback and provides it to the user in the form of voice or text via a display means. For example, a message may be played over the speaker saying, "Your self-introduction is not specific enough. Please add more specific examples."

[1160] Input: Feedback sent by the server

[1161] Data processing and calculation: Text-based feedback is displayed as voice or text

[1162] Output: Feedback provided to the user (audio or text)

[1163] This series of processes allows users to conduct high-quality interview practice on their own and receive objective and specific feedback.

[1164] (Application example 1)

[1165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1166] With conventional interview practice systems, it was difficult for users to conduct high-quality interview practice on their own, and it was also difficult to obtain specific feedback. In particular, when store staff practiced customer service interviews, there was a lack of a means to objectively evaluate the results of the practice and provide specific areas for improvement.

[1167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1168] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice synthesis means for outputting the generated scenario as voice, a voice recognition means for acquiring the user's answers in voice format, recording them and converting them into text format, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback, and a display means for providing the evaluation and feedback generated by the evaluation means to the user in voice or text format. This enables a user to conduct high-quality interview practice even by themselves and to receive specific feedback even in customer service training at a physical store.

[1169] "User interface means" refers to an interface through which a user inputs their information into the system.

[1170] The "generation means" is a means for analyzing user information and generating an appropriate question and answer scenario based on that information.

[1171] The "audio output means" is a means for outputting the generated scenario to the user by audio.

[1172] The "voice input means" is a means for obtaining the user's response in voice form.

[1173] The "evaluation means" is a means for analyzing the acquired voice data and generating an evaluation or feedback.

[1174] The "display means" is a means for providing the evaluation and feedback generated by the evaluation means to the user in audio or text format.

[1175] The "voice synthesis means" is a means for converting a text-format scenario into voice and outputting it.

[1176] A "voice recognition means" is a means for recording a user's response in voice form and converting it into text form.

[1177]

[1178] The present invention is a system that allows users to independently conduct high-quality interview practice and receive specific feedback. This system is composed of the following elements:

[1179] User Interface Means

[1180] The user first enters their profile information using the system's user interface means. At this stage, the user enters basic information such as their name, desired career, and purpose of the interview. This information is received by the terminal and transmitted to the server.

[1181] Scenario generation method

[1182] The server uses a generation means to generate an appropriate question and answer scenario based on the received user information. This generation means selects questions that are in line with the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario includes a combination of questions and expected answers.

[1183] Voice output means and voice synthesis means

[1184] The terminal uses the voice output means based on the generated scenario and utilizes the voice synthesis means to ask the user a question by voice, for example, "Please introduce yourself."

[1185] Voice input and voice recognition methods

[1186] The user answers the questions by voice, and the terminal records the answers using a voice input means. The recorded voice data is converted into text format by a voice recognition means.

[1187] Analysis of response data

[1188] The device sends the recorded and transcribed voice data to the server, which then uses an evaluation tool to analyze the answers, including the appropriateness of the content, the language used, the attitude, and the pronunciation, and assigns a score for each element to generate feedback.

[1189] Providing feedback

[1190] The server transmits the generated evaluation score and feedback to the terminal. The terminal receives the evaluation score and feedback and provides the feedback to the user in the form of voice or text using a display means. For example, feedback such as "Your self-introduction is not specific enough. Please add more specific examples" is displayed as voice or text.

[1191] Specific examples

[1192] Let's say the user is aiming to become a store staff member, and the purpose of the interview is job hunting. The user enters "store staff member" and "job hunting" as their profile information. The server generates a scenario containing questions related to store staff members and expected answers. For example, a question might be generated such as "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud with "I worked as a store staff member for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and provides this to the user via the device.

[1193] Prompt Sentence Examples

[1194] Example prompt: "Generate an interview scenario for a store associate."

[1195] This allows users to practice interviews in an environment that is close to the real thing, and also provides objective evaluations and specific areas for improvement.The system utilizes natural language processing and speech recognition technology to provide an advanced interview practice environment.

[1196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1197] Step 1:

[1198] The user inputs his / her profile information using the user interface means. At this stage, the user inputs basic information such as his / her name, desired career, and purpose of the interview. The input data is transmitted to the terminal.

[1199] Step 2:

[1200] The device sends the received user information to the server, which analyzes the user information and uses a generation method to generate an appropriate question and answer scenario. In doing so, the server utilizes past data and algorithms to select questions that are in line with the user's desired career and the purpose of the interview.

[1201] Step 3:

[1202] The scenario generated by the server is sent to the terminal. The terminal uses a speech synthesis means to convert the generated scenario into a voice format. An audio file is generated, and the questions in the scenario are output to the user by voice.

[1203] Step 4:

[1204] The user answers questions by voice, and the terminal records the user's answers using a voice input means. This voice data is converted into text format by a voice recognition means.

[1205] Step 5:

[1206] The device then transmits the converted response data to the server, which then uses an evaluation tool to analyze the response. The analysis includes the appropriateness of the response, the language used, the attitude, and the pronunciation, and assigns a score to each element.

[1207] Step 6:

[1208] The server generates an evaluation and feedback based on the analysis results, and the generated feedback and evaluation points are sent to the device.

[1209] Step 7:

[1210] The terminal uses the display means to provide the user with evaluation and feedback in the form of voice or text, for example, by displaying specific advice such as "Your answer is not specific enough. Please add more specific examples."

[1211] Step 8:

[1212] Users can receive feedback and review their practice interviews, providing specific improvements to incorporate into their next practice.

[1213] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1214] This system allows users to practice interviews independently and receive specific feedback. In particular, it combines an emotion engine that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, enabling a more comprehensive interview evaluation. This system is comprised of the following components:

[1215] 1. Initial settings and user information entry

[1216] A user first uses a user interface means to access the system and enter their profile information. The user enters basic information such as their name, desired career, and purpose of the interview. The terminal receives this information, converts it into data packets, and sends them to the server.

[1217] 2. Generating interview scenarios

[1218] Based on the received user information, the server uses a generation means to generate an appropriate question and answer scenario. The generation means selects questions based on the user's desired career and the purpose of the interview based on past data and algorithms. The generated scenario consists of pairs of questions and expected answers. The server converts this scenario into data packets and sends them to the terminal.

[1219] 3. Start practicing for the interview

[1220] Based on the received scenario, the device outputs a question to the user by voice using the voice output means. For example, the question "Please introduce yourself" is output by voice. The user answers this question by voice, and the device records the answer using the voice input means. This is how the question and answer process progresses.

[1221] 4. Sending and analyzing response data

[1222] The device converts the recorded voice data into data packets and sends them to the server. The server receives this data and analyzes the answers using an evaluation tool. At the same time, an emotion engine is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice responses.

[1223] 5. Evaluation and feedback generation

[1224] The server analyzes the voice data through the evaluation means and evaluates the appropriateness of the response, pronunciation, wording, attitude, etc. Furthermore, it generates an evaluation score taking into account the user's emotional state recognized by the emotion engine. This allows for a more comprehensive evaluation. The generated evaluation score and feedback are packaged as a data packet and sent to the terminal.

[1225] 6. Providing Feedback

[1226] The device analyzes the received evaluation scores and feedback data and provides the user with feedback in the form of voice or text. For example, in addition to feedback such as "Your self-introduction is not specific enough. Please add more specific examples," it also provides emotion-based advice such as "You seem a little nervous. Try to relax."

[1227] Specific examples

[1228] Let's say the user is aiming for a sales position and the purpose of the interview is job hunting. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers. For example, a question might be generated: "Tell us about your work experience to date." The device outputs this question aloud, and the user responds aloud: "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as "It would be good to mention your role and specific achievements," and further provides advice based on the analysis results of the emotion engine: "You seem tense in your voice. Take a deep breath and try to relax."

[1229] This system allows users to practice interviews in a realistic environment and receive comprehensive evaluations, specific suggestions for improvement, and emotional support, helping them to fully prepare for the actual interview.

[1230] The processing flow will be explained below.

[1231] Step 1:

[1232] A user accesses the service and uses a user interface to input their profile information (such as name, desired career, purpose of the interview, etc.) The terminal collects this information, converts it into data packets, and sends them to the server.

[1233] Step 2:

[1234] The server receives and analyzes the received user information. Based on the analyzed information, it activates a generation means to generate an appropriate question and answer scenario. The generation means uses algorithms and pattern recognition technology to select questions that are in line with the user's desired career and the purpose of the interview, and generates a scenario. The generated scenario is sent to the terminal as a data packet.

[1235] Step 3:

[1236] The terminal analyzes the received scenario data and asks the user questions aloud using the voice output means. For example, a question such as "Please tell us about your work experience to date" is played aloud. The user answers the questions aloud, and the terminal records the answers using the voice input means.

[1237] Step 4:

[1238] The terminal converts the recorded user's answer data into a data packet and sends it to the server, whereupon the user's voice answer reaches the server.

[1239] Step 5:

[1240] The server activates an evaluation means to analyze the received voice data, which converts the voice data into text using a speech recognition algorithm and analyzes the appropriateness of the content, pronunciation, wording, attitude, etc.

[1241] Step 6:

[1242] The server activates an emotion engine to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. The emotion engine analyzes the tone, speed, and pitch of the voice to identify the emotion.

[1243] Step 7:

[1244] The evaluation means generates a comprehensive evaluation score based on the analysis results of the obtained answers and the emotional state recognized by the emotion engine, and simultaneously generates specific feedback and emotion-based advice based on the user's evaluation.

[1245] Step 8:

[1246] The server converts the generated evaluation score and feedback data into a data packet and transmits it to the terminal.

[1247] Step 9:

[1248] The device analyzes the evaluation scores and feedback data it receives and provides the user with feedback in the form of voice or text. For example, it may provide feedback such as, "It would be good to mention your role and specific achievements," or emotional advice such as, "You sound tense. Take a deep breath and relax."

[1249] Step 10:

[1250] The user can use the provided feedback and advice to improve their answer next time and practice again. The device will record a new voice answer and repeat the process from step 4 onwards.

[1251] Through this series of steps, users can practice interviews in a realistic environment and receive comprehensive evaluations and feedback that take into account their emotional state, helping them to fully prepare for the actual interview.

[1252] Example 2

[1253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1254] In modern interview practice, it is difficult to achieve high-quality training, especially when done alone. Furthermore, conventional interview practice systems often do not provide specific feedback on the user's answers or evaluations that reflect the user's emotional state. As a result, users are unable to accurately identify their own problems and are unable to make effective improvements.

[1255] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1256] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for recognizing emotions from the user's voice and generating an evaluation and feedback based on the emotions. This allows the user to effectively practice for an advanced interview even on their own, and to receive a comprehensive evaluation that reflects specific feedback and their emotional state.

[1257] "User interface means" means a device or software that allows a user to access the system and input required information.

[1258] The "generator" is a device or software that creates an appropriate question and answer scenario based on information provided by the user.

[1259] The "audio output means" is a device or software for outputting questions to the user by voice based on the generated scenario.

[1260] "Voice input means" refers to a device or software for obtaining a user's voice response.

[1261] The "evaluation means" is a device or software that analyzes the user's answers obtained, evaluates their appropriateness, pronunciation, wording, attitude, etc., and generates feedback.

[1262] The "emotion analysis means" is a device or software for recognizing emotions from the user's voice and generating evaluations and feedback based on those emotions.

[1263] A "display means" is a device or software for providing the feedback generated by the evaluation means to the user in audio or text form.

[1264] This system allows a single user to practice interviews with high quality and receive specific feedback. In particular, by combining it with an emotion analysis tool that recognizes emotions from the user's voice and provides evaluations and feedback based on those emotions, a more comprehensive interview evaluation can be achieved. This system is composed of the following elements:

[1265] Initial settings and user information entry

[1266] First, a user accesses the system using a terminal connected to the Internet. They enter their profile information (such as their name, desired occupation, and purpose of the interview) through a dedicated user interface. For example, they enter information such as "I'm looking for a sales position, and the purpose of the interview is job hunting." The entered information is converted into data packets by the terminal and sent to the server.

[1267] Interview scenario generation

[1268] The server analyzes the user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario that matches the user's desired career and the purpose of the interview. For example, a question such as "Please tell us about your work experience to date" is generated as a question about a sales position. The generated scenario is converted into a data packet and sent to the device.

[1269] Interview practice begins

[1270] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the audio output means, it outputs questions based on the scenario by voice. For example, a question such as "Please introduce yourself" is presented to the user by voice. The user answers this question by voice, and the terminal records the answer using the audio input means.

[1271] Sending and analyzing response data

[1272] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data, converts the voice content into text using a natural language processing algorithm, and analyzes the response. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (tension, joy, confidence, etc.) from the voice response.

[1273] Generate ratings and feedback

[1274] The server analyzes the appropriateness of the answers, pronunciation, wording, attitude, etc. using evaluation means. Furthermore, the user's emotional state, as recognized by emotion analysis means, is also included in the evaluation. This generates an overall evaluation score and specific feedback. For example, the server may provide feedback such as "It would be good to add a specific anecdote" or emotion-based advice such as "Your voice sounds tense, so try relaxing."

[1275] Providing feedback

[1276] The server converts the generated evaluation score and feedback into a data packet and sends it to the device. The device analyzes the received feedback data and provides the user with feedback in voice or text format. For example, the device might provide advice based on emotions, such as, "Your self-introduction lacks specificity. Please add more specific examples," or, "You seem a little nervous. Take a deep breath and try to relax."

[1277] Specific examples

[1278] For example, let's take a specific example where a user is aiming for a sales position. The user enters "sales position" and "job hunting" as their profile information. The server generates a scenario containing questions related to sales positions and expected answers to those questions. The generated scenario includes the question, "Tell us about your work experience to date." The device outputs this question aloud, and the user answers aloud, "I worked as a sales assistant for three years..." This answer is recorded by the device and sent to the server. The server analyzes the answer and generates feedback such as, "It would be good to mention specific roles and achievements." Furthermore, based on the analysis results of the emotion engine, advice is provided such as, "Your voice sounds tense. Take a deep breath and try to relax."

[1279] As described above, the present invention allows users to effectively practice for interviews on their own and receive comprehensive evaluations, specific feedback, and emotional support, thereby enabling users to fully prepare for the actual interview.

[1280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1281] Step 1:

[1282] Initial settings and user information entry

[1283] Users access the system using a terminal connected to the Internet. Through a dedicated user interface, they input profile information such as their name, desired occupation, and purpose of the interview (e.g., sales, job hunting). The terminal converts the input information into data packets and sends them to the server, which then receives the user's profile information.

[1284] Input: User profile information (name, occupation, purpose of interview)

[1285] Output: Data packet (including user information)

[1286] Step 2:

[1287] Interview scenario generation

[1288] The server receives and analyzes user information sent from the device. Using a generative AI model, it generates an appropriate question and answer scenario tailored to the user's desired career and the purpose of the interview (e.g., questions about sales positions). The generated scenario is converted into a data packet and sent to the device. The server selects questions based on past data and algorithms, and also creates sample answers.

[1289] Input: Data packet (user information)

[1290] Output: Data packet (generated scenario)

[1291] Step 3:

[1292] Interview practice begins

[1293] The terminal analyzes the scenario data packet received from the server and reads the scenario. Using the voice output means, it outputs a question based on the scenario to the user by voice (e.g., "Please introduce yourself"). The user answers this question by voice, and the terminal records the answer using the voice input means. As a result, the user's speech is saved in the system.

[1294] Input: Data packet (scenario)

[1295] Output: Audio data (user's answer)

[1296] Step 4:

[1297] Sending and analyzing response data

[1298] The device converts the recorded voice data into data packets and sends them to the server. The server receives the voice data and converts the voice content into text using a natural language processing algorithm, which then analyzes it. At the same time, an emotion analysis unit is activated to recognize the user's emotional state (e.g., nervousness, joy, confidence, etc.) from the user's voice response. This allows both the content of the response and the user's emotional state to be analyzed.

[1299] Input: Voice data (user response)

[1300] Output: Text data (analyzed answers), emotion data

[1301] Step 5:

[1302] Generate ratings and feedback

[1303] The server uses an evaluation tool to analyze the appropriateness of the response, pronunciation, wording, and attitude based on the text data. Furthermore, the server also incorporates the user's emotional state, as recognized by the emotion analysis tool, into the evaluation. This generates an overall evaluation score and specific feedback (e.g., adding a specific anecdote would be helpful). The generated evaluation score and feedback are converted into a data packet and sent to the device.

[1304] Input: Text data (answer content), emotion data

[1305] Output: Data packet (ratings and feedback)

[1306] Step 6:

[1307] Providing feedback

[1308] The device analyzes the received data packets and provides feedback to the user in voice or text format (e.g., your self-introduction is not specific enough). In addition to specific areas for improvement, emotion-based advice is also provided (e.g., take a deep breath to relax). This allows users to receive practical feedback.

[1309] Input: Data packet (ratings and feedback)

[1310] Output: Feedback (audio or text)

[1311] (Application example 2)

[1312] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1313] Conventional interview practice systems have difficulty understanding the user's own emotions and state of tension, and providing appropriate feedback and advice. Furthermore, the training process for new staff at logistics centers can sometimes delay the acquisition of skills that will enable them to be immediately effective. This can lead to issues such as reduced efficiency throughout the center.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1315] In this invention, the server includes a user interface means for inputting user information, a generation means for analyzing the user information and generating an appropriate question and answer scenario, a voice output means for outputting questions to the user by voice based on the generated scenario, a voice input means for acquiring the user's answers in voice form, an evaluation means for analyzing the acquired answers and generating an evaluation and feedback thereon, and an emotion analysis means for analyzing the user's emotional state and providing an evaluation and feedback based on the analysis results. This makes it possible to provide specific advice based on the user's emotional state, allowing new staff at a logistics center to effectively hone their skills and become immediately effective.

[1316] 1. "User interface means" refers to an interface through which a user inputs or obtains information into a system.

[1317] 2. "Generation means" refers to the means for analyzing input user information and creating an appropriate question and answer scenario based on that information.

[1318] 3. "Voice output means" refers to a means for outputting questions to the user by voice based on the generated scenario.

[1319] 4. "Voice input means" refers to the means for obtaining a user's response in voice form and inputting it into the system.

[1320] 5. "Evaluation means" refers to the means for analyzing a user's voice response, evaluating its content and appropriateness, and generating feedback.

[1321] 6. "Emotion analysis means" refers to a means for analyzing the emotional state of a user from their voice response and providing evaluation and feedback based on the analysis results.

[1322] 7. "Feedback" refers to information including evaluation of the user's answers, suggestions for improvement, and advice.

[1323] 8. "Logistics center" refers to a facility where goods and materials are stored, sorted, and prepared for delivery.

[1324] 9. "New staff" refers to employees newly hired at the logistics center.

[1325] 10. "Immediately effective" refers to having techniques and skills that can be immediately used in actual work.

[1326] A system for realizing this application example will be described in detail below.

[1327] System configuration

[1328] First, the interview training system for new staff at the logistics center consists of the following main components:

[1329] 1. User Interface Methods

[1330] 2. Generation means

[1331] 3. Audio output means

[1332] 4. Voice input method

[1333] 5. Evaluation Methods

[1334] 6. Emotion analysis method

[1335] User Interface Means

[1336] The user interface means is an interface that allows staff to access the system and input basic information such as their name, the work they will be responsible for, past experience, etc. For example, a smartphone or PC application can be used.

[1337] generation means

[1338] The generation method analyzes the input staff information and creates appropriate Q&A scenarios based on that information. This method uses past data and algorithms. Specifically, it uses natural language generation technology to automatically generate questions related to the staff member's work.

[1339] Audio output means

[1340] The voice output means outputs a question to the staff member by voice based on the generated scenario. For example, the voice output function of a smartphone can be used to voice the question, "Do you have experience in inventory management?"

[1341] Voice input means

[1342] The voice input method receives the staff's response in voice format and inputs it into the system. The voice data is recorded as is and sent to the server. For example, the voice is converted into text using the Google Speech-to-Text API.

[1343] Evaluation methods

[1344] The evaluation means analyzes the answer data acquired on the server side and evaluates its content and appropriateness. The content, pronunciation, and wording of the answer are evaluated here. Feedback is also generated based on the evaluation results.

[1345] Emotion analysis means

[1346] The emotion analysis means analyzes the emotional state (e.g., nervousness, confidence, etc.) of staff members' voice responses and provides evaluation and feedback based on the analysis results. For example, IBM Watson Tone Analyzer API is used to recognize emotions from voice data.

[1347] Program processing

[1348] In this system, a staff member first inputs basic information using a smartphone or PC. The server receives that information, and a generation means generates an appropriate question-and-answer scenario. Based on the generated scenario, a voice output means outputs questions to the staff member by voice, and the staff member answers by voice. A voice input means acquires the answers and sends them to the server. An evaluation means then analyzes the answers, and emotions are also analyzed by an emotion analysis means. Finally, feedback is generated based on the results of the evaluation and emotion analysis and provided to the staff member.

[1349] Specific examples

[1350] For example, if a new staff member at a distribution center is in charge of "inventory management," the following questions might be generated:

[1351] 1. Enter staff information

[1352] Name: Taro

[1353] Responsibilities: Inventory management

[1354] Experience: 2 years of inventory management experience

[1355] 2. Generated Questions

[1356] "Explain the main steps in inventory management."

[1357] "Tell us about a difficult situation you've had with inventory management in the past and how you dealt with it."

[1358] 3. Staff responses and feedback

[1359] Staff members answer questions and the answers are sent to the server.

[1360] The server analyzes the answers and generates feedback such as "Please explain the specific steps in more detail" and emotion-based advice such as "Your voice sounds tense. Relax."

[1361] Prompt Sentence Examples

[1362] Example prompts to be input to the generative AI model:

[1363] "Encourage candidates to describe in detail their inventory management experience. See if they use specific examples to explain the procedures and their role. Also, assess their emotional state (tension, confidence, etc.) and provide feedback accordingly."

[1364] In this way, the present invention allows new staff at a logistics center to effectively hone their skills so that they can be immediately put to work through practical interview training.

[1365] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1366] Step 1:

[1367] A user uses a smartphone or a PC to input basic information (such as name, work to be performed, past experience, etc.) through a user interface. The input information is converted into a data packet and sent to the server.

[1368] Input: Basic information such as name, expected duties, past experience, etc.

[1369] Output: Basic information about the data packet format.

[1370] Specific behavior: A user accesses the application, enters information into the designated fields, and then presses the "Submit" button to send the information to the server.

[1371] Step 2:

[1372] The server receives the basic information sent by the user and generates an appropriate question and answer scenario using a generation means. The generated scenario consists of a list of questions and answers, which are converted into data packets again and sent to the terminal.

[1373] Input: Basic information in data packet format.

[1374] Output: Question and answer scenario in data packet format.

[1375] Specific operation: The generation means uses natural language generation technology to automatically select relevant questions based on the input information and compile them into a scenario.

[1376] Step 3:

[1377] The terminal receives the scenario and outputs a question to the user by voice using the voice output means. For example, a question such as "Do you have experience in inventory management?" is presented by voice.

[1378] Input: A question and answer scenario in the form of a data packet.

[1379] Output: Question in audio format.

[1380] Specific operation: The generated question is played aloud using the device's voice output function, and the user listens to it and prepares to answer the question.

[1381] Step 4:

[1382] The user answers questions by voice, and the voice input means captures the answers, records them, converts them into data packets, and sends them to the server.

[1383] Input: The user's spoken response.

[1384] Output: Voice response in the form of data packets.

[1385] Specific operation: The user answers questions using a microphone, and the answers are recorded and sent to the server in the form of data packets.

[1386] Step 5:

[1387] The server analyzes the received voice data and converts it into text using the Google Speech-to-Text API. It also uses the IBM Watson Tone Analyzer API to analyze the emotional state of staff. The analysis results are compiled as evaluation data, including the content of responses and emotional state.

[1388] Input: Voice response in the form of a data packet.

[1389] Output: Text response data and sentiment analysis data.

[1390] Specific operation: The server converts the voice data into text and then uses a sentiment analysis engine to analyze the user's emotional state. Evaluation data is generated as the analysis result.

[1391] Step 6:

[1392] The server generates a rating and feedback based on the analysis, including specific advice on the appropriateness of the answer, pronunciation, wording, and appropriate responses. The rating and feedback are converted into a data packet and sent to the device.

[1393] Input: Text response data and sentiment analysis data.

[1394] Output: Evaluation and feedback in the form of a data packet.

[1395] Specific operation: The server's evaluation means creates specific feedback based on the analysis results and sends it to the terminal in the form of a data packet.

[1396] Step 7:

[1397] The device provides the user with the rating and feedback it receives, either as audio output or text, and includes specific advice based on emotions.

[1398] Input: Ratings and feedback in the form of data packets.

[1399] Output: Feedback in the form of audio or text.

[1400] Specific operation: The device provides feedback to the user by playing the feedback aloud or displaying it as text on the screen.

[1401] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1402] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1403] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1404] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1405] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1406] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1407] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1408] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1409] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1410] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1411] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1412] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1413] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1414] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1415] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1416] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1417] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1418] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1419] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1420] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1421] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1422] The following is further disclosed regarding the above embodiment.

[1423] (Claim 1)

[1424] a user interface means for inputting user information;

[1425] a generating means for analyzing user information and generating an appropriate question and answer scenario;

[1426] a voice output means for outputting a question to a user by voice based on the generated scenario;

[1427] a voice input means for acquiring a user's response in voice form;

[1428] The system includes an evaluation means for analyzing the responses obtained and generating an evaluation and feedback thereon.

[1429] (Claim 2)

[1430] 2. The system according to claim 1, wherein the evaluation means evaluates the appropriateness, pronunciation, and wording of the user's answer content and generates feedback.

[1431] (Claim 3)

[1432] 10. The system of claim 1, further comprising display means for providing the feedback generated by the evaluation means to the user in audio or text form.

[1433] "Example 1"

[1434] (Claim 1)

[1435] a user interface means for inputting user information;

[1436] a communication means for transmitting the input user information to the processing device;

[1437] A generation means for analyzing user information and generating an appropriate Q&A scenario using a generation AI model;

[1438] a voice output means for outputting a question by voice based on the generated scenario;

[1439] a voice input means for acquiring and recording a user's response in voice form;

[1440] a communication means for transmitting the acquired answer to the processing device;

[1441] an evaluation means for analyzing the responses and generating a rating and feedback thereon;

[1442] The system includes a display means for providing the generated feedback to the user.

[1443] (Claim 2)

[1444] 2. The system according to claim 1, wherein the evaluation means evaluates the appropriateness of the user's answer content, pronunciation, wording, and attitude, and generates feedback.

[1445] (Claim 3)

[1446] 10. The system of claim 1, further comprising display means for providing the feedback generated by the evaluation means to the user in audio or text form.

[1447] "Application Example 1"

[1448] (Claim 1)

[1449] a user interface means for inputting user information;

[1450] a generating means for analyzing user information and generating an appropriate question and answer scenario;

[1451] a voice output means for outputting a question to a user by voice based on the generated scenario;

[1452] a voice input means for acquiring a user's response in voice form;

[1453] evaluation means for analyzing the obtained answers and generating an evaluation and feedback thereon;

[1454] a display means for providing the user with the ratings and feedback generated by the rating means in audio or text form;

[1455] a voice synthesis means for outputting the scenario generated by the voice output means in voice form;

[1456] A system in which the voice input means includes a voice recognition means for recording the user's responses and converting them into text form.

[1457] (Claim 2)

[1458] 10. The system of claim 1, wherein the evaluation means provides the generated feedback to the user in audio or text format.

[1459] (Claim 3)

[1460] 2. The system according to claim 1, wherein the evaluation means evaluates the appropriateness, pronunciation, and wording of the user's answer content and generates feedback.

[1461] "Example 2: Combining Emotion Engines"

[1462] (Claim 1)

[1463] a user interface means for inputting user information;

[1464] a generating means for analyzing user information and generating an appropriate question and answer scenario;

[1465] a voice output means for outputting a question to a user by voice based on the generated scenario;

[1466] a voice input means for acquiring a user's response in voice form;

[1467] evaluation means for analyzing the obtained answers and generating an evaluation and feedback thereon;

[1468] A system including an emotion analysis means for recognizing emotions from a user's voice and generating evaluations and feedback based on the emotions.

[1469] (Claim 2)

[1470] 2. The system according to claim 1, wherein the evaluation means evaluates the appropriateness of the user's answer content, pronunciation, wording, and attitude, and generates feedback.

[1471] (Claim 3)

[1472] 10. The system of claim 1, further comprising display means for providing the feedback generated by the evaluation means to the user in audio or text form.

[1473] "Application example 2 when combining emotion engines"

[1474] (Claim 1)

[1475] a user interface means for inputting user information;

[1476] a generating means for analyzing user information and generating an appropriate question and answer scenario;

[1477] a voice output means for outputting a question to a user by voice based on the generated scenario;

[1478] a voice input means for acquiring a user's response in voice form;

[1479] evaluation means for analyzing the obtained answers and generating an evaluation and feedback thereon;

[1480] A system including an emotion analysis means for analyzing a user's emotional state and providing evaluation and feedback based on the analysis results.

[1481] (Claim 2)

[1482] 2. The system according to claim 1, wherein the evaluation means evaluates the appropriateness, pronunciation, and wording of the user's answer content and generates feedback.

[1483] (Claim 3)

[1484] 10. The system of claim 1, further comprising display means for providing the feedback generated by the evaluation means to the user in audio or text form, and further providing specific advice based on the user's emotional state. [Explanation of symbols]

[1485] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a user interface means for inputting user information; a generating means for analyzing user information and generating an appropriate question and answer scenario; a voice output means for outputting a question to a user by voice based on the generated scenario; a voice input means for acquiring a user's response in voice form; The system includes an evaluation means for analyzing the responses obtained and generating an evaluation and feedback thereon.

2. 2. The system according to claim 1, wherein the evaluation means evaluates the appropriateness, pronunciation, and wording of the user's answer content and generates feedback.

3. 10. The system of claim 1, further comprising display means for providing the feedback generated by the evaluation means to the user in audio or text form.

Images