System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input data of an entry sheet or a resume; means for analyzing the input data and automatically generating a question with an interviewer; means for the user to input an answer to the generated question; means for converting the user's answer into voice and text data; means for analyzing the converted data and performing evaluation; means for generating individual feedback based on an evaluation result; and means for presenting the generated feedback to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the past, recruitment agencies faced the problem of requiring a great deal of time and effort when providing individual support to large numbers of job-seekers and job-changers. Providing highly accurate support that takes into account each applicant's personality and abilities was particularly difficult, creating a demand for efficient guidance methods. Another issue was the difficulty of receiving specific, individual feedback that would help applicants identify and improve their own shortcomings when preparing for interviews. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system including: a means for a user to input data from an application form or resume; a means for analyzing the input data and automatically generating questions for an interviewer; a means for the user to input answers to the generated questions; a means for converting the user's answers into voice and text data; a means for analyzing the converted data and conducting an evaluation; a means for generating individual feedback based on the evaluation results; and a means for presenting the generated feedback to the user.

[0006] This system automatically analyzes user input and response data to generate appropriate interviewers and questions, thereby reducing the burden on recruitment agencies. Furthermore, by providing feedback based on evaluations of user responses using natural language processing and voice analysis, applicants can efficiently identify problems and obtain specific improvement measures. This allows applicants to more effectively prepare for interviews.

[0007] An "entry sheet" is a document in which applicants write their profile and reasons for applying when applying for a job or changing jobs, and is used when applying to a company.

[0008] A "resume" is a document that lists information such as an applicant's name, address, educational background, work history, qualifications, and self-introduction, and is submitted to a company.

[0009] "Analysis" is the process of extracting and understanding information based on input data, and in the present invention is a process for evaluating the contents of application forms and resumes submitted by users.

[0010] An "interviewer" is a person whose role is to ask questions of applicants during job or career interviews and evaluate their aptitude and abilities.

[0011] "Questions" are questions that interviewers ask applicants in order to evaluate the applicant's abilities, aptitude, and motivation for applying.

[0012] An "answer" is a response given by an applicant to a question posed by an interviewer, and is given using language or voice.

[0013] "Voice data" is a digital recording of speech that is used in the present invention to record a user's responses.

[0014] "Text data" is character information recorded in digital format, and is used in the present invention to analyze user responses as character information.

[0015] "Natural language processing" is a technology that allows computers to understand and process human language, and is used for text analysis, translation, generation, etc.

[0016] "Evaluation" is the process of assigning a score or making a judgment based on the content and quality of the user's answer, and in the present invention, this is done using voice analysis and natural language processing.

[0017] "Feedback" refers to advice and suggestions for improvement provided based on the evaluation results of the user's answers, and serves as a guide for the user to improve themselves.

[0018] A "system" is a series of components in which multiple devices and programs work together to achieve a specific function, and in this invention, it is the system that works with the interviewer to generate questions, analyze answers, and provide feedback. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that utilizes AI to automatically generate interviewers and questions for job seekers and those looking to change jobs, evaluate the answers, and provide individual feedback. This system includes the following means.

[0041] Entering application forms and resumes

[0042] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[0043] Interviewer and Question Generation

[0044] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[0045] User responses and data conversion

[0046] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[0047] Analysis and evaluation of responses

[0048] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the evaluation results, the server assigns a score to the user's answers.

[0049] Generating and Presenting Feedback

[0050] The server generates individualized feedback based on the scoring results. The feedback includes points about the user's answers that are good and that need improvement, as well as suggestions for specific improvements and other ways of expressing the answers. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[0051] Specific examples

[0052] As a concrete example, consider a case where the company the user is applying to is an IT company and the user has experience in project management. The user details their project management experience on an application form and sends it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized.

[0053] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[0057] Step 2:

[0058] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[0059] Step 3:

[0060] The terminal transmits the user's input data to the server.

[0061] Step 4:

[0062] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[0063] Step 5:

[0064] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[0065] Step 6:

[0066] The server sends the generated interviewer settings and questions to the terminal.

[0067] Step 7:

[0068] The terminal displays the received interviewer settings and questions to the user.

[0069] Step 8:

[0070] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[0071] Step 9:

[0072] The terminal converts the user's voice response into text data.

[0073] Step 10:

[0074] The terminal transmits the converted text data and the original voice data to the server.

[0075] Step 11:

[0076] The server analyzes the received voice and text data.

[0077] Specifically, natural language processing technology is used to analyze the content of the responses, and voice analysis technology is used to detect voice intonation, pauses, and slip-ups.

[0078] Step 12:

[0079] The server evaluates and scores the user's answers based on the analysis results.

[0080] Step 13:

[0081] The server generates personalized feedback based on the scoring results, including points of merit, areas for improvement, and suggestions for specific improvements or other ways of expressing them.

[0082] Step 14:

[0083] The server transmits the generated feedback to the terminal.

[0084] Step 15:

[0085] The terminal displays the received feedback to the user.

[0086] Step 16:

[0087] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] In traditional job hunting and career change processes, when job seekers individually prepare for interviews, they lack concrete methods for determining appropriate answers to interviewer questions and evaluating the quality of those answers. Furthermore, it is difficult to obtain the feedback needed to improve interview techniques. This makes it difficult for job seekers to effectively showcase their skills.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for a user to input data for an application form or resume, means for analyzing the input data and automatically generating interviewer and questions, means for the user to input answers to the generated questions, means for converting the user's voice data into text data, means for analyzing the converted data and performing an evaluation, means for generating individual feedback based on the evaluation results, and means for presenting the generated feedback to the user. This allows the user to improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[0093] "User" refers to an individual who uses this system to input application form and resume data and simulate an interview.

[0094] An "entry sheet" is a document used to submit personal information to a company when seeking employment, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[0095] A resume is a document that details a job seeker's career history and skills, and is submitted along with an application form.

[0096] The "server" is a central computer system that receives and analyzes application and resume data, generates interviewers and questions, evaluates user response data, and provides feedback.

[0097] A "terminal" is a computer or smart device that allows a user to enter data for an application form or resume, answer generated questions, and receive feedback.

[0098] "Analysis" is the process in which the server processes data from application forms and resumes to identify the company and industry the user is applying to, and then generates interviewers and questions accordingly.

[0099] The "interviewer" is a virtual interviewer that the server generates based on the company and industry that the user is applying to, and who asks appropriate questions.

[0100] The "questions" are generated by the server based on the data in the user's application form or resume, and are part of a virtual interview, similar to the interviewer.

[0101] "Voice data" refers to data generated by voice when a user answers a question using a microphone.

[0102] "Text data" is voice data converted into character information, and is data used by the server for analysis and evaluation.

[0103] "Natural language processing technology" refers to technology used by computers to understand, analyze, and generate human language, including generative AI models such as GPT-4.

[0104] "Scoring" is a process in which the server analyzes the content of the user's answer and then numerically evaluates the quality and appropriateness of the answer.

[0105] "Feedback" is generated by the server based on the scoring results, and includes good points and areas for improvement in the user's answer, specific improvement measures, and other ways of expressing them.

[0106] The present invention is a system for supporting job hunting and career change, and is designed to allow users to input data from application forms and resumes, analyze the data, automatically generate interviewers and questions, evaluate the answers, and provide individual feedback. This system includes the following means.

[0107] System configuration

[0108] This system consists of a terminal used by the user, a server that receives and analyzes data, and software required for data conversion and analysis.

[0109] Entering application forms and resumes

[0110] The user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. When the user enters the required information in each field and presses the send button, the data is sent to the server.

[0111] Data analysis and interviewer and question generation

[0112] The server receives the application form and resume data sent by the user and stores it in a database. The stored data is then analyzed, using natural language processing technology to identify the company and industry the user is applying to.

[0113] The server generates an interviewer based on the identified company and industry. This interviewer is a virtual persona that generates questions appropriate for the specific industry or company. To generate questions, a generative AI model (e.g., GPT-4) is used.

[0114] Posting questions and accepting answers

[0115] The generated question is displayed to the user through the device. The user answers the question by voice using the device's microphone. Once the answer is complete, the voice data is sent to the server by pressing the send button on the device.

[0116] Audio data conversion

[0117] The device converts the user's voice data into text data, for example, using voice recognition software (e.g., Google Cloud Speech-to-Text API), and the text data is sent to the server along with the voice data.

[0118] Analysis and evaluation of responses

[0119] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the analysis results, the server assigns a score to the user's answers.

[0120] Generating and Presenting Feedback

[0121] The server generates individual feedback based on the scoring results. The feedback includes points of merit and improvement for the user's answer, as well as suggestions for specific improvements and other ways of expressing the answer. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[0122] Specific examples

[0123] As a specific example, suppose a user is applying to an information technology company and has experience in project management. The user details their project management experience in an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides specific areas for improvement and points that should be emphasized as feedback.

[0124] Prompt Sentence Examples

[0125] Examples of prompts are:

[0126] "Based on the project management experience stated in the application form, what questions should a user generate when attending an interview?"

[0127] This allows the system to support users in more effectively preparing for interviews, ultimately helping them learn how to present themselves to companies in the right way.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1:

[0130] The user uses a terminal to input data for an application form and resume. A form containing fields such as name, address, educational background, work history, skills, and reasons for applying is displayed on the terminal. When the user enters the required information in each field and presses the send button, the data is sent to the server. The input data includes the user's personal information and work history, and the output is the application data sent to the server.

[0131] Step 2:

[0132] The server receives the application form and resume data sent from the terminal. It stores the received data in a database. The input is the application data sent from the terminal, which is then stored in the database, and the output is the user information stored in the database.

[0133] Step 3:

[0134] The server retrieves the user's application form and resume data from the database. The server analyzes the retrieved data and identifies the company and industry the user is applying to. The input is the application data retrieved from the database, which is analyzed using natural language processing technology. The output is the analysis result, which identifies the company and industry.

[0135] Step 4:

[0136] The server generates an interviewer based on the analysis results and generates appropriate questions. This process uses a generative AI model (e.g., GPT-4). The input is specific information about the company and industry from the analysis results, and the server generates interviewers and questions based on this information. The output is the configuration information for the generated interviewer and specific questions.

[0137] Step 5:

[0138] The server sends the generated interviewer information and questions to the terminal. The input is the generated interviewer and question information, which is sent to the terminal. The output is the interviewer and question information received by the terminal.

[0139] Step 6:

[0140] The terminal displays the interviewer information and questions received from the server to the user. The user answers the questions by voice using the terminal's microphone. The inputs are the questions displayed on the terminal and the user's voice responses, and the output is the voice data captured by the microphone.

[0141] Step 7:

[0142] The device converts the user's voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the user's voice data, and the output is the converted text data.

[0143] Step 8:

[0144] The terminal sends voice data and text data to the server. The input is the converted text data and the original voice data, which are then sent to the server. The output is the voice data and text data received by the server.

[0145] Step 9:

[0146] The server analyzes the received voice and text data. This analysis uses natural language processing and voice analysis technologies. The received voice and text data are input and analyzed. The output is an evaluation score as the analysis result.

[0147] Step 10:

[0148] The server scores the user's answers based on the analysis results. The input is the analysis results, and the score is calculated based on these. The output is the scoring result.

[0149] Step 11:

[0150] The server generates feedback based on the scoring results. The feedback includes what was good about the user's answer, what needs to be improved, and specific improvement measures. The input is the scoring results, and the feedback is generated based on these. The output is the generated feedback information.

[0151] Step 12:

[0152] The server sends feedback information to the terminal. The input is the generated feedback information, which is then sent to the terminal. The output is the feedback information received by the terminal.

[0153] Step 13:

[0154] The device displays feedback to the user, who then attempts to improve himself based on the presented feedback. The input is the feedback information received by the device, which is displayed to the user. The output is the user who received the feedback.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Interviews are a crucial element in modern job hunting and career change activities, but many job seekers struggle with a lack of interview preparation and practical experience. In particular, there are few systems that allow practical training in real environments such as brick-and-mortar stores. Furthermore, the lack of effective feedback makes it difficult for job seekers to identify areas for improvement.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes means for a user to input data such as an application form or resume, means for analyzing the input data and automatically generating interviewers and questions, means for displaying the generated questions on the display of the smart glasses, means for a user to input answers by voice and convert the voice data into text data, means for analyzing and evaluating the user's answers, means for generating individual feedback based on the evaluation results, and means for displaying the generated feedback on the display of the smart glasses. This enables job seekers to practice interviews in real time at a physical store and receive immediate, specific feedback.

[0160] An "entry sheet" is a document that a job seeker submits to a company of their choice, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[0161] A resume is a document that lists a job seeker's personal information, work history, educational background, etc., and is used by companies as a reference during interviews.

[0162] "Analysis" is the process of analyzing input data and extracting specific information or patterns.

[0163] An "interviewer" is a simulated character or virtual persona that conducts interviews with job seekers in a recruitment process.

[0164] "Questions" are inquiries or tasks that an interviewer makes to a job seeker.

[0165] "Smart glasses" are glasses-type devices that have built-in displays and microphones and can display information and record audio in real time.

[0166] "Displaying on a display" means visually presenting information such as text or images on the screen of a device.

[0167] "Voice data" refers to sound waves recorded as digital data, and includes the content of the user's speech.

[0168] "Text data" refers to information written in letters and symbols, and is data obtained by converting voice data into text.

[0169] "Evaluation" refers to analyzing the content of a user's response and determining its quality and appropriateness.

[0170] "Feedback" refers to suggestions for improvement or advice provided based on the evaluation results.

[0171] This invention aims to build an interview simulation system using smart glasses, primarily for use in brick-and-mortar stores. In this system, users input application forms, resumes, etc., and the server analyzes the data to automatically generate appropriate questions and interviewers. The generated questions are then displayed on the smart glasses' display, allowing the user to respond verbally. The voice data is then converted into text data, which is then analyzed and evaluated by the server. Finally, feedback based on the evaluation results is displayed on the smart glasses.

[0172] Hardware and software used

[0173] Smart glasses: Devices with built-in displays and microphones that display information and collect voice.

[0174] Server: A computer system that performs data analysis, question generation, speech and text data analysis, and feedback generation.

[0175] Natural Language Processing (NLP) technologies: Use the Google Cloud Natural Language API or our own trained generative AI models.

[0176] Speech analysis technology: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[0177] As a concrete example, the following scenario can be given:

[0178] 1. User application form entry:

[0179] The user fills out each item on the application form or resume at the reception terminal in the physical store and sends it to the server.

[0180] 2. Question and Interviewer Generation:

[0181] The server analyzes the received data and generates interviewers and questions with expertise that match the industry and position the user is applying for.

[0182] For example: "Tell me about your project management experience."

[0183] 3. User response recording:

[0184] Questions are displayed on the smart glasses' display, and the user answers verbally through a microphone, which converts the voice data into text data.

[0185] 4. Analysis and evaluation of answers:

[0186] The server analyzes the voice and text data and evaluates the answers using natural language processing technology, as well as voice intonation, pauses, and incorrect answers.

[0187] 5. Generating and Presenting Feedback:

[0188] Based on the evaluation results, the server generates specific feedback and presents it on the smart glasses' display.

[0189] Example: "Detail your leadership experiences and accomplishments and include specific examples of teamwork."

[0190] An example of a prompt for a generative AI model is:

[0191] User input data: Company type: IT company, Position: Project manager

[0192] Q: Tell me about your project management experience so far.

[0193] User Voice Answer: I have demonstrated leadership in past projects and completed them on time.

[0194] Expected feedback: Please provide a detailed description of your leadership experiences and achievements, and include specific examples of teamwork.

[0195] Users can practice interviews in real time and receive immediate and effective feedback, which is expected to improve their interview skills.

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] The user uses a reception terminal in a physical store to input data for an application form or resume. The input data includes name, address, educational background, work history, skills, and reasons for applying. When the user presses the send button, the terminal sends this data to the server. The input data is sent to the server as application form or resume information.

[0199] Step 2:

[0200] The server analyzes the data from the application forms and resumes it receives. During the analysis process, the input data is analyzed using natural language processing technology to identify the industry and position of the company the job seeker is applying for. For example, if a job seeker is applying for a project manager position at an IT company, that information is extracted. Based on the results of this analysis, appropriate questions and interviewers are generated.

[0201] Step 3:

[0202] The server generates a question and displays it on the display of the smart glasses. The user puts on the smart glasses and reads the question. For example, the question "Tell me about your project management experience" is displayed on the display.

[0203] Step 4:

[0204] The user answers by voice through the microphone in the smart glasses. The voice response is collected and sent to the server through the smart glasses. The voice data is sent to the server as input data, and then the voice data is converted to text data using the Google Cloud Speech-to-Text API.

[0205] Step 5:

[0206] The server analyzes the voice and text data and evaluates the user's answers using natural language processing technology. This evaluation process determines the quality and appropriateness of the answers, and analyzes voice intonation, pauses, and incorrect answers. The analyzed evaluation results are generated as output data.

[0207] Step 6:

[0208] The server generates individual feedback based on the evaluation results. For example, it might say, "It would be good to explain your leadership experiences and achievements in detail and add specific examples of teamwork." This feedback data is generated and saved as output data.

[0209] Step 7:

[0210] The feedback is presented on the display of the smart glasses, and the user can review the presented feedback and use it to improve themselves. The feedback is displayed and provided to the user.

[0211] This allows job seekers to practice interviews in real time in a physical store and receive specific and immediate feedback, which is expected to improve their interview skills.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] The present invention relates to a system that utilizes AI for job-seekers and career changers to automatically generate interview questions, evaluate the answers, and provide personalized feedback. By incorporating an emotion engine into this system, it is possible to recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[0214] Entering application forms and resumes

[0215] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[0216] Interviewer and Question Generation

[0217] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[0218] User responses and data conversion

[0219] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[0220] Analysis and evaluation of responses

[0221] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's responses, while voice analysis is used to detect vocal intonation, pauses, and slip-ups. An emotion engine can also be used to analyze the user's emotional state. This allows the user's emotional state to be reflected in the evaluation along with the content of the responses.

[0222] Generating and Presenting Feedback

[0223] The server generates individualized feedback based on the scoring results and the analysis results of the emotion engine. The feedback includes points that indicate what is good and what needs to be improved about the user's answer, as well as suggestions for specific improvements and other ways of expressing themselves. The generated feedback is sent to the device and presented to the user. The user can use this feedback to improve themselves.

[0224] Specific examples

[0225] As a specific example, consider a case where the company the user is applying to is an IT company and has experience in project management. The user details their project management experience on an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized. Furthermore, it can use an emotion engine to analyze the user's level of confidence and nervousness, and provide emotion-based feedback such as, "I like the fact that you speak with confidence."

[0226] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[0227] The processing flow will be explained below.

[0228] Step 1:

[0229] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[0230] Step 2:

[0231] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[0232] Step 3:

[0233] The terminal transmits the user's input data to the server.

[0234] Step 4:

[0235] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[0236] Step 5:

[0237] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[0238] Step 6:

[0239] The server sends the generated interviewer settings and questions to the terminal.

[0240] Step 7:

[0241] The terminal displays the received interviewer settings and questions to the user.

[0242] Step 8:

[0243] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[0244] Step 9:

[0245] The terminal converts the user's voice response into text data.

[0246] Step 10:

[0247] The terminal transmits the converted text data and the original voice data to the server.

[0248] Step 11:

[0249] The server analyzes the received voice and text data. Specifically, it uses natural language processing technology to analyze the content of the responses, and voice analysis technology to detect voice intonation, pauses, and slip-ups.

[0250] Step 12:

[0251] The server uses an emotion engine in addition to analysis to analyze the user's emotional state (e.g., confidence, anxiety, joy, sadness). It understands emotions from voice data and reads emotions from text data.

[0252] Step 13:

[0253] The server evaluates and scores the user's answers based on the analysis results, taking into account their emotional state.

[0254] Step 14:

[0255] The server generates personalized feedback based on the scoring and sentiment analysis results. The feedback includes points of merit and improvement for the user's answer, specific suggestions for improvement, suggestions for alternative ways of expressing oneself, and even advice based on emotions (e.g., "I liked that you spoke with confidence" or "You seem a little nervous, so try to relax").

[0256] Step 15:

[0257] The server transmits the generated feedback to the terminal.

[0258] Step 16:

[0259] The terminal displays the received feedback to the user.

[0260] Step 17:

[0261] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[0262] With this specific processing flow, the system of the present invention not only helps users hone their skills by gaining experience in interviews and learn how to present themselves to the right companies, but also enables more advanced self-improvement by providing feedback based on their emotional state.

[0263] Example 2

[0264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] In traditional job hunting and career change processes, interview practice and feedback are extremely important, but are difficult to provide individually. Providing appropriate feedback that takes into account the user's emotional state is particularly important, but manual evaluation has its limitations. Furthermore, while the intonation and pauses in a user's voice responses are important, these analyses have not yet been automated. Therefore, there is a need for a system that can provide individualized and precise feedback.

[0266] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input data from an information recording medium, a means for analyzing the input data and automatically generating a question and answerer and a question, a means for the user to input an answer to the generated question, a means for converting the user's answer into voice and text data, a means for analyzing the converted data and making an evaluation, a means for analyzing the user's emotional state using an emotion analysis device and reflecting the analysis in the evaluation, a means for generating individual feedback based on the evaluation result, and a means for presenting the generated feedback to the user. This makes it possible to provide precise feedback that takes into account the user's emotional state and voice characteristics.

[0267] "Information recording medium" refers to a medium that records information about a user, such as an application form or resume, in digital format.

[0268] The "question and answerer" refers to a virtual interviewer that is automatically generated based on the data input by the user, and is a character that asks questions that correspond to a specific industry or company.

[0269] "Means for converting" refers to the technical means for converting the data provided by the user via voice into text data, including voice recognition software.

[0270] "Analysis means" refers to a series of technologies for analyzing input data and converted data, including natural language processing technology and voice analysis technology.

[0271] "Means for evaluation" refers to technical means for evaluating the content of a user's responses and emotional state based on the analyzed data.

[0272] An "emotion analysis device" refers to technology that analyzes a user's emotional state from their voice or text data and reflects the results in their evaluation.

[0273] "Personalized feedback" refers to feedback that includes user-specific improvements and advice that is generated based on the user's specific answers and emotional state.

[0274] The present invention relates to a system that utilizes AI to automatically generate interview questions for job seekers and those seeking a career change, evaluates the answers, and provides individualized feedback. By incorporating an emotion analysis device, the system can recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[0275] Entering application forms and resumes

[0276] First, the user uses a terminal to enter information for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. For example, the input form is provided using a web browser or a dedicated application. When the user enters the required information and presses the send button, the data is sent to the server.

[0277] Receiving and parsing input data

[0278] The server receives the application form and resume data sent from the device. The received data is analyzed using natural language processing technology. Specifically, it uses Python's NLTK library and SpaCy to analyze the text data and identify the user's desired industry and company. Further processing is carried out based on the results of this analysis.

[0279] Interviewer Generation

[0280] Based on the analysis results, the server generates a Q&A person suited to the user's desired company and industry. This interviewer is customized using a generative AI model. For example, a natural language generation model such as ChatGPT is used to create a virtual interviewer with settings suited to interviews at specific companies and industries.

[0281] Question Generation

[0282] The server then generates appropriate questions for the user. By inputting appropriate prompts into the generative AI model, questions that match the user's aspirations are generated. For example, for a user with experience in project management, the server generates the question, "Please tell us specifically about your project management experience."

[0283] Posing the Question

[0284] The server then sends the generated question to the device, which displays it to the user, who answers verbally using a microphone while looking at the screen. The device then converts the speech to text using speech recognition software such as Google Cloud Speech-to-Text.

[0285] Sending response data

[0286] The device then sends the converted text and voice data to a server, which then receives the data and analyzes it using natural language processing and voice analysis technologies.

[0287] Analysis of responses and assessment of emotional state

[0288] The server uses natural language processing technology (e.g., the BERT model) to analyze the content of the user's responses. It also uses speech analysis technology to evaluate voice intonation, pauses, and slip-ups. It also uses EmotionAPI and Affectiva to analyze the user's emotional state. The analysis results include the content of the responses, voice quality, and emotional state.

[0289] Generate feedback

[0290] The server generates personalized feedback based on the analysis results. The feedback includes points for evaluation, areas for improvement, and specific measures for improvement. For example, feedback such as, "Your specific details about your project management experience are good. However, it would be even better if you also included the tools you used and the lessons you learned."

[0291] Providing feedback

[0292] The device receives the feedback sent from the server and displays it to the user, who can then view the feedback on the device screen and get clues to improve their answers.

[0293] Examples of specific examples and prompts

[0294] As a concrete example, consider the case where User A is applying to an IT company and has experience in project management. User A details his / her "project management experience" on an application form and sends it to the server. The server analyzes this data and generates an appropriate question and answerer along with the question "Tell us about your project management experience." When User A answers, the server analyzes the content and generates feedback such as "The details are specific, but it would be even better if you also added the tools you used and the lessons you learned," and sends it to the terminal.

[0295] An example of a prompt sentence to input to the generative AI model is as follows:

[0296] Plan: Generate interview questions and feedback for a user with project management experience who is applying to an IT company. Generate questions based on the information in the application form, analyze the user's answers using natural language processing and an emotion engine, and provide feedback such as the following:

[0297] Application form: As for project management experience, please state that you have led multiple projects in the past and achieved results.

[0298] Q: Tell us about your project management experience.

[0299] User Answer: "I led multiple projects and achieved results."

[0300] Feedback: "You're good at describing your project management experience in detail, but you could expand on specific accomplishments, tools you used, and lessons learned. You also speak with confidence, but you seem a little nervous. I think you could practice speaking in a more relaxed manner."

[0301] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[0302] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0303] Step 1:

[0304] The user inputs data for the information recording medium.

[0305] Specifically, the user uses a terminal to fill out a form containing information such as name, address, educational background, work history, skills, and motivation for applying. After entering the information, the user presses the send button, which sends the entered data from the terminal to the server.

[0306] Input: Application form and resume data entered by the user into the terminal

[0307] Output: User entry data sent to the server

[0308] Step 2:

[0309] The server receives and parses the input data.

[0310] Specifically, the server receives application forms and resumes sent from the device. The received data is analyzed using natural language processing technologies such as Python's NLTK library and SpaCy. This allows the user's desired company and industry to be identified.

[0311] Input: Application form and resume data sent from the device

[0312] Output: Information on the company and industry the user is interested in as a result of the analysis

[0313] Step 3:

[0314] The server generates the interviewers.

[0315] Specifically, the server uses a generative AI model based on the analysis results to generate a question and answerer suited to the company and industry the user is applying for. This question and answerer is designed as a virtual interviewer with extensive knowledge of the specific industry or company.

[0316] Input: Information on the company and industry you are applying to as a result of the analysis

[0317] Output: Virtual interviewer as question and answerer

[0318] Step 4:

[0319] The server generates the question.

[0320] Specifically, the server inputs a prompt into the generative AI model and generates a question that matches the user's desired profile. For example, a question like, "Please tell us specifically about your project management experience" might be generated.

[0321] Input: Prompt statement and user parsed data

[0322] Output: Generated questions

[0323] Step 5:

[0324] The terminal presents the question to the user.

[0325] Specifically, the device displays the questions sent from the server to the user, who then answers verbally using a microphone while looking at the device screen.

[0326] Input: The question sent by the server

[0327] Output: The terminal screen showing the question

[0328] Step 6:

[0329] The user responds verbally, and the device converts the speech into text.

[0330] The user answers questions by speaking into a microphone, and the device uses voice recognition software such as Google Cloud Speech-to-Text to convert the answers into text.

[0331] Input: User's spoken response

[0332] Output: User answers converted to text data

[0333] Step 7:

[0334] The terminal transmits the response data to the server.

[0335] Specifically, the terminal converts the user's answers into text data and transmits them to the server, which then receives the data.

[0336] Input: User answers converted to text data

[0337] Output: Response data sent to the server

[0338] Step 8:

[0339] The server analyzes the responses and assesses the emotional state.

[0340] Specifically, the server analyzes the user's responses using natural language processing technology (such as the BERT model), evaluates the intonation, pauses, and slip-ups of the user's voice using speech analysis technology, and analyzes the user's emotional state using EmotionAPI and Affectiva.

[0341] Input: Response data sent to the server

[0342] Output: Answer content, voice characteristics, emotional state as analysis results

[0343] Step 9:

[0344] The server generates personalized feedback.

[0345] Specifically, the server generates individual feedback based on the analysis results, including evaluation points for the answer, areas for improvement, and specific improvement measures.

[0346] Input: Answer content as analysis result, voice characteristics, emotional state

[0347] Output: Generated feedback

[0348] Step 10:

[0349] The terminal presents feedback to the user.

[0350] Specifically, the device receives the feedback sent from the server and displays it to the user, who can then improve themselves based on this feedback.

[0351] Input: Feedback sent by the server

[0352] Output: Feedback displayed on the terminal

[0353] Through these steps, the system helps users improve themselves through interview practice, allowing them to receive more detailed feedback, including their emotional state.

[0354] (Application example 2)

[0355] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] Improving the technical skills and troubleshooting abilities of robot operators is an essential challenge in modern factories. However, conventional education and training systems struggle to assess actual operating skills and emotional states in real time and provide individualized feedback. This makes it difficult for operators to effectively acquire the necessary skills. Furthermore, conventional systems do not provide feedback that takes into account the user's emotional state, making it difficult to alleviate stress and tension.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input raw data or document data, a means for analyzing the input data and automatically generating questions for the interlocutor, a means for the user to input answers to the generated questions, a means for providing interactive training including questions about the operator's operating skills and how to deal with the situation, and a means for analyzing the operator's emotional state using an emotion engine and reflecting the results of the evaluation. This makes it possible to effectively improve the operator's technical skills and provide individual feedback based on their emotional state, such as stress or tension.

[0358] "Raw data" refers to data acquired in real time or as is.

[0359] "Document data" refers to information recorded in the form of a resume, application form, etc.

[0360] The "interlocutor" refers to a virtual interviewer or interviewer that is generated to have a dialogue with the user.

[0361] A "question" refers to a question or interrogation posed by the interlocutor to the user.

[0362] "Operator" refers to the engineers and workers who operate robots and equipment within the factory.

[0363] "Operational skills" refers to the technical ability of an operator to accurately operate a robot or equipment.

[0364] "Solutions" refers to solutions and approaches to the troubles and problems that robots and equipment face.

[0365] "Interactive training" refers to educational and training programs that are conducted interactively with the user.

[0366] An "emotion engine" refers to technology that analyzes a user's emotional state and generates feedback based on the results.

[0367] "Evaluation results" refer to the analysis results of the user's answers and operations, and include scores for skills and emotional state.

[0368] "Feedback" refers to information that evaluates a user's actions or responses and points out areas for improvement.

[0369] "Server" refers to a computer system that performs functions such as data analysis, participant generation, and feedback provision.

[0370] To implement this invention, the following hardware and software are used. The hardware used is a microphone, a smartphone, or a computer. The software used is the SpeechRecognition library (Python), the Transformers library (Python), and an AI model for emotion analysis. Specific system program processing is described below.

[0371] System configuration and processing content

[0372] The server provides an interface for users to input raw data and document data. Users fill out application forms and resumes using their smartphones or computers. The input data is sent to the server, which analyzes it. Based on the analysis results, the server automatically generates an interlocutor (a virtual interviewer) and questions. These questions are related to specific operational skills and coping methods.

[0373] The generated questions are displayed on the user's device. The user answers the questions by voice using a microphone. The device converts the user's voice data into text data and sends the data to the server.

[0374] The server uses natural language processing technology to analyze the received voice and text data. It also uses an emotion engine to analyze the user's emotional state and reflect it in the evaluation. Based on the evaluation results, the server generates personalized feedback. This feedback may include specific improvements to the user's answers, technical advice, or even feedback based on the user's emotional state. The generated feedback is sent to the user's device and presented to the user.

[0375] Specific examples

[0376] For example, in an operator training system, the question "What would you do if your robot suddenly stopped working?" is generated and presented to the user. The user answers, "First, I would check the error code and try to restart it manually. If that doesn't solve the problem, I would contact technical support." If this answer is judged to be a confident answer after natural language processing and sentiment analysis, the system provides feedback such as, "Your answer is appropriate. It would be a good idea to thoroughly check the error code again before contacting technical support."

[0377] Prompt Sentence Examples

[0378] Project name: Factory robot operator training system

[0379] Question: "What would you do if your robot suddenly stopped working?"

[0380] Answer: "First, check the error code and try a manual reboot. If that doesn't work, contact technical support."

[0381] Sentiment analysis: "This shows confidence. It's the right approach for an engineer."

[0382] Feedback: "The answer is good. I would recommend double-checking the error code before contacting technical support."

[0383] In this way, the system of the present invention can effectively improve the operator's technical skills and provide personalized feedback based on emotional states such as stress and tension.

[0384] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0385] Step 1:

[0386] The server receives raw data or document data from user devices as input. Users use their smartphones or computers to enter information for job applications or resumes and send it to the server. The output is a database entry containing the input information.

[0387] Step 2:

[0388] The server analyzes the input data and automatically generates interlocutors and questions. This analysis process uses natural language processing technology to identify the desired company, industry, operational skills, and countermeasures. The input is data from an application form or resume, and the output is the generated questions and a virtual interlocutor.

[0389] Step 3:

[0390] The terminal presents the generated question to the user, who then answers by voice using a microphone. The input is the presented question, and the output is voice data.

[0391] Step 4:

[0392] The device uses the SpeechRecognition library to convert the user's voice data into text data. The input is the user's voice data, and the output is the data converted from that voice into text.

[0393] Step 5:

[0394] The server receives the converted text and voice data and evaluates them using natural language processing technology and a sentiment analysis engine. The natural language processing technology evaluates the accuracy and quality of the response, while the sentiment analysis engine analyzes the voice intonation and emotional state. The input is text and voice data, and the output is the evaluation result.

[0395] Step 6:

[0396] The server generates feedback based on the evaluation results. The feedback includes suggestions for improving the answers, technical advice, and comments based on the user's emotional state. The input is the evaluation results, and the output is individual feedback.

[0397] Step 7:

[0398] The terminal presents the generated feedback to the user, who can then further improve himself based on this feedback. The input is the feedback, and the output is the feedback display for the user.

[0399] This series of processing steps allows the system of the present invention to provide effective training based on the user's technical ability and emotional state.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0401] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0403] [Second embodiment]

[0404] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0405] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0406] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0408] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0410] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0411] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0412] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0415] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0416] This invention is a system that utilizes AI to automatically generate interviewers and questions for job seekers and those looking to change jobs, evaluate the answers, and provide individual feedback. This system includes the following means.

[0417] Entering application forms and resumes

[0418] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[0419] Interviewer and Question Generation

[0420] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[0421] User responses and data conversion

[0422] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[0423] Analysis and evaluation of responses

[0424] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the evaluation results, the server assigns a score to the user's answers.

[0425] Generating and Presenting Feedback

[0426] The server generates individualized feedback based on the scoring results. The feedback includes points about the user's answers that are good and that need improvement, as well as suggestions for specific improvements and other ways of expressing the answers. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[0427] Specific examples

[0428] As a concrete example, consider a case where the company the user is applying to is an IT company and the user has experience in project management. The user details their project management experience on an application form and sends it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized.

[0429] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[0430] The processing flow will be explained below.

[0431] Step 1:

[0432] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[0433] Step 2:

[0434] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[0435] Step 3:

[0436] The terminal transmits the user's input data to the server.

[0437] Step 4:

[0438] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[0439] Step 5:

[0440] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[0441] Step 6:

[0442] The server sends the generated interviewer settings and questions to the terminal.

[0443] Step 7:

[0444] The terminal displays the received interviewer settings and questions to the user.

[0445] Step 8:

[0446] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[0447] Step 9:

[0448] The terminal converts the user's voice response into text data.

[0449] Step 10:

[0450] The terminal transmits the converted text data and the original voice data to the server.

[0451] Step 11:

[0452] The server analyzes the received voice and text data.

[0453] Specifically, natural language processing technology is used to analyze the content of the responses, and voice analysis technology is used to detect voice intonation, pauses, and slip-ups.

[0454] Step 12:

[0455] The server evaluates and scores the user's answers based on the analysis results.

[0456] Step 13:

[0457] The server generates personalized feedback based on the scoring results, including points of merit, areas for improvement, and suggestions for specific improvements or other ways of expressing them.

[0458] Step 14:

[0459] The server transmits the generated feedback to the terminal.

[0460] Step 15:

[0461] The terminal displays the received feedback to the user.

[0462] Step 16:

[0463] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[0464] Example 1

[0465] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0466] In traditional job hunting and career change processes, when job seekers individually prepare for interviews, they lack concrete methods for determining appropriate answers to interviewer questions and evaluating the quality of those answers. Furthermore, it is difficult to obtain the feedback needed to improve interview techniques. This makes it difficult for job seekers to effectively showcase their skills.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0468] In this invention, the server includes means for a user to input data for an application form or resume, means for analyzing the input data and automatically generating interviewer and questions, means for the user to input answers to the generated questions, means for converting the user's voice data into text data, means for analyzing the converted data and performing an evaluation, means for generating individual feedback based on the evaluation results, and means for presenting the generated feedback to the user. This allows the user to improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[0469] "User" refers to an individual who uses this system to input application form and resume data and simulate an interview.

[0470] An "entry sheet" is a document used to submit personal information to a company when seeking employment, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[0471] A resume is a document that details a job seeker's career history and skills, and is submitted along with an application form.

[0472] The "server" is a central computer system that receives and analyzes application and resume data, generates interviewers and questions, evaluates user response data, and provides feedback.

[0473] A "terminal" is a computer or smart device that allows a user to enter data for an application form or resume, answer generated questions, and receive feedback.

[0474] "Analysis" is the process in which the server processes data from application forms and resumes to identify the company and industry the user is applying to, and then generates interviewers and questions accordingly.

[0475] The "interviewer" is a virtual interviewer that the server generates based on the company and industry that the user is applying to, and who asks appropriate questions.

[0476] The "questions" are generated by the server based on the data in the user's application form or resume, and are part of a virtual interview, similar to the interviewer.

[0477] "Voice data" refers to data generated by voice when a user answers a question using a microphone.

[0478] "Text data" is voice data converted into character information, and is data used by the server for analysis and evaluation.

[0479] "Natural language processing technology" refers to technology used by computers to understand, analyze, and generate human language, including generative AI models such as GPT-4.

[0480] "Scoring" is a process in which the server analyzes the content of the user's answer and then numerically evaluates the quality and appropriateness of the answer.

[0481] "Feedback" is generated by the server based on the scoring results, and includes good points and areas for improvement in the user's answer, specific improvement measures, and other ways of expressing them.

[0482] The present invention is a system for supporting job hunting and career change, and is designed to allow users to input data from application forms and resumes, analyze the data, automatically generate interviewers and questions, evaluate the answers, and provide individual feedback. This system includes the following means.

[0483] System configuration

[0484] This system consists of a terminal used by the user, a server that receives and analyzes data, and software required for data conversion and analysis.

[0485] Entering application forms and resumes

[0486] The user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. When the user enters the required information in each field and presses the send button, the data is sent to the server.

[0487] Data analysis and interviewer and question generation

[0488] The server receives the application form and resume data sent by the user and stores it in a database. The stored data is then analyzed, using natural language processing technology to identify the company and industry the user is applying to.

[0489] The server generates an interviewer based on the identified company and industry. This interviewer is a virtual persona that generates questions appropriate for the specific industry or company. To generate questions, a generative AI model (e.g., GPT-4) is used.

[0490] Posting questions and accepting answers

[0491] The generated question is displayed to the user through the device. The user answers the question by voice using the device's microphone. Once the answer is complete, the voice data is sent to the server by pressing the send button on the device.

[0492] Audio data conversion

[0493] The device converts the user's voice data into text data, for example, using voice recognition software (e.g., Google Cloud Speech-to-Text API), and the text data is sent to the server along with the voice data.

[0494] Analysis and evaluation of responses

[0495] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the analysis results, the server assigns a score to the user's answers.

[0496] Generating and Presenting Feedback

[0497] The server generates individual feedback based on the scoring results. The feedback includes points of merit and improvement for the user's answer, as well as suggestions for specific improvements and other ways of expressing the answer. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[0498] Specific examples

[0499] As a specific example, suppose a user is applying to an information technology company and has experience in project management. The user details their project management experience in an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides specific areas for improvement and points that should be emphasized as feedback.

[0500] Prompt Sentence Examples

[0501] Examples of prompts are:

[0502] "Based on the project management experience stated in the application form, what questions should a user generate when attending an interview?"

[0503] This allows the system to support users in more effectively preparing for interviews, ultimately helping them learn how to present themselves to companies in the right way.

[0504] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0505] Step 1:

[0506] The user uses a terminal to input data for an application form and resume. A form containing fields such as name, address, educational background, work history, skills, and reasons for applying is displayed on the terminal. When the user enters the required information in each field and presses the send button, the data is sent to the server. The input data includes the user's personal information and work history, and the output is the application data sent to the server.

[0507] Step 2:

[0508] The server receives the application form and resume data sent from the terminal. It stores the received data in a database. The input is the application data sent from the terminal, which is then stored in the database, and the output is the user information stored in the database.

[0509] Step 3:

[0510] The server retrieves the user's application form and resume data from the database. The server analyzes the retrieved data and identifies the company and industry the user is applying to. The input is the application data retrieved from the database, which is analyzed using natural language processing technology. The output is the analysis result, which identifies the company and industry.

[0511] Step 4:

[0512] The server generates an interviewer based on the analysis results and generates appropriate questions. This process uses a generative AI model (e.g., GPT-4). The input is specific information about the company and industry from the analysis results, and the server generates interviewers and questions based on this information. The output is the configuration information for the generated interviewer and specific questions.

[0513] Step 5:

[0514] The server sends the generated interviewer information and questions to the terminal. The input is the generated interviewer and question information, which is sent to the terminal. The output is the interviewer and question information received by the terminal.

[0515] Step 6:

[0516] The terminal displays the interviewer information and questions received from the server to the user. The user answers the questions by voice using the terminal's microphone. The inputs are the questions displayed on the terminal and the user's voice responses, and the output is the voice data captured by the microphone.

[0517] Step 7:

[0518] The device converts the user's voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the user's voice data, and the output is the converted text data.

[0519] Step 8:

[0520] The terminal sends voice data and text data to the server. The input is the converted text data and the original voice data, which are then sent to the server. The output is the voice data and text data received by the server.

[0521] Step 9:

[0522] The server analyzes the received voice and text data. This analysis uses natural language processing and voice analysis technologies. The received voice and text data are input and analyzed. The output is an evaluation score as the analysis result.

[0523] Step 10:

[0524] The server scores the user's answers based on the analysis results. The input is the analysis results, and the score is calculated based on these. The output is the scoring result.

[0525] Step 11:

[0526] The server generates feedback based on the scoring results. The feedback includes what was good about the user's answer, what needs to be improved, and specific improvement measures. The input is the scoring results, and the feedback is generated based on these. The output is the generated feedback information.

[0527] Step 12:

[0528] The server sends feedback information to the terminal. The input is the generated feedback information, which is then sent to the terminal. The output is the feedback information received by the terminal.

[0529] Step 13:

[0530] The device displays feedback to the user, who then attempts to improve himself based on the presented feedback. The input is the feedback information received by the device, which is displayed to the user. The output is the user who received the feedback.

[0531] (Application example 1)

[0532] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0533] Interviews are a crucial element in modern job hunting and career change activities, but many job seekers struggle with a lack of interview preparation and practical experience. In particular, there are few systems that allow practical training in real environments such as brick-and-mortar stores. Furthermore, the lack of effective feedback makes it difficult for job seekers to identify areas for improvement.

[0534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0535] In this invention, the server includes means for a user to input data such as an application form or resume, means for analyzing the input data and automatically generating interviewers and questions, means for displaying the generated questions on the display of the smart glasses, means for a user to input answers by voice and convert the voice data into text data, means for analyzing and evaluating the user's answers, means for generating individual feedback based on the evaluation results, and means for displaying the generated feedback on the display of the smart glasses. This enables job seekers to practice interviews in real time at a physical store and receive immediate, specific feedback.

[0536] An "entry sheet" is a document that a job seeker submits to a company of their choice, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[0537] A resume is a document that lists a job seeker's personal information, work history, educational background, etc., and is used by companies as a reference during interviews.

[0538] "Analysis" is the process of analyzing input data and extracting specific information or patterns.

[0539] An "interviewer" is a simulated character or virtual persona that conducts interviews with job seekers in a recruitment process.

[0540] "Questions" are inquiries or tasks that an interviewer makes to a job seeker.

[0541] "Smart glasses" are glasses-type devices that have built-in displays and microphones and can display information and record audio in real time.

[0542] "Displaying on a display" means visually presenting information such as text or images on the screen of a device.

[0543] "Voice data" refers to sound waves recorded as digital data, and includes the content of the user's speech.

[0544] "Text data" refers to information written in letters and symbols, and is data obtained by converting voice data into text.

[0545] "Evaluation" refers to analyzing the content of a user's response and determining its quality and appropriateness.

[0546] "Feedback" refers to suggestions for improvement or advice provided based on the evaluation results.

[0547] This invention aims to build an interview simulation system using smart glasses, primarily for use in brick-and-mortar stores. In this system, users input application forms, resumes, etc., and the server analyzes the data to automatically generate appropriate questions and interviewers. The generated questions are then displayed on the smart glasses' display, allowing the user to respond verbally. The voice data is then converted into text data, which is then analyzed and evaluated by the server. Finally, feedback based on the evaluation results is displayed on the smart glasses.

[0548] Hardware and software used

[0549] Smart glasses: Devices with built-in displays and microphones that display information and collect voice.

[0550] Server: A computer system that performs data analysis, question generation, speech and text data analysis, and feedback generation.

[0551] Natural Language Processing (NLP) technologies: Use the Google Cloud Natural Language API or our own trained generative AI models.

[0552] Speech analysis technology: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[0553] As a concrete example, the following scenario can be given:

[0554] 1. User application form entry:

[0555] The user fills out each item on the application form or resume at the reception terminal in the physical store and sends it to the server.

[0556] 2. Question and Interviewer Generation:

[0557] The server analyzes the received data and generates interviewers and questions with expertise that match the industry and position the user is applying for.

[0558] For example: "Tell me about your project management experience."

[0559] 3. User response recording:

[0560] Questions are displayed on the smart glasses' display, and the user answers verbally through a microphone, which converts the voice data into text data.

[0561] 4. Analysis and evaluation of answers:

[0562] The server analyzes the voice and text data and evaluates the answers using natural language processing technology, as well as voice intonation, pauses, and incorrect answers.

[0563] 5. Generating and Presenting Feedback:

[0564] Based on the evaluation results, the server generates specific feedback and presents it on the smart glasses' display.

[0565] Example: "Detail your leadership experiences and accomplishments and include specific examples of teamwork."

[0566] An example of a prompt for a generative AI model is:

[0567] User input data: Company type: IT company, Position: Project manager

[0568] Q: Tell me about your project management experience so far.

[0569] User Voice Answer: I have demonstrated leadership in past projects and completed them on time.

[0570] Expected feedback: Please provide a detailed description of your leadership experiences and achievements, and include specific examples of teamwork.

[0571] Users can practice interviews in real time and receive immediate and effective feedback, which is expected to improve their interview skills.

[0572] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0573] Step 1:

[0574] The user uses a reception terminal in a physical store to input data for an application form or resume. The input data includes name, address, educational background, work history, skills, and reasons for applying. When the user presses the send button, the terminal sends this data to the server. The input data is sent to the server as application form or resume information.

[0575] Step 2:

[0576] The server analyzes the data from the application forms and resumes it receives. During the analysis process, the input data is analyzed using natural language processing technology to identify the industry and position of the company the job seeker is applying for. For example, if a job seeker is applying for a project manager position at an IT company, that information is extracted. Based on the results of this analysis, appropriate questions and interviewers are generated.

[0577] Step 3:

[0578] The server generates a question and displays it on the display of the smart glasses. The user puts on the smart glasses and reads the question. For example, the question "Tell me about your project management experience" is displayed on the display.

[0579] Step 4:

[0580] The user answers by voice through the microphone in the smart glasses. The voice response is collected and sent to the server through the smart glasses. The voice data is sent to the server as input data, and then the voice data is converted to text data using the Google Cloud Speech-to-Text API.

[0581] Step 5:

[0582] The server analyzes the voice and text data and evaluates the user's answers using natural language processing technology. This evaluation process determines the quality and appropriateness of the answers, and analyzes voice intonation, pauses, and incorrect answers. The analyzed evaluation results are generated as output data.

[0583] Step 6:

[0584] The server generates individual feedback based on the evaluation results. For example, it might say, "It would be good to explain your leadership experiences and achievements in detail and add specific examples of teamwork." This feedback data is generated and saved as output data.

[0585] Step 7:

[0586] The feedback is presented on the display of the smart glasses, and the user can review the presented feedback and use it to improve themselves. The feedback is displayed and provided to the user.

[0587] This allows job seekers to practice interviews in real time in a physical store and receive specific and immediate feedback, which is expected to improve their interview skills.

[0588] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0589] The present invention relates to a system that utilizes AI for job-seekers and career changers to automatically generate interview questions, evaluate the answers, and provide personalized feedback. By incorporating an emotion engine into this system, it is possible to recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[0590] Entering application forms and resumes

[0591] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[0592] Interviewer and Question Generation

[0593] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[0594] User responses and data conversion

[0595] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[0596] Analysis and evaluation of responses

[0597] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's responses, while voice analysis is used to detect vocal intonation, pauses, and slip-ups. An emotion engine can also be used to analyze the user's emotional state. This allows the user's emotional state to be reflected in the evaluation along with the content of the responses.

[0598] Generating and Presenting Feedback

[0599] The server generates individualized feedback based on the scoring results and the analysis results of the emotion engine. The feedback includes points that indicate what is good and what needs to be improved about the user's answer, as well as suggestions for specific improvements and other ways of expressing themselves. The generated feedback is sent to the device and presented to the user. The user can use this feedback to improve themselves.

[0600] Specific examples

[0601] As a specific example, consider a case where the company the user is applying to is an IT company and has experience in project management. The user details their project management experience on an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized. Furthermore, it can use an emotion engine to analyze the user's level of confidence and nervousness, and provide emotion-based feedback such as, "I like the fact that you speak with confidence."

[0602] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[0603] The processing flow will be explained below.

[0604] Step 1:

[0605] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[0606] Step 2:

[0607] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[0608] Step 3:

[0609] The terminal transmits the user's input data to the server.

[0610] Step 4:

[0611] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[0612] Step 5:

[0613] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[0614] Step 6:

[0615] The server sends the generated interviewer settings and questions to the terminal.

[0616] Step 7:

[0617] The terminal displays the received interviewer settings and questions to the user.

[0618] Step 8:

[0619] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[0620] Step 9:

[0621] The terminal converts the user's voice response into text data.

[0622] Step 10:

[0623] The terminal transmits the converted text data and the original voice data to the server.

[0624] Step 11:

[0625] The server analyzes the received voice and text data. Specifically, it uses natural language processing technology to analyze the content of the responses, and voice analysis technology to detect voice intonation, pauses, and slip-ups.

[0626] Step 12:

[0627] The server uses an emotion engine in addition to analysis to analyze the user's emotional state (e.g., confidence, anxiety, joy, sadness). It understands emotions from voice data and reads emotions from text data.

[0628] Step 13:

[0629] The server evaluates and scores the user's answers based on the analysis results, taking into account their emotional state.

[0630] Step 14:

[0631] The server generates personalized feedback based on the scoring and sentiment analysis results. The feedback includes points of merit and improvement for the user's answer, specific suggestions for improvement, suggestions for alternative ways of expressing oneself, and even advice based on emotions (e.g., "I liked that you spoke with confidence" or "You seem a little nervous, so try to relax").

[0632] Step 15:

[0633] The server transmits the generated feedback to the terminal.

[0634] Step 16:

[0635] The terminal displays the received feedback to the user.

[0636] Step 17:

[0637] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[0638] With this specific processing flow, the system of the present invention not only helps users hone their skills by gaining experience in interviews and learn how to present themselves to the right companies, but also enables more advanced self-improvement by providing feedback based on their emotional state.

[0639] Example 2

[0640] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0641] In traditional job hunting and career change processes, interview practice and feedback are extremely important, but are difficult to provide individually. Providing appropriate feedback that takes into account the user's emotional state is particularly important, but manual evaluation has its limitations. Furthermore, while the intonation and pauses in a user's voice responses are important, these analyses have not yet been automated. Therefore, there is a need for a system that can provide individualized and precise feedback.

[0642] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input data from an information recording medium, a means for analyzing the input data and automatically generating a question and answerer and a question, a means for the user to input an answer to the generated question, a means for converting the user's answer into voice and text data, a means for analyzing the converted data and making an evaluation, a means for analyzing the user's emotional state using an emotion analysis device and reflecting the analysis in the evaluation, a means for generating individual feedback based on the evaluation result, and a means for presenting the generated feedback to the user. This makes it possible to provide precise feedback that takes into account the user's emotional state and voice characteristics.

[0643] "Information recording medium" refers to a medium that records information about a user, such as an application form or resume, in digital format.

[0644] The "question and answerer" refers to a virtual interviewer that is automatically generated based on the data input by the user, and is a character that asks questions that correspond to a specific industry or company.

[0645] "Means for converting" refers to the technical means for converting the data provided by the user via voice into text data, including voice recognition software.

[0646] "Analysis means" refers to a series of technologies for analyzing input data and converted data, including natural language processing technology and voice analysis technology.

[0647] "Means for evaluation" refers to technical means for evaluating the content of a user's responses and emotional state based on the analyzed data.

[0648] An "emotion analysis device" refers to technology that analyzes a user's emotional state from their voice or text data and reflects the results in their evaluation.

[0649] "Personalized feedback" refers to feedback that includes user-specific improvements and advice that is generated based on the user's specific answers and emotional state.

[0650] The present invention relates to a system that utilizes AI to automatically generate interview questions for job seekers and those seeking a career change, evaluates the answers, and provides individualized feedback. By incorporating an emotion analysis device, the system can recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[0651] Entering application forms and resumes

[0652] First, the user uses a terminal to enter information for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. For example, the input form is provided using a web browser or a dedicated application. When the user enters the required information and presses the send button, the data is sent to the server.

[0653] Receiving and parsing input data

[0654] The server receives the application form and resume data sent from the device. The received data is analyzed using natural language processing technology. Specifically, it uses Python's NLTK library and SpaCy to analyze the text data and identify the user's desired industry and company. Further processing is carried out based on the results of this analysis.

[0655] Interviewer Generation

[0656] Based on the analysis results, the server generates a Q&A person suited to the user's desired company and industry. This interviewer is customized using a generative AI model. For example, a natural language generation model such as ChatGPT is used to create a virtual interviewer with settings suited to interviews at specific companies and industries.

[0657] Question Generation

[0658] The server then generates appropriate questions for the user. By inputting appropriate prompts into the generative AI model, questions that match the user's aspirations are generated. For example, for a user with experience in project management, the server generates the question, "Please tell us specifically about your project management experience."

[0659] Posing the Question

[0660] The server then sends the generated question to the device, which displays it to the user, who answers verbally using a microphone while looking at the screen. The device then converts the speech to text using speech recognition software such as Google Cloud Speech-to-Text.

[0661] Sending response data

[0662] The device then sends the converted text and voice data to a server, which then receives the data and analyzes it using natural language processing and voice analysis technologies.

[0663] Analysis of responses and assessment of emotional state

[0664] The server uses natural language processing technology (e.g., the BERT model) to analyze the content of the user's responses. It also uses speech analysis technology to evaluate voice intonation, pauses, and slip-ups. It also uses EmotionAPI and Affectiva to analyze the user's emotional state. The analysis results include the content of the responses, voice quality, and emotional state.

[0665] Generate feedback

[0666] The server generates personalized feedback based on the analysis results. The feedback includes points for evaluation, areas for improvement, and specific measures for improvement. For example, feedback such as, "Your specific details about your project management experience are good. However, it would be even better if you also included the tools you used and the lessons you learned."

[0667] Providing feedback

[0668] The device receives the feedback sent from the server and displays it to the user, who can then view the feedback on the device screen and get clues to improve their answers.

[0669] Examples of specific examples and prompts

[0670] As a concrete example, consider the case where User A is applying to an IT company and has experience in project management. User A details his / her "project management experience" on an application form and sends it to the server. The server analyzes this data and generates an appropriate question and answerer along with the question "Tell us about your project management experience." When User A answers, the server analyzes the content and generates feedback such as "The details are specific, but it would be even better if you also added the tools you used and the lessons you learned," and sends it to the terminal.

[0671] An example of a prompt sentence to input to the generative AI model is as follows:

[0672] Plan: Generate interview questions and feedback for a user with project management experience who is applying to an IT company. Generate questions based on the information in the application form, analyze the user's answers using natural language processing and an emotion engine, and provide feedback such as the following:

[0673] Application form: As for project management experience, please state that you have led multiple projects in the past and achieved results.

[0674] Q: Tell us about your project management experience.

[0675] User Answer: "I led multiple projects and achieved results."

[0676] Feedback: "You're good at describing your project management experience in detail, but you could expand on specific accomplishments, tools you used, and lessons learned. You also speak with confidence, but you seem a little nervous. I think you could practice speaking in a more relaxed manner."

[0677] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[0678] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0679] Step 1:

[0680] The user inputs data for the information recording medium.

[0681] Specifically, the user uses a terminal to fill out a form containing information such as name, address, educational background, work history, skills, and motivation for applying. After entering the information, the user presses the send button, which sends the entered data from the terminal to the server.

[0682] Input: Application form and resume data entered by the user into the terminal

[0683] Output: User entry data sent to the server

[0684] Step 2:

[0685] The server receives and parses the input data.

[0686] Specifically, the server receives application forms and resumes sent from the device. The received data is analyzed using natural language processing technologies such as Python's NLTK library and SpaCy. This allows the user's desired company and industry to be identified.

[0687] Input: Application form and resume data sent from the device

[0688] Output: Information on the company and industry the user is interested in as a result of the analysis

[0689] Step 3:

[0690] The server generates the interviewers.

[0691] Specifically, the server uses a generative AI model based on the analysis results to generate a question and answerer suited to the company and industry the user is applying for. This question and answerer is designed as a virtual interviewer with extensive knowledge of the specific industry or company.

[0692] Input: Information on the company and industry you are applying to as a result of the analysis

[0693] Output: Virtual interviewer as question and answerer

[0694] Step 4:

[0695] The server generates the question.

[0696] Specifically, the server inputs a prompt into the generative AI model and generates a question that matches the user's desired profile. For example, a question like, "Please tell us specifically about your project management experience" might be generated.

[0697] Input: Prompt statement and user parsed data

[0698] Output: Generated questions

[0699] Step 5:

[0700] The terminal presents the question to the user.

[0701] Specifically, the device displays the questions sent from the server to the user, who then answers verbally using a microphone while looking at the device screen.

[0702] Input: The question sent by the server

[0703] Output: The terminal screen showing the question

[0704] Step 6:

[0705] The user responds verbally, and the device converts the speech into text.

[0706] The user answers questions by speaking into a microphone, and the device uses voice recognition software such as Google Cloud Speech-to-Text to convert the answers into text.

[0707] Input: User's spoken response

[0708] Output: User answers converted to text data

[0709] Step 7:

[0710] The terminal transmits the response data to the server.

[0711] Specifically, the terminal converts the user's answers into text data and transmits them to the server, which then receives the data.

[0712] Input: User answers converted to text data

[0713] Output: Response data sent to the server

[0714] Step 8:

[0715] The server analyzes the responses and assesses the emotional state.

[0716] Specifically, the server analyzes the user's responses using natural language processing technology (such as the BERT model), evaluates the intonation, pauses, and slip-ups of the user's voice using speech analysis technology, and analyzes the user's emotional state using EmotionAPI and Affectiva.

[0717] Input: Response data sent to the server

[0718] Output: Answer content, voice characteristics, emotional state as analysis results

[0719] Step 9:

[0720] The server generates personalized feedback.

[0721] Specifically, the server generates individual feedback based on the analysis results, including evaluation points for the answer, areas for improvement, and specific improvement measures.

[0722] Input: Answer content as analysis result, voice characteristics, emotional state

[0723] Output: Generated feedback

[0724] Step 10:

[0725] The terminal presents feedback to the user.

[0726] Specifically, the device receives the feedback sent from the server and displays it to the user, who can then improve themselves based on this feedback.

[0727] Input: Feedback sent by the server

[0728] Output: Feedback displayed on the terminal

[0729] Through these steps, the system helps users improve themselves through interview practice, allowing them to receive more detailed feedback, including their emotional state.

[0730] (Application example 2)

[0731] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] Improving the technical skills and troubleshooting abilities of robot operators is an essential challenge in modern factories. However, conventional education and training systems struggle to assess actual operating skills and emotional states in real time and provide individualized feedback. This makes it difficult for operators to effectively acquire the necessary skills. Furthermore, conventional systems do not provide feedback that takes into account the user's emotional state, making it difficult to alleviate stress and tension.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input raw data or document data, a means for analyzing the input data and automatically generating questions for the interlocutor, a means for the user to input answers to the generated questions, a means for providing interactive training including questions about the operator's operating skills and how to deal with the situation, and a means for analyzing the operator's emotional state using an emotion engine and reflecting the results of the evaluation. This makes it possible to effectively improve the operator's technical skills and provide individual feedback based on their emotional state, such as stress or tension.

[0734] "Raw data" refers to data acquired in real time or as is.

[0735] "Document data" refers to information recorded in the form of a resume, application form, etc.

[0736] The "interlocutor" refers to a virtual interviewer or interviewer that is generated to have a dialogue with the user.

[0737] A "question" refers to a question or interrogation posed by the interlocutor to the user.

[0738] "Operator" refers to the engineers and workers who operate robots and equipment within the factory.

[0739] "Operational skills" refers to the technical ability of an operator to accurately operate a robot or equipment.

[0740] "Solutions" refers to solutions and approaches to the troubles and problems that robots and equipment face.

[0741] "Interactive training" refers to educational and training programs that are conducted interactively with the user.

[0742] An "emotion engine" refers to technology that analyzes a user's emotional state and generates feedback based on the results.

[0743] "Evaluation results" refer to the analysis results of the user's answers and operations, and include scores for skills and emotional state.

[0744] "Feedback" refers to information that evaluates a user's actions or responses and points out areas for improvement.

[0745] "Server" refers to a computer system that performs functions such as data analysis, participant generation, and feedback provision.

[0746] To implement this invention, the following hardware and software are used. The hardware used is a microphone, a smartphone, or a computer. The software used is the SpeechRecognition library (Python), the Transformers library (Python), and an AI model for emotion analysis. Specific system program processing is described below.

[0747] System configuration and processing content

[0748] The server provides an interface for users to input raw data and document data. Users fill out application forms and resumes using their smartphones or computers. The input data is sent to the server, which analyzes it. Based on the analysis results, the server automatically generates an interlocutor (a virtual interviewer) and questions. These questions are related to specific operational skills and coping methods.

[0749] The generated questions are displayed on the user's device. The user answers the questions by voice using a microphone. The device converts the user's voice data into text data and sends the data to the server.

[0750] The server uses natural language processing technology to analyze the received voice and text data. It also uses an emotion engine to analyze the user's emotional state and reflect it in the evaluation. Based on the evaluation results, the server generates personalized feedback. This feedback may include specific improvements to the user's answers, technical advice, or even feedback based on the user's emotional state. The generated feedback is sent to the user's device and presented to the user.

[0751] Specific examples

[0752] For example, in an operator training system, the question "What would you do if your robot suddenly stopped working?" is generated and presented to the user. The user answers, "First, I would check the error code and try to restart it manually. If that doesn't solve the problem, I would contact technical support." If this answer is judged to be a confident answer after natural language processing and sentiment analysis, the system provides feedback such as, "Your answer is appropriate. It would be a good idea to thoroughly check the error code again before contacting technical support."

[0753] Prompt Sentence Examples

[0754] Project name: Factory robot operator training system

[0755] Question: "What would you do if your robot suddenly stopped working?"

[0756] Answer: "First, check the error code and try a manual reboot. If that doesn't work, contact technical support."

[0757] Sentiment analysis: "This shows confidence. It's the right approach for an engineer."

[0758] Feedback: "The answer is good. I would recommend double-checking the error code before contacting technical support."

[0759] In this way, the system of the present invention can effectively improve the operator's technical skills and provide personalized feedback based on emotional states such as stress and tension.

[0760] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0761] Step 1:

[0762] The server receives raw data or document data from user devices as input. Users use their smartphones or computers to enter information for job applications or resumes and send it to the server. The output is a database entry containing the input information.

[0763] Step 2:

[0764] The server analyzes the input data and automatically generates interlocutors and questions. This analysis process uses natural language processing technology to identify the desired company, industry, operational skills, and countermeasures. The input is data from an application form or resume, and the output is the generated questions and a virtual interlocutor.

[0765] Step 3:

[0766] The terminal presents the generated question to the user, who then answers by voice using a microphone. The input is the presented question, and the output is voice data.

[0767] Step 4:

[0768] The device uses the SpeechRecognition library to convert the user's voice data into text data. The input is the user's voice data, and the output is the data converted from that voice into text.

[0769] Step 5:

[0770] The server receives the converted text and voice data and evaluates them using natural language processing technology and a sentiment analysis engine. The natural language processing technology evaluates the accuracy and quality of the response, while the sentiment analysis engine analyzes the voice intonation and emotional state. The input is text and voice data, and the output is the evaluation result.

[0771] Step 6:

[0772] The server generates feedback based on the evaluation results. The feedback includes suggestions for improving the answers, technical advice, and comments based on the user's emotional state. The input is the evaluation results, and the output is individual feedback.

[0773] Step 7:

[0774] The terminal presents the generated feedback to the user, who can then further improve himself based on this feedback. The input is the feedback, and the output is the feedback display for the user.

[0775] This series of processing steps allows the system of the present invention to provide effective training based on the user's technical ability and emotional state.

[0776] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0777] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0778] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0779] [Third embodiment]

[0780] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0781] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0782] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0783] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0784] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0785] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0786] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0787] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0788] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0789] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0790] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0791] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0792] This invention is a system that utilizes AI to automatically generate interviewers and questions for job seekers and those looking to change jobs, evaluate the answers, and provide individual feedback. This system includes the following means.

[0793] Entering application forms and resumes

[0794] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[0795] Interviewer and Question Generation

[0796] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[0797] User responses and data conversion

[0798] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[0799] Analysis and evaluation of responses

[0800] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the evaluation results, the server assigns a score to the user's answers.

[0801] Generating and Presenting Feedback

[0802] The server generates individualized feedback based on the scoring results. The feedback includes points about the user's answers that are good and that need improvement, as well as suggestions for specific improvements and other ways of expressing the answers. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[0803] Specific examples

[0804] As a concrete example, consider a case where the company the user is applying to is an IT company and the user has experience in project management. The user details their project management experience on an application form and sends it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized.

[0805] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[0806] The processing flow will be explained below.

[0807] Step 1:

[0808] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[0809] Step 2:

[0810] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[0811] Step 3:

[0812] The terminal transmits the user's input data to the server.

[0813] Step 4:

[0814] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[0815] Step 5:

[0816] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[0817] Step 6:

[0818] The server sends the generated interviewer settings and questions to the terminal.

[0819] Step 7:

[0820] The terminal displays the received interviewer settings and questions to the user.

[0821] Step 8:

[0822] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[0823] Step 9:

[0824] The terminal converts the user's voice response into text data.

[0825] Step 10:

[0826] The terminal transmits the converted text data and the original voice data to the server.

[0827] Step 11:

[0828] The server analyzes the received voice and text data.

[0829] Specifically, natural language processing technology is used to analyze the content of the responses, and voice analysis technology is used to detect voice intonation, pauses, and slip-ups.

[0830] Step 12:

[0831] The server evaluates and scores the user's answers based on the analysis results.

[0832] Step 13:

[0833] The server generates personalized feedback based on the scoring results, including points of merit, areas for improvement, and suggestions for specific improvements or other ways of expressing them.

[0834] Step 14:

[0835] The server transmits the generated feedback to the terminal.

[0836] Step 15:

[0837] The terminal displays the received feedback to the user.

[0838] Step 16:

[0839] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[0840] Example 1

[0841] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0842] In traditional job hunting and career change processes, when job seekers individually prepare for interviews, they lack concrete methods for determining appropriate answers to interviewer questions and evaluating the quality of those answers. Furthermore, it is difficult to obtain the feedback needed to improve interview techniques. This makes it difficult for job seekers to effectively showcase their skills.

[0843] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0844] In this invention, the server includes means for a user to input data for an application form or resume, means for analyzing the input data and automatically generating interviewer and questions, means for the user to input answers to the generated questions, means for converting the user's voice data into text data, means for analyzing the converted data and performing an evaluation, means for generating individual feedback based on the evaluation results, and means for presenting the generated feedback to the user. This allows the user to improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[0845] "User" refers to an individual who uses this system to input application form and resume data and simulate an interview.

[0846] An "entry sheet" is a document used to submit personal information to a company when seeking employment, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[0847] A resume is a document that details a job seeker's career history and skills, and is submitted along with an application form.

[0848] The "server" is a central computer system that receives and analyzes application and resume data, generates interviewers and questions, evaluates user response data, and provides feedback.

[0849] A "terminal" is a computer or smart device that allows a user to enter data for an application form or resume, answer generated questions, and receive feedback.

[0850] "Analysis" is the process in which the server processes data from application forms and resumes to identify the company and industry the user is applying to, and then generates interviewers and questions accordingly.

[0851] The "interviewer" is a virtual interviewer that the server generates based on the company and industry that the user is applying to, and who asks appropriate questions.

[0852] The "questions" are generated by the server based on the data in the user's application form or resume, and are part of a virtual interview, similar to the interviewer.

[0853] "Voice data" refers to data generated by voice when a user answers a question using a microphone.

[0854] "Text data" is voice data converted into character information, and is data used by the server for analysis and evaluation.

[0855] "Natural language processing technology" refers to technology used by computers to understand, analyze, and generate human language, including generative AI models such as GPT-4.

[0856] "Scoring" is a process in which the server analyzes the content of the user's answer and then numerically evaluates the quality and appropriateness of the answer.

[0857] "Feedback" is generated by the server based on the scoring results, and includes good points and areas for improvement in the user's answer, specific improvement measures, and other ways of expressing them.

[0858] The present invention is a system for supporting job hunting and career change, and is designed to allow users to input data from application forms and resumes, analyze the data, automatically generate interviewers and questions, evaluate the answers, and provide individual feedback. This system includes the following means.

[0859] System configuration

[0860] This system consists of a terminal used by the user, a server that receives and analyzes data, and software required for data conversion and analysis.

[0861] Entering application forms and resumes

[0862] The user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. When the user enters the required information in each field and presses the send button, the data is sent to the server.

[0863] Data analysis and interviewer and question generation

[0864] The server receives the application form and resume data sent by the user and stores it in a database. The stored data is then analyzed, using natural language processing technology to identify the company and industry the user is applying to.

[0865] The server generates an interviewer based on the identified company and industry. This interviewer is a virtual persona that generates questions appropriate for the specific industry or company. To generate questions, a generative AI model (e.g., GPT-4) is used.

[0866] Posting questions and accepting answers

[0867] The generated question is displayed to the user through the device. The user answers the question by voice using the device's microphone. Once the answer is complete, the voice data is sent to the server by pressing the send button on the device.

[0868] Audio data conversion

[0869] The device converts the user's voice data into text data, for example, using voice recognition software (e.g., Google Cloud Speech-to-Text API), and the text data is sent to the server along with the voice data.

[0870] Analysis and evaluation of responses

[0871] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the analysis results, the server assigns a score to the user's answers.

[0872] Generating and Presenting Feedback

[0873] The server generates individual feedback based on the scoring results. The feedback includes points of merit and improvement for the user's answer, as well as suggestions for specific improvements and other ways of expressing the answer. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[0874] Specific examples

[0875] As a specific example, suppose a user is applying to an information technology company and has experience in project management. The user details their project management experience in an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides specific areas for improvement and points that should be emphasized as feedback.

[0876] Prompt Sentence Examples

[0877] Examples of prompts are:

[0878] "Based on the project management experience stated in the application form, what questions should a user generate when attending an interview?"

[0879] This allows the system to support users in more effectively preparing for interviews, ultimately helping them learn how to present themselves to companies in the right way.

[0880] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0881] Step 1:

[0882] The user uses a terminal to input data for an application form and resume. A form containing fields such as name, address, educational background, work history, skills, and reasons for applying is displayed on the terminal. When the user enters the required information in each field and presses the send button, the data is sent to the server. The input data includes the user's personal information and work history, and the output is the application data sent to the server.

[0883] Step 2:

[0884] The server receives the application form and resume data sent from the terminal. It stores the received data in a database. The input is the application data sent from the terminal, which is then stored in the database, and the output is the user information stored in the database.

[0885] Step 3:

[0886] The server retrieves the user's application form and resume data from the database. The server analyzes the retrieved data and identifies the company and industry the user is applying to. The input is the application data retrieved from the database, which is analyzed using natural language processing technology. The output is the analysis result, which identifies the company and industry.

[0887] Step 4:

[0888] The server generates an interviewer based on the analysis results and generates appropriate questions. This process uses a generative AI model (e.g., GPT-4). The input is specific information about the company and industry from the analysis results, and the server generates interviewers and questions based on this information. The output is the configuration information for the generated interviewer and specific questions.

[0889] Step 5:

[0890] The server sends the generated interviewer information and questions to the terminal. The input is the generated interviewer and question information, which is sent to the terminal. The output is the interviewer and question information received by the terminal.

[0891] Step 6:

[0892] The terminal displays the interviewer information and questions received from the server to the user. The user answers the questions by voice using the terminal's microphone. The inputs are the questions displayed on the terminal and the user's voice responses, and the output is the voice data captured by the microphone.

[0893] Step 7:

[0894] The device converts the user's voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the user's voice data, and the output is the converted text data.

[0895] Step 8:

[0896] The terminal sends voice data and text data to the server. The input is the converted text data and the original voice data, which are then sent to the server. The output is the voice data and text data received by the server.

[0897] Step 9:

[0898] The server analyzes the received voice and text data. This analysis uses natural language processing and voice analysis technologies. The received voice and text data are input and analyzed. The output is an evaluation score as the analysis result.

[0899] Step 10:

[0900] The server scores the user's answers based on the analysis results. The input is the analysis results, and the score is calculated based on these. The output is the scoring result.

[0901] Step 11:

[0902] The server generates feedback based on the scoring results. The feedback includes what was good about the user's answer, what needs to be improved, and specific improvement measures. The input is the scoring results, and the feedback is generated based on these. The output is the generated feedback information.

[0903] Step 12:

[0904] The server sends feedback information to the terminal. The input is the generated feedback information, which is then sent to the terminal. The output is the feedback information received by the terminal.

[0905] Step 13:

[0906] The device displays feedback to the user, who then attempts to improve himself based on the presented feedback. The input is the feedback information received by the device, which is displayed to the user. The output is the user who received the feedback.

[0907] (Application example 1)

[0908] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0909] Interviews are a crucial element in modern job hunting and career change activities, but many job seekers struggle with a lack of interview preparation and practical experience. In particular, there are few systems that allow practical training in real environments such as brick-and-mortar stores. Furthermore, the lack of effective feedback makes it difficult for job seekers to identify areas for improvement.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0911] In this invention, the server includes means for a user to input data such as an application form or resume, means for analyzing the input data and automatically generating interviewers and questions, means for displaying the generated questions on the display of the smart glasses, means for a user to input answers by voice and convert the voice data into text data, means for analyzing and evaluating the user's answers, means for generating individual feedback based on the evaluation results, and means for displaying the generated feedback on the display of the smart glasses. This enables job seekers to practice interviews in real time at a physical store and receive immediate, specific feedback.

[0912] An "entry sheet" is a document that a job seeker submits to a company of their choice, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[0913] A resume is a document that lists a job seeker's personal information, work history, educational background, etc., and is used by companies as a reference during interviews.

[0914] "Analysis" is the process of analyzing input data and extracting specific information or patterns.

[0915] An "interviewer" is a simulated character or virtual persona that conducts interviews with job seekers in a recruitment process.

[0916] "Questions" are inquiries or tasks that an interviewer makes to a job seeker.

[0917] "Smart glasses" are glasses-type devices that have built-in displays and microphones and can display information and record audio in real time.

[0918] "Displaying on a display" means visually presenting information such as text or images on the screen of a device.

[0919] "Voice data" refers to sound waves recorded as digital data, and includes the content of the user's speech.

[0920] "Text data" refers to information written in letters and symbols, and is data obtained by converting voice data into text.

[0921] "Evaluation" refers to analyzing the content of a user's response and determining its quality and appropriateness.

[0922] "Feedback" refers to suggestions for improvement or advice provided based on the evaluation results.

[0923] This invention aims to build an interview simulation system using smart glasses, primarily for use in brick-and-mortar stores. In this system, users input application forms, resumes, etc., and the server analyzes the data to automatically generate appropriate questions and interviewers. The generated questions are then displayed on the smart glasses' display, allowing the user to respond verbally. The voice data is then converted into text data, which is then analyzed and evaluated by the server. Finally, feedback based on the evaluation results is displayed on the smart glasses.

[0924] Hardware and software used

[0925] Smart glasses: Devices with built-in displays and microphones that display information and collect voice.

[0926] Server: A computer system that performs data analysis, question generation, speech and text data analysis, and feedback generation.

[0927] Natural Language Processing (NLP) technologies: Use the Google Cloud Natural Language API or our own trained generative AI models.

[0928] Speech analysis technology: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[0929] As a concrete example, the following scenario can be given:

[0930] 1. User application form entry:

[0931] The user fills out each item on the application form or resume at the reception terminal in the physical store and sends it to the server.

[0932] 2. Question and Interviewer Generation:

[0933] The server analyzes the received data and generates interviewers and questions with expertise that match the industry and position the user is applying for.

[0934] For example: "Tell me about your project management experience."

[0935] 3. User response recording:

[0936] Questions are displayed on the smart glasses' display, and the user answers verbally through a microphone, which converts the voice data into text data.

[0937] 4. Analysis and evaluation of answers:

[0938] The server analyzes the voice and text data and evaluates the answers using natural language processing technology, as well as voice intonation, pauses, and incorrect answers.

[0939] 5. Generating and Presenting Feedback:

[0940] Based on the evaluation results, the server generates specific feedback and presents it on the smart glasses' display.

[0941] Example: "Detail your leadership experiences and accomplishments and include specific examples of teamwork."

[0942] An example of a prompt for a generative AI model is:

[0943] User input data: Company type: IT company, Position: Project manager

[0944] Q: Tell me about your project management experience so far.

[0945] User Voice Answer: I have demonstrated leadership in past projects and completed them on time.

[0946] Expected feedback: Please provide a detailed description of your leadership experiences and achievements, and include specific examples of teamwork.

[0947] Users can practice interviews in real time and receive immediate and effective feedback, which is expected to improve their interview skills.

[0948] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0949] Step 1:

[0950] The user uses a reception terminal in a physical store to input data for an application form or resume. The input data includes name, address, educational background, work history, skills, and reasons for applying. When the user presses the send button, the terminal sends this data to the server. The input data is sent to the server as application form or resume information.

[0951] Step 2:

[0952] The server analyzes the data from the application forms and resumes it receives. During the analysis process, the input data is analyzed using natural language processing technology to identify the industry and position of the company the job seeker is applying for. For example, if a job seeker is applying for a project manager position at an IT company, that information is extracted. Based on the results of this analysis, appropriate questions and interviewers are generated.

[0953] Step 3:

[0954] The server generates a question and displays it on the display of the smart glasses. The user puts on the smart glasses and reads the question. For example, the question "Tell me about your project management experience" is displayed on the display.

[0955] Step 4:

[0956] The user answers by voice through the microphone in the smart glasses. The voice response is collected and sent to the server through the smart glasses. The voice data is sent to the server as input data, and then the voice data is converted to text data using the Google Cloud Speech-to-Text API.

[0957] Step 5:

[0958] The server analyzes the voice and text data and evaluates the user's answers using natural language processing technology. This evaluation process determines the quality and appropriateness of the answers, and analyzes voice intonation, pauses, and incorrect answers. The analyzed evaluation results are generated as output data.

[0959] Step 6:

[0960] The server generates individual feedback based on the evaluation results. For example, it might say, "It would be good to explain your leadership experiences and achievements in detail and add specific examples of teamwork." This feedback data is generated and saved as output data.

[0961] Step 7:

[0962] The feedback is presented on the display of the smart glasses, and the user can review the presented feedback and use it to improve themselves. The feedback is displayed and provided to the user.

[0963] This allows job seekers to practice interviews in real time in a physical store and receive specific and immediate feedback, which is expected to improve their interview skills.

[0964] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0965] The present invention relates to a system that utilizes AI for job-seekers and career changers to automatically generate interview questions, evaluate the answers, and provide personalized feedback. By incorporating an emotion engine into this system, it is possible to recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[0966] Entering application forms and resumes

[0967] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[0968] Interviewer and Question Generation

[0969] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[0970] User responses and data conversion

[0971] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[0972] Analysis and evaluation of responses

[0973] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's responses, while voice analysis is used to detect vocal intonation, pauses, and slip-ups. An emotion engine can also be used to analyze the user's emotional state. This allows the user's emotional state to be reflected in the evaluation along with the content of the responses.

[0974] Generating and Presenting Feedback

[0975] The server generates individualized feedback based on the scoring results and the analysis results of the emotion engine. The feedback includes points that indicate what is good and what needs to be improved about the user's answer, as well as suggestions for specific improvements and other ways of expressing themselves. The generated feedback is sent to the device and presented to the user. The user can use this feedback to improve themselves.

[0976] Specific examples

[0977] As a specific example, consider a case where the company the user is applying to is an IT company and has experience in project management. The user details their project management experience on an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized. Furthermore, it can use an emotion engine to analyze the user's level of confidence and nervousness, and provide emotion-based feedback such as, "I like the fact that you speak with confidence."

[0978] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[0979] The processing flow will be explained below.

[0980] Step 1:

[0981] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[0982] Step 2:

[0983] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[0984] Step 3:

[0985] The terminal transmits the user's input data to the server.

[0986] Step 4:

[0987] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[0988] Step 5:

[0989] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[0990] Step 6:

[0991] The server sends the generated interviewer settings and questions to the terminal.

[0992] Step 7:

[0993] The terminal displays the received interviewer settings and questions to the user.

[0994] Step 8:

[0995] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[0996] Step 9:

[0997] The terminal converts the user's voice response into text data.

[0998] Step 10:

[0999] The terminal transmits the converted text data and the original voice data to the server.

[1000] Step 11:

[1001] The server analyzes the received voice and text data. Specifically, it uses natural language processing technology to analyze the content of the responses, and voice analysis technology to detect voice intonation, pauses, and slip-ups.

[1002] Step 12:

[1003] The server uses an emotion engine in addition to analysis to analyze the user's emotional state (e.g., confidence, anxiety, joy, sadness). It understands emotions from voice data and reads emotions from text data.

[1004] Step 13:

[1005] The server evaluates and scores the user's answers based on the analysis results, taking into account their emotional state.

[1006] Step 14:

[1007] The server generates personalized feedback based on the scoring and sentiment analysis results. The feedback includes points of merit and improvement for the user's answer, specific suggestions for improvement, suggestions for alternative ways of expressing oneself, and even advice based on emotions (e.g., "I liked that you spoke with confidence" or "You seem a little nervous, so try to relax").

[1008] Step 15:

[1009] The server transmits the generated feedback to the terminal.

[1010] Step 16:

[1011] The terminal displays the received feedback to the user.

[1012] Step 17:

[1013] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[1014] With this specific processing flow, the system of the present invention not only helps users hone their skills by gaining experience in interviews and learn how to present themselves to the right companies, but also enables more advanced self-improvement by providing feedback based on their emotional state.

[1015] Example 2

[1016] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1017] In traditional job hunting and career change processes, interview practice and feedback are extremely important, but are difficult to provide individually. Providing appropriate feedback that takes into account the user's emotional state is particularly important, but manual evaluation has its limitations. Furthermore, while the intonation and pauses in a user's voice responses are important, these analyses have not yet been automated. Therefore, there is a need for a system that can provide individualized and precise feedback.

[1018] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input data from an information recording medium, a means for analyzing the input data and automatically generating a question and answerer and a question, a means for the user to input an answer to the generated question, a means for converting the user's answer into voice and text data, a means for analyzing the converted data and making an evaluation, a means for analyzing the user's emotional state using an emotion analysis device and reflecting the analysis in the evaluation, a means for generating individual feedback based on the evaluation result, and a means for presenting the generated feedback to the user. This makes it possible to provide precise feedback that takes into account the user's emotional state and voice characteristics.

[1019] "Information recording medium" refers to a medium that records information about a user, such as an application form or resume, in digital format.

[1020] The "question and answerer" refers to a virtual interviewer that is automatically generated based on the data input by the user, and is a character that asks questions that correspond to a specific industry or company.

[1021] "Means for converting" refers to the technical means for converting the data provided by the user via voice into text data, including voice recognition software.

[1022] "Analysis means" refers to a series of technologies for analyzing input data and converted data, including natural language processing technology and voice analysis technology.

[1023] "Means for evaluation" refers to technical means for evaluating the content of a user's responses and emotional state based on the analyzed data.

[1024] An "emotion analysis device" refers to technology that analyzes a user's emotional state from their voice or text data and reflects the results in their evaluation.

[1025] "Personalized feedback" refers to feedback that includes user-specific improvements and advice that is generated based on the user's specific answers and emotional state.

[1026] The present invention relates to a system that utilizes AI to automatically generate interview questions for job seekers and those seeking a career change, evaluates the answers, and provides individualized feedback. By incorporating an emotion analysis device, the system can recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[1027] Entering application forms and resumes

[1028] First, the user uses a terminal to enter information for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. For example, the input form is provided using a web browser or a dedicated application. When the user enters the required information and presses the send button, the data is sent to the server.

[1029] Receiving and parsing input data

[1030] The server receives the application form and resume data sent from the device. The received data is analyzed using natural language processing technology. Specifically, it uses Python's NLTK library and SpaCy to analyze the text data and identify the user's desired industry and company. Further processing is carried out based on the results of this analysis.

[1031] Interviewer Generation

[1032] Based on the analysis results, the server generates a Q&A person suited to the user's desired company and industry. This interviewer is customized using a generative AI model. For example, a natural language generation model such as ChatGPT is used to create a virtual interviewer with settings suited to interviews at specific companies and industries.

[1033] Question Generation

[1034] The server then generates appropriate questions for the user. By inputting appropriate prompts into the generative AI model, questions that match the user's aspirations are generated. For example, for a user with experience in project management, the server generates the question, "Please tell us specifically about your project management experience."

[1035] Posing the Question

[1036] The server then sends the generated question to the device, which displays it to the user, who answers verbally using a microphone while looking at the screen. The device then converts the speech to text using speech recognition software such as Google Cloud Speech-to-Text.

[1037] Sending response data

[1038] The device then sends the converted text and voice data to a server, which then receives the data and analyzes it using natural language processing and voice analysis technologies.

[1039] Analysis of responses and assessment of emotional state

[1040] The server uses natural language processing technology (e.g., the BERT model) to analyze the content of the user's responses. It also uses speech analysis technology to evaluate voice intonation, pauses, and slip-ups. It also uses EmotionAPI and Affectiva to analyze the user's emotional state. The analysis results include the content of the responses, voice quality, and emotional state.

[1041] Generate feedback

[1042] The server generates personalized feedback based on the analysis results. The feedback includes points for evaluation, areas for improvement, and specific measures for improvement. For example, feedback such as, "Your specific details about your project management experience are good. However, it would be even better if you also included the tools you used and the lessons you learned."

[1043] Providing feedback

[1044] The device receives the feedback sent from the server and displays it to the user, who can then view the feedback on the device screen and get clues to improve their answers.

[1045] Examples of specific examples and prompts

[1046] As a concrete example, consider the case where User A is applying to an IT company and has experience in project management. User A details his / her "project management experience" on an application form and sends it to the server. The server analyzes this data and generates an appropriate question and answerer along with the question "Tell us about your project management experience." When User A answers, the server analyzes the content and generates feedback such as "The details are specific, but it would be even better if you also added the tools you used and the lessons you learned," and sends it to the terminal.

[1047] An example of a prompt sentence to input to the generative AI model is as follows:

[1048] Plan: Generate interview questions and feedback for a user with project management experience who is applying to an IT company. Generate questions based on the information in the application form, analyze the user's answers using natural language processing and an emotion engine, and provide feedback such as the following:

[1049] Application form: As for project management experience, please state that you have led multiple projects in the past and achieved results.

[1050] Q: Tell us about your project management experience.

[1051] User Answer: "I led multiple projects and achieved results."

[1052] Feedback: "You're good at describing your project management experience in detail, but you could expand on specific accomplishments, tools you used, and lessons learned. You also speak with confidence, but you seem a little nervous. I think you could practice speaking in a more relaxed manner."

[1053] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[1054] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1055] Step 1:

[1056] The user inputs data for the information recording medium.

[1057] Specifically, the user uses a terminal to fill out a form containing information such as name, address, educational background, work history, skills, and motivation for applying. After entering the information, the user presses the send button, which sends the entered data from the terminal to the server.

[1058] Input: Application form and resume data entered by the user into the terminal

[1059] Output: User entry data sent to the server

[1060] Step 2:

[1061] The server receives and parses the input data.

[1062] Specifically, the server receives application forms and resumes sent from the device. The received data is analyzed using natural language processing technologies such as Python's NLTK library and SpaCy. This allows the user's desired company and industry to be identified.

[1063] Input: Application form and resume data sent from the device

[1064] Output: Information on the company and industry the user is interested in as a result of the analysis

[1065] Step 3:

[1066] The server generates the interviewers.

[1067] Specifically, the server uses a generative AI model based on the analysis results to generate a question and answerer suited to the company and industry the user is applying for. This question and answerer is designed as a virtual interviewer with extensive knowledge of the specific industry or company.

[1068] Input: Information on the company and industry you are applying to as a result of the analysis

[1069] Output: Virtual interviewer as question and answerer

[1070] Step 4:

[1071] The server generates the question.

[1072] Specifically, the server inputs a prompt into the generative AI model and generates a question that matches the user's desired profile. For example, a question like, "Please tell us specifically about your project management experience" might be generated.

[1073] Input: Prompt statement and user parsed data

[1074] Output: Generated questions

[1075] Step 5:

[1076] The terminal presents the question to the user.

[1077] Specifically, the device displays the questions sent from the server to the user, who then answers verbally using a microphone while looking at the device screen.

[1078] Input: The question sent by the server

[1079] Output: The terminal screen showing the question

[1080] Step 6:

[1081] The user responds verbally, and the device converts the speech into text.

[1082] The user answers questions by speaking into a microphone, and the device uses voice recognition software such as Google Cloud Speech-to-Text to convert the answers into text.

[1083] Input: User's spoken response

[1084] Output: User answers converted to text data

[1085] Step 7:

[1086] The terminal transmits the response data to the server.

[1087] Specifically, the terminal converts the user's answers into text data and transmits them to the server, which then receives the data.

[1088] Input: User answers converted to text data

[1089] Output: Response data sent to the server

[1090] Step 8:

[1091] The server analyzes the responses and assesses the emotional state.

[1092] Specifically, the server analyzes the user's responses using natural language processing technology (such as the BERT model), evaluates the intonation, pauses, and slip-ups of the user's voice using speech analysis technology, and analyzes the user's emotional state using EmotionAPI and Affectiva.

[1093] Input: Response data sent to the server

[1094] Output: Answer content, voice characteristics, emotional state as analysis results

[1095] Step 9:

[1096] The server generates personalized feedback.

[1097] Specifically, the server generates individual feedback based on the analysis results, including evaluation points for the answer, areas for improvement, and specific improvement measures.

[1098] Input: Answer content as analysis result, voice characteristics, emotional state

[1099] Output: Generated feedback

[1100] Step 10:

[1101] The terminal presents feedback to the user.

[1102] Specifically, the device receives the feedback sent from the server and displays it to the user, who can then improve themselves based on this feedback.

[1103] Input: Feedback sent by the server

[1104] Output: Feedback displayed on the terminal

[1105] Through these steps, the system helps users improve themselves through interview practice, allowing them to receive more detailed feedback, including their emotional state.

[1106] (Application example 2)

[1107] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Improving the technical skills and troubleshooting abilities of robot operators is an essential challenge in modern factories. However, conventional education and training systems struggle to assess actual operating skills and emotional states in real time and provide individualized feedback. This makes it difficult for operators to effectively acquire the necessary skills. Furthermore, conventional systems do not provide feedback that takes into account the user's emotional state, making it difficult to alleviate stress and tension.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input raw data or document data, a means for analyzing the input data and automatically generating questions for the interlocutor, a means for the user to input answers to the generated questions, a means for providing interactive training including questions about the operator's operating skills and how to deal with the situation, and a means for analyzing the operator's emotional state using an emotion engine and reflecting the results of the evaluation. This makes it possible to effectively improve the operator's technical skills and provide individual feedback based on their emotional state, such as stress or tension.

[1110] "Raw data" refers to data acquired in real time or as is.

[1111] "Document data" refers to information recorded in the form of a resume, application form, etc.

[1112] The "interlocutor" refers to a virtual interviewer or interviewer that is generated to have a dialogue with the user.

[1113] A "question" refers to a question or interrogation posed by the interlocutor to the user.

[1114] "Operator" refers to the engineers and workers who operate robots and equipment within the factory.

[1115] "Operational skills" refers to the technical ability of an operator to accurately operate a robot or equipment.

[1116] "Solutions" refers to solutions and approaches to the troubles and problems that robots and equipment face.

[1117] "Interactive training" refers to educational and training programs that are conducted interactively with the user.

[1118] An "emotion engine" refers to technology that analyzes a user's emotional state and generates feedback based on the results.

[1119] "Evaluation results" refer to the analysis results of the user's answers and operations, and include scores for skills and emotional state.

[1120] "Feedback" refers to information that evaluates a user's actions or responses and points out areas for improvement.

[1121] "Server" refers to a computer system that performs functions such as data analysis, participant generation, and feedback provision.

[1122] To implement this invention, the following hardware and software are used. The hardware used is a microphone, a smartphone, or a computer. The software used is the SpeechRecognition library (Python), the Transformers library (Python), and an AI model for emotion analysis. Specific system program processing is described below.

[1123] System configuration and processing content

[1124] The server provides an interface for users to input raw data and document data. Users fill out application forms and resumes using their smartphones or computers. The input data is sent to the server, which analyzes it. Based on the analysis results, the server automatically generates an interlocutor (a virtual interviewer) and questions. These questions are related to specific operational skills and coping methods.

[1125] The generated questions are displayed on the user's device. The user answers the questions by voice using a microphone. The device converts the user's voice data into text data and sends the data to the server.

[1126] The server uses natural language processing technology to analyze the received voice and text data. It also uses an emotion engine to analyze the user's emotional state and reflect it in the evaluation. Based on the evaluation results, the server generates personalized feedback. This feedback may include specific improvements to the user's answers, technical advice, or even feedback based on the user's emotional state. The generated feedback is sent to the user's device and presented to the user.

[1127] Specific examples

[1128] For example, in an operator training system, the question "What would you do if your robot suddenly stopped working?" is generated and presented to the user. The user answers, "First, I would check the error code and try to restart it manually. If that doesn't solve the problem, I would contact technical support." If this answer is judged to be a confident answer after natural language processing and sentiment analysis, the system provides feedback such as, "Your answer is appropriate. It would be a good idea to thoroughly check the error code again before contacting technical support."

[1129] Prompt Sentence Examples

[1130] Project name: Factory robot operator training system

[1131] Question: "What would you do if your robot suddenly stopped working?"

[1132] Answer: "First, check the error code and try a manual reboot. If that doesn't work, contact technical support."

[1133] Sentiment analysis: "This shows confidence. It's the right approach for an engineer."

[1134] Feedback: "The answer is good. I would recommend double-checking the error code before contacting technical support."

[1135] In this way, the system of the present invention can effectively improve the operator's technical skills and provide personalized feedback based on emotional states such as stress and tension.

[1136] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1137] Step 1:

[1138] The server receives raw data or document data from user devices as input. Users use their smartphones or computers to enter information for job applications or resumes and send it to the server. The output is a database entry containing the input information.

[1139] Step 2:

[1140] The server analyzes the input data and automatically generates interlocutors and questions. This analysis process uses natural language processing technology to identify the desired company, industry, operational skills, and countermeasures. The input is data from an application form or resume, and the output is the generated questions and a virtual interlocutor.

[1141] Step 3:

[1142] The terminal presents the generated question to the user, who then answers by voice using a microphone. The input is the presented question, and the output is voice data.

[1143] Step 4:

[1144] The device uses the SpeechRecognition library to convert the user's voice data into text data. The input is the user's voice data, and the output is the data converted from that voice into text.

[1145] Step 5:

[1146] The server receives the converted text and voice data and evaluates them using natural language processing technology and a sentiment analysis engine. The natural language processing technology evaluates the accuracy and quality of the response, while the sentiment analysis engine analyzes the voice intonation and emotional state. The input is text and voice data, and the output is the evaluation result.

[1147] Step 6:

[1148] The server generates feedback based on the evaluation results. The feedback includes suggestions for improving the answers, technical advice, and comments based on the user's emotional state. The input is the evaluation results, and the output is individual feedback.

[1149] Step 7:

[1150] The terminal presents the generated feedback to the user, who can then further improve himself based on this feedback. The input is the feedback, and the output is the feedback display for the user.

[1151] This series of processing steps allows the system of the present invention to provide effective training based on the user's technical ability and emotional state.

[1152] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1153] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1154] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1155] [Fourth embodiment]

[1156] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1157] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1158] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1159] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1160] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1161] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1162] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1163] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1164] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1165] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1166] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1167] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1168] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1169] This invention is a system that utilizes AI to automatically generate interviewers and questions for job seekers and those looking to change jobs, evaluate the answers, and provide individual feedback. This system includes the following means.

[1170] Entering application forms and resumes

[1171] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[1172] Interviewer and Question Generation

[1173] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[1174] User responses and data conversion

[1175] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[1176] Analysis and evaluation of responses

[1177] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the evaluation results, the server assigns a score to the user's answers.

[1178] Generating and Presenting Feedback

[1179] The server generates individualized feedback based on the scoring results. The feedback includes points about the user's answers that are good and that need improvement, as well as suggestions for specific improvements and other ways of expressing the answers. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[1180] Specific examples

[1181] As a concrete example, consider a case where the company the user is applying to is an IT company and the user has experience in project management. The user details their project management experience on an application form and sends it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized.

[1182] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[1183] The processing flow will be explained below.

[1184] Step 1:

[1185] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[1186] Step 2:

[1187] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[1188] Step 3:

[1189] The terminal transmits the user's input data to the server.

[1190] Step 4:

[1191] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[1192] Step 5:

[1193] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[1194] Step 6:

[1195] The server sends the generated interviewer settings and questions to the terminal.

[1196] Step 7:

[1197] The terminal displays the received interviewer settings and questions to the user.

[1198] Step 8:

[1199] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[1200] Step 9:

[1201] The terminal converts the user's voice response into text data.

[1202] Step 10:

[1203] The terminal transmits the converted text data and the original voice data to the server.

[1204] Step 11:

[1205] The server analyzes the received voice and text data.

[1206] Specifically, natural language processing technology is used to analyze the content of the responses, and voice analysis technology is used to detect voice intonation, pauses, and slip-ups.

[1207] Step 12:

[1208] The server evaluates and scores the user's answers based on the analysis results.

[1209] Step 13:

[1210] The server generates personalized feedback based on the scoring results, including points of merit, areas for improvement, and suggestions for specific improvements or other ways of expressing them.

[1211] Step 14:

[1212] The server transmits the generated feedback to the terminal.

[1213] Step 15:

[1214] The terminal displays the received feedback to the user.

[1215] Step 16:

[1216] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[1217] Example 1

[1218] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1219] In traditional job hunting and career change processes, when job seekers individually prepare for interviews, they lack concrete methods for determining appropriate answers to interviewer questions and evaluating the quality of those answers. Furthermore, it is difficult to obtain the feedback needed to improve interview techniques. This makes it difficult for job seekers to effectively showcase their skills.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1221] In this invention, the server includes means for a user to input data for an application form or resume, means for analyzing the input data and automatically generating interviewer and questions, means for the user to input answers to the generated questions, means for converting the user's voice data into text data, means for analyzing the converted data and performing an evaluation, means for generating individual feedback based on the evaluation results, and means for presenting the generated feedback to the user. This allows the user to improve their skills by gaining experience in interviews and learn how to properly appeal to companies.

[1222] "User" refers to an individual who uses this system to input application form and resume data and simulate an interview.

[1223] An "entry sheet" is a document used to submit personal information to a company when seeking employment, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[1224] A resume is a document that details a job seeker's career history and skills, and is submitted along with an application form.

[1225] The "server" is a central computer system that receives and analyzes application and resume data, generates interviewers and questions, evaluates user response data, and provides feedback.

[1226] A "terminal" is a computer or smart device that allows a user to enter data for an application form or resume, answer generated questions, and receive feedback.

[1227] "Analysis" is the process in which the server processes data from application forms and resumes to identify the company and industry the user is applying to, and then generates interviewers and questions accordingly.

[1228] The "interviewer" is a virtual interviewer that the server generates based on the company and industry that the user is applying to, and who asks appropriate questions.

[1229] The "questions" are generated by the server based on the data in the user's application form or resume, and are part of a virtual interview, similar to the interviewer.

[1230] "Voice data" refers to data generated by voice when a user answers a question using a microphone.

[1231] "Text data" is voice data converted into character information, and is data used by the server for analysis and evaluation.

[1232] "Natural language processing technology" refers to technology used by computers to understand, analyze, and generate human language, including generative AI models such as GPT-4.

[1233] "Scoring" is a process in which the server analyzes the content of the user's answer and then numerically evaluates the quality and appropriateness of the answer.

[1234] "Feedback" is generated by the server based on the scoring results, and includes good points and areas for improvement in the user's answer, specific improvement measures, and other ways of expressing them.

[1235] The present invention is a system for supporting job hunting and career change, and is designed to allow users to input data from application forms and resumes, analyze the data, automatically generate interviewers and questions, evaluate the answers, and provide individual feedback. This system includes the following means.

[1236] System configuration

[1237] This system consists of a terminal used by the user, a server that receives and analyzes data, and software required for data conversion and analysis.

[1238] Entering application forms and resumes

[1239] The user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. When the user enters the required information in each field and presses the send button, the data is sent to the server.

[1240] Data analysis and interviewer and question generation

[1241] The server receives the application form and resume data sent by the user and stores it in a database. The stored data is then analyzed, using natural language processing technology to identify the company and industry the user is applying to.

[1242] The server generates an interviewer based on the identified company and industry. This interviewer is a virtual persona that generates questions appropriate for the specific industry or company. To generate questions, a generative AI model (e.g., GPT-4) is used.

[1243] Posting questions and accepting answers

[1244] The generated question is displayed to the user through the device. The user answers the question by voice using the device's microphone. Once the answer is complete, the voice data is sent to the server by pressing the send button on the device.

[1245] Audio data conversion

[1246] The device converts the user's voice data into text data, for example, using voice recognition software (e.g., Google Cloud Speech-to-Text API), and the text data is sent to the server along with the voice data.

[1247] Analysis and evaluation of responses

[1248] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's answers, and voice analysis is used to detect voice intonation, pauses, and slip-ups. Based on the analysis results, the server assigns a score to the user's answers.

[1249] Generating and Presenting Feedback

[1250] The server generates individual feedback based on the scoring results. The feedback includes points of merit and improvement for the user's answer, as well as suggestions for specific improvements and other ways of expressing the answer. The generated feedback is sent to the terminal and presented to the user. The user can use this feedback to improve themselves.

[1251] Specific examples

[1252] As a specific example, suppose a user is applying to an information technology company and has experience in project management. The user details their project management experience in an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I have led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides specific areas for improvement and points that should be emphasized as feedback.

[1253] Prompt Sentence Examples

[1254] Examples of prompts are:

[1255] "Based on the project management experience stated in the application form, what questions should a user generate when attending an interview?"

[1256] This allows the system to support users in more effectively preparing for interviews, ultimately helping them learn how to present themselves to companies in the right way.

[1257] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1258] Step 1:

[1259] The user uses a terminal to input data for an application form and resume. A form containing fields such as name, address, educational background, work history, skills, and reasons for applying is displayed on the terminal. When the user enters the required information in each field and presses the send button, the data is sent to the server. The input data includes the user's personal information and work history, and the output is the application data sent to the server.

[1260] Step 2:

[1261] The server receives the application form and resume data sent from the terminal. It stores the received data in a database. The input is the application data sent from the terminal, which is then stored in the database, and the output is the user information stored in the database.

[1262] Step 3:

[1263] The server retrieves the user's application form and resume data from the database. The server analyzes the retrieved data and identifies the company and industry the user is applying to. The input is the application data retrieved from the database, which is analyzed using natural language processing technology. The output is the analysis result, which identifies the company and industry.

[1264] Step 4:

[1265] The server generates an interviewer based on the analysis results and generates appropriate questions. This process uses a generative AI model (e.g., GPT-4). The input is specific information about the company and industry from the analysis results, and the server generates interviewers and questions based on this information. The output is the configuration information for the generated interviewer and specific questions.

[1266] Step 5:

[1267] The server sends the generated interviewer information and questions to the terminal. The input is the generated interviewer and question information, which is sent to the terminal. The output is the interviewer and question information received by the terminal.

[1268] Step 6:

[1269] The terminal displays the interviewer information and questions received from the server to the user. The user answers the questions by voice using the terminal's microphone. The inputs are the questions displayed on the terminal and the user's voice responses, and the output is the voice data captured by the microphone.

[1270] Step 7:

[1271] The device converts the user's voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the user's voice data, and the output is the converted text data.

[1272] Step 8:

[1273] The terminal sends voice data and text data to the server. The input is the converted text data and the original voice data, which are then sent to the server. The output is the voice data and text data received by the server.

[1274] Step 9:

[1275] The server analyzes the received voice and text data. This analysis uses natural language processing and voice analysis technologies. The received voice and text data are input and analyzed. The output is an evaluation score as the analysis result.

[1276] Step 10:

[1277] The server scores the user's answers based on the analysis results. The input is the analysis results, and the score is calculated based on these. The output is the scoring result.

[1278] Step 11:

[1279] The server generates feedback based on the scoring results. The feedback includes what was good about the user's answer, what needs to be improved, and specific improvement measures. The input is the scoring results, and the feedback is generated based on these. The output is the generated feedback information.

[1280] Step 12:

[1281] The server sends feedback information to the terminal. The input is the generated feedback information, which is then sent to the terminal. The output is the feedback information received by the terminal.

[1282] Step 13:

[1283] The device displays feedback to the user, who then attempts to improve himself based on the presented feedback. The input is the feedback information received by the device, which is displayed to the user. The output is the user who received the feedback.

[1284] (Application example 1)

[1285] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] Interviews are a crucial element in modern job hunting and career change activities, but many job seekers struggle with a lack of interview preparation and practical experience. In particular, there are few systems that allow practical training in real environments such as brick-and-mortar stores. Furthermore, the lack of effective feedback makes it difficult for job seekers to identify areas for improvement.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1288] In this invention, the server includes means for a user to input data such as an application form or resume, means for analyzing the input data and automatically generating interviewers and questions, means for displaying the generated questions on the display of the smart glasses, means for a user to input answers by voice and convert the voice data into text data, means for analyzing and evaluating the user's answers, means for generating individual feedback based on the evaluation results, and means for displaying the generated feedback on the display of the smart glasses. This enables job seekers to practice interviews in real time at a physical store and receive immediate, specific feedback.

[1289] An "entry sheet" is a document that a job seeker submits to a company of their choice, and includes information such as name, address, educational background, work history, skills, and reasons for applying.

[1290] A resume is a document that lists a job seeker's personal information, work history, educational background, etc., and is used by companies as a reference during interviews.

[1291] "Analysis" is the process of analyzing input data and extracting specific information or patterns.

[1292] An "interviewer" is a simulated character or virtual persona that conducts interviews with job seekers in a recruitment process.

[1293] "Questions" are inquiries or tasks that an interviewer makes to a job seeker.

[1294] "Smart glasses" are glasses-type devices that have built-in displays and microphones and can display information and record audio in real time.

[1295] "Displaying on a display" means visually presenting information such as text or images on the screen of a device.

[1296] "Voice data" refers to sound waves recorded as digital data, and includes the content of the user's speech.

[1297] "Text data" refers to information written in letters and symbols, and is data obtained by converting voice data into text.

[1298] "Evaluation" refers to analyzing the content of a user's response and determining its quality and appropriateness.

[1299] "Feedback" refers to suggestions for improvement or advice provided based on the evaluation results.

[1300] This invention aims to build an interview simulation system using smart glasses, primarily for use in brick-and-mortar stores. In this system, users input application forms, resumes, etc., and the server analyzes the data to automatically generate appropriate questions and interviewers. The generated questions are then displayed on the smart glasses' display, allowing the user to respond verbally. The voice data is then converted into text data, which is then analyzed and evaluated by the server. Finally, feedback based on the evaluation results is displayed on the smart glasses.

[1301] Hardware and software used

[1302] Smart glasses: Devices with built-in displays and microphones that display information and collect voice.

[1303] Server: A computer system that performs data analysis, question generation, speech and text data analysis, and feedback generation.

[1304] Natural Language Processing (NLP) technologies: Use the Google Cloud Natural Language API or our own trained generative AI models.

[1305] Speech analysis technology: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[1306] As a concrete example, the following scenario can be given:

[1307] 1. User application form entry:

[1308] The user fills out each item on the application form or resume at the reception terminal in the physical store and sends it to the server.

[1309] 2. Question and Interviewer Generation:

[1310] The server analyzes the received data and generates interviewers and questions with expertise that match the industry and position the user is applying for.

[1311] For example: "Tell me about your project management experience."

[1312] 3. User response recording:

[1313] Questions are displayed on the smart glasses' display, and the user answers verbally through a microphone, which converts the voice data into text data.

[1314] 4. Analysis and evaluation of answers:

[1315] The server analyzes the voice and text data and evaluates the answers using natural language processing technology, as well as voice intonation, pauses, and incorrect answers.

[1316] 5. Generating and Presenting Feedback:

[1317] Based on the evaluation results, the server generates specific feedback and presents it on the smart glasses' display.

[1318] Example: "Detail your leadership experiences and accomplishments and include specific examples of teamwork."

[1319] An example of a prompt for a generative AI model is:

[1320] User input data: Company type: IT company, Position: Project manager

[1321] Q: Tell me about your project management experience so far.

[1322] User Voice Answer: I have demonstrated leadership in past projects and completed them on time.

[1323] Expected feedback: Please provide a detailed description of your leadership experiences and achievements, and include specific examples of teamwork.

[1324] Users can practice interviews in real time and receive immediate and effective feedback, which is expected to improve their interview skills.

[1325] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1326] Step 1:

[1327] The user uses a reception terminal in a physical store to input data for an application form or resume. The input data includes name, address, educational background, work history, skills, and reasons for applying. When the user presses the send button, the terminal sends this data to the server. The input data is sent to the server as application form or resume information.

[1328] Step 2:

[1329] The server analyzes the data from the application forms and resumes it receives. During the analysis process, the input data is analyzed using natural language processing technology to identify the industry and position of the company the job seeker is applying for. For example, if a job seeker is applying for a project manager position at an IT company, that information is extracted. Based on the results of this analysis, appropriate questions and interviewers are generated.

[1330] Step 3:

[1331] The server generates a question and displays it on the display of the smart glasses. The user puts on the smart glasses and reads the question. For example, the question "Tell me about your project management experience" is displayed on the display.

[1332] Step 4:

[1333] The user answers by voice through the microphone in the smart glasses. The voice response is collected and sent to the server through the smart glasses. The voice data is sent to the server as input data, and then the voice data is converted to text data using the Google Cloud Speech-to-Text API.

[1334] Step 5:

[1335] The server analyzes the voice and text data and evaluates the user's answers using natural language processing technology. This evaluation process determines the quality and appropriateness of the answers, and analyzes voice intonation, pauses, and incorrect answers. The analyzed evaluation results are generated as output data.

[1336] Step 6:

[1337] The server generates individual feedback based on the evaluation results. For example, it might say, "It would be good to explain your leadership experiences and achievements in detail and add specific examples of teamwork." This feedback data is generated and saved as output data.

[1338] Step 7:

[1339] The feedback is presented on the display of the smart glasses, and the user can review the presented feedback and use it to improve themselves. The feedback is displayed and provided to the user.

[1340] This allows job seekers to practice interviews in real time in a physical store and receive specific and immediate feedback, which is expected to improve their interview skills.

[1341] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1342] The present invention relates to a system that utilizes AI for job-seekers and career changers to automatically generate interview questions, evaluate the answers, and provide personalized feedback. By incorporating an emotion engine into this system, it is possible to recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[1343] Entering application forms and resumes

[1344] First, the user uses the terminal to enter data for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. The user enters the required information in each field and presses the send button to send the data to the server.

[1345] Interviewer and Question Generation

[1346] The server receives and analyzes the application form and resume data entered by the user. As a result of the analysis, the server identifies the company and industry the user is applying to and generates a corresponding interviewer. This interviewer has settings appropriate for interviews in a specific industry or company. The server then generates appropriate questions for the user. For example, for a user applying to an IT company, the server generates a question such as, "Tell us about your project management experience to date."

[1347] User responses and data conversion

[1348] The generated question is displayed to the user through the terminal, and the user answers the question by voice using a microphone. Once the answer is complete, the terminal converts the user's voice data into text data and sends it to the server.

[1349] Analysis and evaluation of responses

[1350] The server analyzes the received voice and text data. Natural language processing and voice analysis technologies are used for the analysis. Natural language processing is used to evaluate the content of the user's responses, while voice analysis is used to detect vocal intonation, pauses, and slip-ups. An emotion engine can also be used to analyze the user's emotional state. This allows the user's emotional state to be reflected in the evaluation along with the content of the responses.

[1351] Generating and Presenting Feedback

[1352] The server generates individualized feedback based on the scoring results and the analysis results of the emotion engine. The feedback includes points that indicate what is good and what needs to be improved about the user's answer, as well as suggestions for specific improvements and other ways of expressing themselves. The generated feedback is sent to the device and presented to the user. The user can use this feedback to improve themselves.

[1353] Specific examples

[1354] As a specific example, consider a case where the company the user is applying to is an IT company and has experience in project management. The user details their project management experience on an application form and submits it to the server. The server analyzes this data and generates an appropriate interviewer along with the question, "Tell us about your project management experience." If the user answers, "I led multiple projects and achieved results," the server uses natural language processing technology to analyze the content and provides feedback on specific areas for improvement and points that should be emphasized. Furthermore, it can use an emotion engine to analyze the user's level of confidence and nervousness, and provide emotion-based feedback such as, "I like the fact that you speak with confidence."

[1355] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[1356] The processing flow will be explained below.

[1357] Step 1:

[1358] The terminal displays a form for the user to fill out an application form or resume, including information such as name, address, educational background, work history, skills, and motivation for applying.

[1359] Step 2:

[1360] The user enters the required information into each field and clicks the "Submit" button when the information is complete.

[1361] Step 3:

[1362] The terminal transmits the user's input data to the server.

[1363] Step 4:

[1364] The server analyzes the data from the application forms and resumes it receives, using natural language processing technology.

[1365] Step 5:

[1366] The server identifies the company and industry the user is interested in based on the analysis results, generates an appropriate interviewer character, and then generates questions for the interviewer to ask.

[1367] Step 6:

[1368] The server sends the generated interviewer settings and questions to the terminal.

[1369] Step 7:

[1370] The terminal displays the received interviewer settings and questions to the user.

[1371] Step 8:

[1372] The user answers the displayed questions by voice using a microphone, and when the answer is complete, presses the "Submit" button.

[1373] Step 9:

[1374] The terminal converts the user's voice response into text data.

[1375] Step 10:

[1376] The terminal transmits the converted text data and the original voice data to the server.

[1377] Step 11:

[1378] The server analyzes the received voice and text data. Specifically, it uses natural language processing technology to analyze the content of the responses, and voice analysis technology to detect voice intonation, pauses, and slip-ups.

[1379] Step 12:

[1380] The server uses an emotion engine in addition to analysis to analyze the user's emotional state (e.g., confidence, anxiety, joy, sadness). It understands emotions from voice data and reads emotions from text data.

[1381] Step 13:

[1382] The server evaluates and scores the user's answers based on the analysis results, taking into account their emotional state.

[1383] Step 14:

[1384] The server generates personalized feedback based on the scoring and sentiment analysis results. The feedback includes points of merit and improvement for the user's answer, specific suggestions for improvement, suggestions for alternative ways of expressing oneself, and even advice based on emotions (e.g., "I liked that you spoke with confidence" or "You seem a little nervous, so try to relax").

[1385] Step 15:

[1386] The server transmits the generated feedback to the terminal.

[1387] Step 16:

[1388] The terminal displays the received feedback to the user.

[1389] Step 17:

[1390] The user can review the feedback provided and use it to improve themselves, retrying the question or moving on to a new question if necessary.

[1391] With this specific processing flow, the system of the present invention not only helps users hone their skills by gaining experience in interviews and learn how to present themselves to the right companies, but also enables more advanced self-improvement by providing feedback based on their emotional state.

[1392] Example 2

[1393] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1394] In traditional job hunting and career change processes, interview practice and feedback are extremely important, but are difficult to provide individually. Providing appropriate feedback that takes into account the user's emotional state is particularly important, but manual evaluation has its limitations. Furthermore, while the intonation and pauses in a user's voice responses are important, these analyses have not yet been automated. Therefore, there is a need for a system that can provide individualized and precise feedback.

[1395] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input data from an information recording medium, a means for analyzing the input data and automatically generating a question and answerer and a question, a means for the user to input an answer to the generated question, a means for converting the user's answer into voice and text data, a means for analyzing the converted data and making an evaluation, a means for analyzing the user's emotional state using an emotion analysis device and reflecting the analysis in the evaluation, a means for generating individual feedback based on the evaluation result, and a means for presenting the generated feedback to the user. This makes it possible to provide precise feedback that takes into account the user's emotional state and voice characteristics.

[1396] "Information recording medium" refers to a medium that records information about a user, such as an application form or resume, in digital format.

[1397] The "question and answerer" refers to a virtual interviewer that is automatically generated based on the data input by the user, and is a character that asks questions that correspond to a specific industry or company.

[1398] "Means for converting" refers to the technical means for converting the data provided by the user via voice into text data, including voice recognition software.

[1399] "Analysis means" refers to a series of technologies for analyzing input data and converted data, including natural language processing technology and voice analysis technology.

[1400] "Means for evaluation" refers to technical means for evaluating the content of a user's responses and emotional state based on the analyzed data.

[1401] An "emotion analysis device" refers to technology that analyzes a user's emotional state from their voice or text data and reflects the results in their evaluation.

[1402] "Personalized feedback" refers to feedback that includes user-specific improvements and advice that is generated based on the user's specific answers and emotional state.

[1403] The present invention relates to a system that utilizes AI to automatically generate interview questions for job seekers and those seeking a career change, evaluates the answers, and provides individualized feedback. By incorporating an emotion analysis device, the system can recognize the user's emotional state and provide feedback based on that. Specific embodiments of the present invention are described below.

[1404] Entering application forms and resumes

[1405] First, the user uses a terminal to enter information for an application form or resume. The terminal displays a form containing items such as name, address, educational background, work history, skills, and reasons for applying. For example, the input form is provided using a web browser or a dedicated application. When the user enters the required information and presses the send button, the data is sent to the server.

[1406] Receiving and parsing input data

[1407] The server receives the application form and resume data sent from the device. The received data is analyzed using natural language processing technology. Specifically, it uses Python's NLTK library and SpaCy to analyze the text data and identify the user's desired industry and company. Further processing is carried out based on the results of this analysis.

[1408] Interviewer Generation

[1409] Based on the analysis results, the server generates a Q&A person suited to the user's desired company and industry. This interviewer is customized using a generative AI model. For example, a natural language generation model such as ChatGPT is used to create a virtual interviewer with settings suited to interviews at specific companies and industries.

[1410] Question Generation

[1411] The server then generates appropriate questions for the user. By inputting appropriate prompts into the generative AI model, questions that match the user's aspirations are generated. For example, for a user with experience in project management, the server generates the question, "Please tell us specifically about your project management experience."

[1412] Posing the Question

[1413] The server then sends the generated question to the device, which displays it to the user, who answers verbally using a microphone while looking at the screen. The device then converts the speech to text using speech recognition software such as Google Cloud Speech-to-Text.

[1414] Sending response data

[1415] The device then sends the converted text and voice data to a server, which then receives the data and analyzes it using natural language processing and voice analysis technologies.

[1416] Analysis of responses and assessment of emotional state

[1417] The server uses natural language processing technology (e.g., the BERT model) to analyze the content of the user's responses. It also uses speech analysis technology to evaluate voice intonation, pauses, and slip-ups. It also uses EmotionAPI and Affectiva to analyze the user's emotional state. The analysis results include the content of the responses, voice quality, and emotional state.

[1418] Generate feedback

[1419] The server generates personalized feedback based on the analysis results. The feedback includes points for evaluation, areas for improvement, and specific measures for improvement. For example, feedback such as, "Your specific details about your project management experience are good. However, it would be even better if you also included the tools you used and the lessons you learned."

[1420] Providing feedback

[1421] The device receives the feedback sent from the server and displays it to the user, who can then view the feedback on the device screen and get clues to improve their answers.

[1422] Examples of specific examples and prompts

[1423] As a concrete example, consider the case where User A is applying to an IT company and has experience in project management. User A details his / her "project management experience" on an application form and sends it to the server. The server analyzes this data and generates an appropriate question and answerer along with the question "Tell us about your project management experience." When User A answers, the server analyzes the content and generates feedback such as "The details are specific, but it would be even better if you also added the tools you used and the lessons you learned," and sends it to the terminal.

[1424] An example of a prompt sentence to input to the generative AI model is as follows:

[1425] Plan: Generate interview questions and feedback for a user with project management experience who is applying to an IT company. Generate questions based on the information in the application form, analyze the user's answers using natural language processing and an emotion engine, and provide feedback such as the following:

[1426] Application form: As for project management experience, please state that you have led multiple projects in the past and achieved results.

[1427] Q: Tell us about your project management experience.

[1428] User Answer: "I led multiple projects and achieved results."

[1429] Feedback: "You're good at describing your project management experience in detail, but you could expand on specific accomplishments, tools you used, and lessons learned. You also speak with confidence, but you seem a little nervous. I think you could practice speaking in a more relaxed manner."

[1430] As described above, the system of the present invention helps users improve their skills by gaining experience in interviews and learn how to present themselves appropriately to companies, and also enables more advanced self-improvement by providing feedback based on their emotional state.

[1431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1432] Step 1:

[1433] The user inputs data for the information recording medium.

[1434] Specifically, the user uses a terminal to fill out a form containing information such as name, address, educational background, work history, skills, and motivation for applying. After entering the information, the user presses the send button, which sends the entered data from the terminal to the server.

[1435] Input: Application form and resume data entered by the user into the terminal

[1436] Output: User entry data sent to the server

[1437] Step 2:

[1438] The server receives and parses the input data.

[1439] Specifically, the server receives application forms and resumes sent from the device. The received data is analyzed using natural language processing technologies such as Python's NLTK library and SpaCy. This allows the user's desired company and industry to be identified.

[1440] Input: Application form and resume data sent from the device

[1441] Output: Information on the company and industry the user is interested in as a result of the analysis

[1442] Step 3:

[1443] The server generates the interviewers.

[1444] Specifically, the server uses a generative AI model based on the analysis results to generate a question and answerer suited to the company and industry the user is applying for. This question and answerer is designed as a virtual interviewer with extensive knowledge of the specific industry or company.

[1445] Input: Information on the company and industry you are applying to as a result of the analysis

[1446] Output: Virtual interviewer as question and answerer

[1447] Step 4:

[1448] The server generates the question.

[1449] Specifically, the server inputs a prompt into the generative AI model and generates a question that matches the user's desired profile. For example, a question like, "Please tell us specifically about your project management experience" might be generated.

[1450] Input: Prompt statement and user parsed data

[1451] Output: Generated questions

[1452] Step 5:

[1453] The terminal presents the question to the user.

[1454] Specifically, the device displays the questions sent from the server to the user, who then answers verbally using a microphone while looking at the device screen.

[1455] Input: The question sent by the server

[1456] Output: The terminal screen showing the question

[1457] Step 6:

[1458] The user responds verbally, and the device converts the speech into text.

[1459] The user answers questions by speaking into a microphone, and the device uses voice recognition software such as Google Cloud Speech-to-Text to convert the answers into text.

[1460] Input: User's spoken response

[1461] Output: User answers converted to text data

[1462] Step 7:

[1463] The terminal transmits the response data to the server.

[1464] Specifically, the terminal converts the user's answers into text data and transmits them to the server, which then receives the data.

[1465] Input: User answers converted to text data

[1466] Output: Response data sent to the server

[1467] Step 8:

[1468] The server analyzes the responses and assesses the emotional state.

[1469] Specifically, the server analyzes the user's responses using natural language processing technology (such as the BERT model), evaluates the intonation, pauses, and slip-ups of the user's voice using speech analysis technology, and analyzes the user's emotional state using EmotionAPI and Affectiva.

[1470] Input: Response data sent to the server

[1471] Output: Answer content, voice characteristics, emotional state as analysis results

[1472] Step 9:

[1473] The server generates personalized feedback.

[1474] Specifically, the server generates individual feedback based on the analysis results, including evaluation points for the answer, areas for improvement, and specific improvement measures.

[1475] Input: Answer content as analysis result, voice characteristics, emotional state

[1476] Output: Generated feedback

[1477] Step 10:

[1478] The terminal presents feedback to the user.

[1479] Specifically, the device receives the feedback sent from the server and displays it to the user, who can then improve themselves based on this feedback.

[1480] Input: Feedback sent by the server

[1481] Output: Feedback displayed on the terminal

[1482] Through these steps, the system helps users improve themselves through interview practice, allowing them to receive more detailed feedback, including their emotional state.

[1483] (Application example 2)

[1484] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1485] Improving the technical skills and troubleshooting abilities of robot operators is an essential challenge in modern factories. However, conventional education and training systems struggle to assess actual operating skills and emotional states in real time and provide individualized feedback. This makes it difficult for operators to effectively acquire the necessary skills. Furthermore, conventional systems do not provide feedback that takes into account the user's emotional state, making it difficult to alleviate stress and tension.

[1486] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input raw data or document data, a means for analyzing the input data and automatically generating questions for the interlocutor, a means for the user to input answers to the generated questions, a means for providing interactive training including questions about the operator's operating skills and how to deal with the situation, and a means for analyzing the operator's emotional state using an emotion engine and reflecting the results of the evaluation. This makes it possible to effectively improve the operator's technical skills and provide individual feedback based on their emotional state, such as stress or tension.

[1487] "Raw data" refers to data acquired in real time or as is.

[1488] "Document data" refers to information recorded in the form of a resume, application form, etc.

[1489] The "interlocutor" refers to a virtual interviewer or interviewer that is generated to have a dialogue with the user.

[1490] A "question" refers to a question or interrogation posed by the interlocutor to the user.

[1491] "Operator" refers to the engineers and workers who operate robots and equipment within the factory.

[1492] "Operational skills" refers to the technical ability of an operator to accurately operate a robot or equipment.

[1493] "Solutions" refers to solutions and approaches to the troubles and problems that robots and equipment face.

[1494] "Interactive training" refers to educational and training programs that are conducted interactively with the user.

[1495] An "emotion engine" refers to technology that analyzes a user's emotional state and generates feedback based on the results.

[1496] "Evaluation results" refer to the analysis results of the user's answers and operations, and include scores for skills and emotional state.

[1497] "Feedback" refers to information that evaluates a user's actions or responses and points out areas for improvement.

[1498] "Server" refers to a computer system that performs functions such as data analysis, participant generation, and feedback provision.

[1499] To implement this invention, the following hardware and software are used. The hardware used is a microphone, a smartphone, or a computer. The software used is the SpeechRecognition library (Python), the Transformers library (Python), and an AI model for emotion analysis. Specific system program processing is described below.

[1500] System configuration and processing content

[1501] The server provides an interface for users to input raw data and document data. Users fill out application forms and resumes using their smartphones or computers. The input data is sent to the server, which analyzes it. Based on the analysis results, the server automatically generates an interlocutor (a virtual interviewer) and questions. These questions are related to specific operational skills and coping methods.

[1502] The generated questions are displayed on the user's device. The user answers the questions by voice using a microphone. The device converts the user's voice data into text data and sends the data to the server.

[1503] The server uses natural language processing technology to analyze the received voice and text data. It also uses an emotion engine to analyze the user's emotional state and reflect it in the evaluation. Based on the evaluation results, the server generates personalized feedback. This feedback may include specific improvements to the user's answers, technical advice, or even feedback based on the user's emotional state. The generated feedback is sent to the user's device and presented to the user.

[1504] Specific examples

[1505] For example, in an operator training system, the question "What would you do if your robot suddenly stopped working?" is generated and presented to the user. The user answers, "First, I would check the error code and try to restart it manually. If that doesn't solve the problem, I would contact technical support." If this answer is judged to be a confident answer after natural language processing and sentiment analysis, the system provides feedback such as, "Your answer is appropriate. It would be a good idea to thoroughly check the error code again before contacting technical support."

[1506] Prompt Sentence Examples

[1507] Project name: Factory robot operator training system

[1508] Question: "What would you do if your robot suddenly stopped working?"

[1509] Answer: "First, check the error code and try a manual reboot. If that doesn't work, contact technical support."

[1510] Sentiment analysis: "This shows confidence. It's the right approach for an engineer."

[1511] Feedback: "The answer is good. I would recommend double-checking the error code before contacting technical support."

[1512] In this way, the system of the present invention can effectively improve the operator's technical skills and provide personalized feedback based on emotional states such as stress and tension.

[1513] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1514] Step 1:

[1515] The server receives raw data or document data from user devices as input. Users use their smartphones or computers to enter information for job applications or resumes and send it to the server. The output is a database entry containing the input information.

[1516] Step 2:

[1517] The server analyzes the input data and automatically generates interlocutors and questions. This analysis process uses natural language processing technology to identify the desired company, industry, operational skills, and countermeasures. The input is data from an application form or resume, and the output is the generated questions and a virtual interlocutor.

[1518] Step 3:

[1519] The terminal presents the generated question to the user, who then answers by voice using a microphone. The input is the presented question, and the output is voice data.

[1520] Step 4:

[1521] The device uses the SpeechRecognition library to convert the user's voice data into text data. The input is the user's voice data, and the output is the data converted from that voice into text.

[1522] Step 5:

[1523] The server receives the converted text and voice data and evaluates them using natural language processing technology and a sentiment analysis engine. The natural language processing technology evaluates the accuracy and quality of the response, while the sentiment analysis engine analyzes the voice intonation and emotional state. The input is text and voice data, and the output is the evaluation result.

[1524] Step 6:

[1525] The server generates feedback based on the evaluation results. The feedback includes suggestions for improving the answers, technical advice, and comments based on the user's emotional state. The input is the evaluation results, and the output is individual feedback.

[1526] Step 7:

[1527] The terminal presents the generated feedback to the user, who can then further improve himself based on this feedback. The input is the feedback, and the output is the feedback display for the user.

[1528] This series of processing steps allows the system of the present invention to provide effective training based on the user's technical ability and emotional state.

[1529] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1530] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1531] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1532] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1533] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1534] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1535] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1536] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1537] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1538] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1539] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1540] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1541] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1542] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1543] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1544] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1545] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1546] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1547] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1548] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1549] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1550] The following is further disclosed regarding the above embodiment.

[1551] (Claim 1)

[1552] A means for users to input data for application forms and resumes,

[1553] A means of analyzing the input data and automatically generating interviewers and questions,

[1554] a means for a user to input answers to the generated questions;

[1555] means for converting the user's response into voice and text data;

[1556] means for analyzing and evaluating the converted data;

[1557] a means for generating personalized feedback based on the evaluation results;

[1558] means for presenting the generated feedback to the user;

[1559] A system including:

[1560] (Claim 2)

[1561] 2. The system according to claim 1, wherein the analyzing means is a means for analyzing the content of the user's response by utilizing natural language processing technology.

[1562] (Claim 3)

[1563] 2. The system of claim 1, wherein the means for analyzing the voice data is means for detecting intonation, pauses, and slip-ups in the user's voice.

[1564] "Example 1"

[1565] (Claim 1)

[1566] A means for users to input data for application forms and resumes,

[1567] A means of analyzing the input data and automatically generating interviewers and questions,

[1568] a means for a user to input answers to the generated questions;

[1569] means for converting user voice data into text data;

[1570] means for analyzing and evaluating the converted data;

[1571] a means for generating personalized feedback based on the evaluation results;

[1572] means for presenting the generated feedback to the user;

[1573] A system including:

[1574] (Claim 2)

[1575] 2. The system according to claim 1, wherein the analyzing means is a means for analyzing the content of the user's response by utilizing natural language processing technology.

[1576] (Claim 3)

[1577] 2. The system of claim 1, wherein the means for analyzing the voice data is means for detecting intonation, pauses, and slip-ups in the user's voice.

[1578] "Application Example 1"

[1579] (Claim 1)

[1580] A means for users to input data for application forms and resumes,

[1581] A means of analyzing the input data and automatically generating interviewers and questions,

[1582] means for displaying the generated question on a display of the smart glasses;

[1583] A means for allowing a user to input a response by voice and converting the voice data into text data;

[1584] means for analyzing and evaluating user responses;

[1585] a means for generating personalized feedback based on the evaluation results;

[1586] means for presenting the generated feedback on a display of the smart glasses;

[1587] A system including:

[1588] (Claim 2)

[1589] 2. The system according to claim 1, wherein the analyzing means is a means for analyzing the content of the user's response by utilizing natural language processing technology.

[1590] (Claim 3)

[1591] 2. The system of claim 1, wherein the means for analyzing the voice data is means for detecting intonation, pauses, and slip-ups in the user's voice.

[1592] "Example 2: Combining Emotion Engines"

[1593] (Claim 1)

[1594] a means for a user to input data of an information recording medium;

[1595] A means for analyzing input data and automatically generating question and answer participants and questions;

[1596] a means for a user to input answers to the generated questions;

[1597] means for converting the user's response into voice and text data;

[1598] means for analyzing and evaluating the converted data;

[1599] a means for analyzing the emotional state of a user using an emotion analysis device and reflecting the result in an evaluation;

[1600] a means for generating personalized feedback based on the evaluation results;

[1601] means for presenting the generated feedback to the user;

[1602] A system including:

[1603] (Claim 2)

[1604] 2. The system according to claim 1, wherein the analyzing means is a means for analyzing the content of the user's response by utilizing natural language processing technology.

[1605] (Claim 3)

[1606] 2. The system of claim 1, wherein the speech analysis device is a means for detecting the user's vocal intonation, pauses, and slip-ups.

[1607] "Application example 2 when combining emotion engines"

[1608] (Claim 1)

[1609] a means for a user to input raw or document data;

[1610] A means of analyzing input data and automatically generating interlocutors and questions;

[1611] A means for a user to input an answer to the generated question;

[1612] means for converting the user's response into voice and text data;

[1613] means for analyzing and evaluating the converted data;

[1614] a means for generating personalized feedback based on the evaluation results;

[1615] means for presenting the generated feedback to the user;

[1616] A means of providing interactive training that includes questioning of operator operating skills and procedures;

[1617] A means for analyzing the emotional state of an operator using an emotion engine and reflecting the result of the evaluation;

[1618] A system including:

[1619] (Claim 2)

[1620] 2. The system according to claim 1, wherein the analyzing means is a means for analyzing the content of the user's response by utilizing natural language processing technology.

[1621] (Claim 3)

[1622] 2. The system of claim 1, wherein the means for analyzing the voice data is means for detecting intonation, pauses, and slip-ups in the user's voice. [Explanation of symbols]

[1623] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to input data for application forms and resumes, A means of analyzing the input data and automatically generating interviewers and questions, a means for a user to input answers to the generated questions; means for converting the user's response into voice and text data; means for analyzing and evaluating the converted data; a means for generating personalized feedback based on the evaluation results; means for presenting the generated feedback to the user; A system including:

2. 2. The system according to claim 1, wherein the analyzing means is means for analyzing the content of the user's response by utilizing natural language processing technology.

3. 2. The system according to claim 1, wherein the means for analyzing the voice data is means for detecting intonation, pauses, and slip-ups in the user's voice.

Images