system

Abstract

We provide the system. [Solution] A conversation generation method utilizing artificial intelligence for automated interviews, A means for collecting input information using a sensor device that captures audio and video, A speech recognition means that converts collected audio data into text data, A method for analyzing emotions from text data and facial expression data, A method for evaluating the suitability of applicants and companies based on analyzed emotions and responses, A means of generating and outputting this data as a report, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional interview process, the subjectivity and prejudice of the interviewer may affect the result, and a great deal of time and resources are required for the interviewer's preparation and the setting of the interview venue. Furthermore, for job seekers, there is a problem that tension and stress may prevent their original abilities from being fully demonstrated. The object of this invention is to improve the fairness of interviews, enable efficient use of resources, and provide an environment in which job seekers can demonstrate their abilities in a natural state.

Means for Solving the Problems

[0005] This invention provides an automated interview system utilizing artificial intelligence, thereby eliminating the need for human intervention and completely automating the interview process. Specifically, it uses a conversation generation means to automatically ask questions and a voice and video sensor device to collect input information. Furthermore, the collected voice data is converted into text data by a voice recognition means and analyzed together with facial expression data by an emotion analysis means. Based on the results, the system evaluates the suitability of the candidate and the company and outputs a report, thereby achieving efficient and fair interview evaluations.

[0006] "Artificial intelligence" is a general term for technologies designed to enable computer systems to think, learn, and solve problems like humans.

[0007] A "conversation generation method" is a function or system that uses artificial intelligence to automatically generate dialogue with a user.

[0008] A "sensor device" is a device that detects information from the external environment, such as sound and video, and converts it into digital data.

[0009] "Speech recognition means" refers to a technology or device that analyzes recorded speech data and converts it into corresponding text data.

[0010] "Emotional analysis means" refers to a technique or method that analyzes data obtained from voice tone and facial expressions to estimate the speaker's emotional state.

[0011] A "fitness assessment tool" is an algorithm or system for comparing a candidate's characteristics with the criteria required by a company and quantifying or evaluating the degree of agreement.

[0012] A "report generation method" refers to a function or system that aggregates analysis results and evaluation content and outputs them as a report in a visually accessible format. [Brief explanation of the drawing]

[0013] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0017] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is an automated interview system that utilizes artificial intelligence, and its specific embodiments are described below.

[0035] In this system, the server first automatically sets up the interview session according to the interview schedule. Next, the terminal uses its connected camera and microphone to acquire audio and video data from the user in real time. At the start of the system, the terminal verbally greets the user with a respectful voice message to encourage them to begin the conversation.

[0036] The server uses an internally built-in generative model to automatically generate questions for the user, which the terminal then presents to the user verbally. These questions are customized based on the talent characteristics sought by the company. For example, when evaluating creativity, questions are prepared to elicit the user's original ideas.

[0037] When a user answers a question, the device records the audio and uses speech recognition technology to convert the content into text. Facial expressions are also captured in real time, and this data is sent to a server.

[0038] On the server, sentiment analysis and evaluation are performed based on the received text and facial expression data. The user's emotional state is estimated from changes in voice tone and facial expressions, and the candidate's logical thinking ability and creativity are analyzed. These analysis results are compared and quantified or graded to evaluate the degree of fit with the characteristics required by the company.

[0039] Next, the server performs a suitability assessment and generates a comprehensive report based on the results. This report is provided to the company's evaluators, and the user receives the interview data results and feedback. For example, the assessment report may indicate that the user was in a relaxed emotional state or demonstrated a high level of logical thinking.

[0040] In this way, the present invention can significantly improve the fairness and efficiency of interviews and can provide value in recruitment activities in many industrial fields.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The server checks the interview schedule and automatically prepares the interview session at the appropriate date and time. This includes loading interview templates and allocating the necessary computing resources.

[0044] Step 2:

[0045] Check the status of the camera and microphone connected to the device. Confirm that the connection is working correctly, test the volume and video quality, and adjust the settings to the optimal level for conducting the interview.

[0046] Step 3:

[0047] The server activates an AI-generated model to create an initial greeting message for the user. This message takes into account the naturalness and politeness of the conversation.

[0048] Step 4:

[0049] The device converts the generated greeting message into audio and plays it for the user. This allows the user to recognize the start of the interview.

[0050] Step 5:

[0051] The server randomizes or customizes the question set based on company metrics and selects the first question.

[0052] Step 6:

[0053] The device presents the selected question to the user verbally. The user listens to the question and understands its intent.

[0054] Step 7:

[0055] Users will answer questions verbally. Efforts will be made to ensure consistency in answers, and opportunities for re-answering will be provided as needed.

[0056] Step 8:

[0057] The device records the user's responses and converts them into text data in real time using speech recognition technology. The accuracy of the conversion results is checked, and corrections are made as needed.

[0058] Step 9:

[0059] The device captures the user's face as video data and records changes in facial expressions. This allows for the collection of data for emotion analysis.

[0060] Step 10:

[0061] The server integrates voice data and facial expression data to perform emotion analysis. Based on the analysis, it evaluates the user's emotional state and level of tension.

[0062] Step 11:

[0063] The server uses the sentiment analysis results and responses to perform a suitability assessment. This assessment is based on an algorithm that calculates the degree of match with the talent characteristics sought by the company.

[0064] Step 12:

[0065] The server generates an evaluation report and notifies the company's evaluators. It also provides users with interview results and feedback.

[0066] Step 13:

[0067] The terminal organizes all interview data, notifies the user that the system is ending, and safely terminates the session.

[0068] (Example 1)

[0069] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0070] Traditional HR interview processes suffer from a lack of fairness and efficiency, and consume a significant amount of human resources. Furthermore, the evaluation process is highly subjective, making it difficult to accurately measure the suitability of candidates for the company.

[0071] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0072] In this invention, the server includes a question and answer generation means utilizing an intelligent agent for conducting automated interviews, a means for collecting input information using a sensing device that acquires acoustic and visual data, and a recognition means for converting the collected audio information into textual information. This automates the interview process and enables objective and efficient evaluation of candidates.

[0073] An "automated interview" is an interview process that utilizes artificial intelligence technology to minimize human intervention.

[0074] An "intelligent agent" is a program or system that uses artificial intelligence to autonomously perform a specific task.

[0075] A "question and answer generation tool" is a mechanism for generating questions based on specific conditions or criteria and for eliciting answers to those questions.

[0076] A "sensing device" is hardware that has the function of acquiring external data such as audio and video.

[0077] "Recognition means" refers to technology that analyzes input audio data and converts it into corresponding text data.

[0078] "Emotion inference techniques" are technologies used to infer a user's emotions from their voice tone and facial expressions.

[0079] A "fitness quantification method" is a system that expresses the degree to which a candidate's characteristics match the characteristics sought by a company using numerical values.

[0080] A "report generation method" is a technology that aggregates interview results and evaluation data and outputs them as a report in a standardized format.

[0081] The "automatic schedule setting method" is a function that automatically sets the date, time, and order of interviews based on predetermined conditions.

[0082] "Dynamic evaluation" is a process that collects data in real time and performs evaluations based on information that changes as it occurs.

[0083] "Real-time information gathering" refers to the process of quickly acquiring data from a target in real time.

[0084] This invention is for constructing an automated interview system, which realizes an advanced interview process through the coordination of an intelligent agent, a sensing device, a recognition means, and related technologies.

[0085] First, the server has a mechanism to manage the automated interview schedule, automatically configuring the system based on the scheduled interviews. This allows the server to efficiently prepare for interviews.

[0086] Next, the device uses its connected camera and microphone to acquire the user's voice and video in real time. General-purpose APIs are used for device operation, and data is sent to the server.

[0087] Artificial intelligence (AI) models are used to generate speech synthesis technology for greetings and questions to the user. The server utilizes the generation AI model to create questions for the user and sets appropriate prompts. A specific example of a prompt could be, "Please generate questions to assess imagination."

[0088] When a user responds, the device records the audio and converts it to text using speech recognition technology. Simultaneously, it captures the user's facial expressions, and this data is aggregated on a server.

[0089] The server performs multifaceted analysis based on the aggregated data. It utilizes machine learning models to calculate the degree of fit, including sentiment inference and evaluation of logical thinking ability. The results are ultimately generated as a report and provided to the organization. Furthermore, the results are also communicated to the user as feedback.

[0090] This automated interview system can significantly improve the efficiency and fairness of interviews, demonstrating its value in various industrial sectors.

[0091] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0092] Step 1:

[0093] The server checks the interview schedule. The interview date, time, and candidate information are read from the database as input. Based on this information, the server prepares the session settings and starts the relevant modules. This completes the preparation for the automated interview process.

[0094] Step 2:

[0095] The device activates its camera and microphone to acquire audio and video data from the user. The input is the user's voice and video, which are captured in real time. The collected data is sent to a server via the network as output. The device API is utilized to ensure accurate data acquisition and transmission.

[0096] Step 3:

[0097] The terminal uses speech synthesis technology to greet the user. Text data from the server is used as input, and speech synthesis platform is used to output voice data. This allows the user to recognize the start of the process.

[0098] Step 4:

[0099] The server generates questions using a generation AI model. The input here is a prompt based on the characteristics requested by the company. An example prompt is "Generate questions that stimulate creativity." As a result, the generated questions are output in text format and sent to the terminal.

[0100] Step 5:

[0101] The terminal presents the user with a generated question. The input is text data sent from the server, which is then processed again using speech synthesis and output as audio data. The user then answers this audio question.

[0102] Step 6:

[0103] The device records the user's responses, and the audio data is converted into text data using recognition technology. The user's audio data is used as input, and the converted text is output. In addition, facial expression data is also captured and sent to the server.

[0104] Step 7:

[0105] The server analyzes the collected text and facial expression data. The input consists of speech-recognized text and facial expression data, and data calculations are performed to evaluate emotion inference and logical thinking ability. As a result of the analysis, the degree of fit is quantified and output as data for organizational evaluation.

[0106] Step 8:

[0107] The server aggregates the analysis results and generates a report. The input consists of evaluated fit and analysis information, and a report is output based on this information in a standard format. This report is provided to the organization's evaluators, and feedback is sent to the user.

[0108] (Application Example 1)

[0109] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0110] Traditional interview methods are constrained by time and location, resulting in inefficient and sometimes unobjective interview processes. Furthermore, subjective evaluations by interviewers can be biased, making it difficult to accurately assess the applicant's suitability for the organization's talent requirements. Additionally, immediate feedback after interviews is often lacking, hindering applicants from having opportunities for self-improvement.

[0111] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0112] In this invention, the server includes a question-and-answer means based on conversation generation, a means for acquiring information using an input device that records audio and video, and a recognition means for converting audio information into text information. This makes it possible to dynamically and objectively evaluate the compatibility between applicants and organizations.

[0113] "Conversation generation" is the process of automatically conducting question-and-answer sessions using artificial intelligence.

[0114] A "question and answer system" is a function that allows the system to automatically generate questions and receive answers from users.

[0115] An "input device" is a device used to acquire data such as audio and video, and includes sensors, microphones, and cameras.

[0116] "Means of acquiring information" refers to the process of collecting audio and video data through digital devices.

[0117] "Recognition means" refers to the process of converting acquired audio into text, and utilizes speech recognition technology.

[0118] "Facial expression information" refers to the digital data recording of the user's facial movements.

[0119] "Means for analyzing emotions" refers to technologies that estimate a user's emotional state from facial expression information and voice data.

[0120] "Methods for evaluating consistency" refer to the process of comparing the characteristics of applicants with the requirements of the organization and quantifying the degree of agreement between them.

[0121] "Smart devices" refer to mobile terminals with advanced information processing capabilities, such as mobile phones, tablets, and wearable devices.

[0122] A "report with feedback" is a document that includes an evaluation based on the interview results, intended to communicate areas for improvement to the user.

[0123] "Methods for dynamically acquiring data" refer to functions that continuously collect information that changes in real time.

[0124] In the system that implements this application, a server plays a primary role. The server generates questions to present to the user using a generative AI model and sends them to the terminal. The terminal, as part of a smart device, uses a microphone and camera to collect audio and video data from the user. Using speech recognition technology, the terminal converts the audio into text data in real time.

[0125] The server analyzes emotions based on acquired text data and facial expression information. This analysis uses proprietary emotion analysis software to estimate the user's emotional state. Next, the analyzed data is used to evaluate the suitability of the applicant to the organization, and the results are quantified or graded. This evaluation result is generated as a report with feedback and sent to both the user and the organization.

[0126] As a specific example of the system, we assume that the terminal is a smartphone. In this case, the smartphone's camera and microphone are utilized, and the user participates in the interview process via the internet. The user can flexibly respond to the questions received through the terminal from anywhere, such as their home or workplace.

[0127] An example of a prompt would be, "Generate in-depth questions about the user's project experience, and then perform a sentiment assessment based on the responses." By using this prompt, the generating AI model can provide high-quality questions tailored to the purpose, creating a foundation for thoroughly evaluating the applicant's characteristics.

[0128] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0129] Step 1:

[0130] The server uses a generative AI model to generate questions based on a prompt. This prompt might be something like, "Generate in-depth questions about the user's project experience." The input is the prompt, and the output is the questions presented to the user. Natural language generation is performed within the server by the AI ​​model, resulting in customized questions.

[0131] Step 2:

[0132] The server sends the generated question to the terminal. The terminal presents this to the user as an audio message. The input here is the question text from the server, and the output is the audio the user hears. The terminal's text-to-speech function converts the text into speech, which is then read aloud to the user.

[0133] Step 3:

[0134] The user answers the presented questions via voice through the device. The input is the user's voice response, and the output is the voice data recorded directly on the device. The user provides voice input, and the device collects the data.

[0135] Step 4:

[0136] The terminal converts the user's voice into text data. It uses speech recognition software to convert the input voice data into text information. The output is sent to the server as text data.

[0137] Step 5:

[0138] The device uses a camera to acquire user facial expression information in real time. The input is live video data, and the output generates still images and dynamic facial expression data necessary for facial expression analysis. The camera captures changes in the user's facial expressions, and this information is transferred to the server as processed data.

[0139] Step 6:

[0140] The server analyzes emotions and evaluates suitability based on text data and facial expression information. The input is text data and facial expression data, and the output is an evaluation result indicating the compatibility between the applicant and the organization. The server uses algorithms to perform data calculations and generate an integrated evaluation score.

[0141] Step 7:

[0142] The server generates a report with feedback based on the evaluation results and sends it to the user and the organization. The input is the evaluation results, and the output is the completed report. The server supports the user's next steps by formatting the data and generating an easy-to-read report.

[0143] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0144] This invention is an automated interview system using artificial intelligence and an emotion engine, and its specific embodiments are described below.

[0145] In this system, the server first automatically sets up the interview session according to the interview schedule. At the designated time, the terminal starts capturing the user's audio and video in real time through the connected camera and microphone.

[0146] Once the interview begins, the server uses a conversation generation model to automatically generate an initial greeting and an explanation of the purpose for the user. The terminal then communicates this to the user verbally, allowing the user to recognize the start of the interview.

[0147] Next, the server generates a set of questions, selects a customized initial question, and presents it to the user through the terminal. The user's responses are recorded on the terminal and converted to text using speech recognition technology. In parallel with this audio data, the terminal also captures the user's facial expressions with high accuracy to collect data necessary for sentiment analysis.

[0148] A key feature of this system is its emotion engine, which analyzes voice tone, facial expressions, and even the user's response time to assess the user's emotional state in real time. For example, if tension is detected, the server adjusts the content and tone of the questions asked by the system to the user to encourage a more relaxed response.

[0149] The server integrates this sentiment data with user responses and calculates the degree of fit with the characteristics the company is looking for. The evaluation results are generated as a report and provided to the company, and the user is notified with feedback.

[0150] For example, if a user temporarily stops responding due to nervousness, the system will detect this and encourage relaxation by inserting easy-to-use random jokes or light conversation, then promptly ask questions again to allow the user to perform at their best.

[0151] This embodiment provides a fairer and less stressful environment compared to conventional interviews, while also enabling more accurate talent evaluation.

[0152] The following describes the processing flow.

[0153] Step 1:

[0154] The server checks the interview schedule and prepares to set up the interview session at the specified date and time. It loads the interview template and required question list and verifies that it is ready.

[0155] Step 2:

[0156] The device tests the connection status of the camera and microphone and checks their performance. It checks the clarity of the audio and the quality of the video and adjusts them to the optimal state.

[0157] Step 3:

[0158] The server uses AI to generate an initial message that greets the user and explains the interview process and purpose. This message is designed to help the user feel at ease during the interview.

[0159] Step 4:

[0160] The device plays an initial message as audio to the user, informing them that the interview is about to begin. This helps the user prepare themselves mentally.

[0161] Step 5:

[0162] The server selects the first question and generates questions based on key themes in the interview process. These questions are aligned with the characteristics of the candidate the company is seeking.

[0163] Step 6:

[0164] The device presents the user with selected questions via voice, and the user responds accordingly. A feature that detects the user's level of anxiety is also utilized during the response process.

[0165] Step 7:

[0166] The device records the user's response and begins the process of converting the audio into text data in real time. At the same time, the user's facial expressions are also captured by the camera.

[0167] Step 8:

[0168] The server receives voice text and facial expression data, and uses an emotion engine to analyze the user's emotional state. This allows it to determine the user's level of tension or excitement.

[0169] Step 9:

[0170] Based on the analysis results, the server dynamically changes the difficulty and tone of questions if necessary to adjust the flow of the conversation. For example, it might add friendly topics to ease tension.

[0171] Step 10:

[0172] The server integrates all the data and evaluates the degree of relevance based on the user's responses and emotional state. This determines how well the user matches the company's needs.

[0173] Step 11:

[0174] The server generates evaluation results in report format and sends them to the company's system for the evaluators. At the same time, it sends the interview evaluation results and feedback to the user.

[0175] Step 12:

[0176] The terminal confirms that all processing is complete, saves the interview data, and then safely shuts down the system. The user is notified that the interview has ended, and it is clearly indicated that the interaction was completed safely.

[0177] (Example 2)

[0178] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0179] Traditional interview systems have suffered from significant influence from the interviewer's subjectivity and the interview environment, making fair and consistent evaluation difficult. Furthermore, it was challenging to appropriately analyze and respond to applicants' emotions and reactions during the interview. Therefore, accurately assessing applicants' actual abilities and aptitudes proved difficult.

[0180] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0181] In this invention, the server includes a dialogue generation means utilizing artificial intelligence for conducting automated interviews, a means for collecting data using a sensor device for acquiring audio and video, and a speech recognition means for converting the collected audio information into text information. This enables the analysis of applicants' emotions and responses during interviews and allows for appropriate responses in real time. This makes it possible to achieve fair and consistent interview evaluations and provide valuable information for both companies and applicants.

[0182] "Automated interviews" are an interview method that uses artificial intelligence to communicate with job seekers without the need for human interviewers.

[0183] "Dialogue generation means" refers to a technology that uses artificial intelligence to automatically generate questions and responses for effective communication with job seekers.

[0184] A "sensor device" is a hardware device used to acquire environmental information such as sound and video.

[0185] "Speech recognition means" refers to processing technology for converting acquired speech information into text information and digitizing it.

[0186] "Facial expression information" refers to digital data acquired from the facial movements and characteristics of job applicants, and is used for emotional analysis.

[0187] "Emotion" refers to a person's psychological state and is a part of the feelings that can be analyzed from tone of voice and facial expressions.

[0188] "Suitability" is a measure used to evaluate how well a job seeker matches the characteristics and abilities that a company is looking for.

[0189] A "report" is a document that organizes and summarizes the interview results and analysis, and is provided to the company.

[0190] A "generative AI model" is an artificial intelligence technology that generates intelligent responses or content based on input data.

[0191] "Control means" refers to functions and technologies used to adjust the operation and response of a system.

[0192] This invention relates to an automated interview system that uses artificial intelligence to manage the entire interview process and enable more objective and effective evaluation. The system mainly consists of three elements: a server, a terminal, and a user.

[0193] First, the server handles the system's primary processing. The server generates interview dialogues using a generative AI model. These dialogues are dynamically updated based on user responses. This generative AI model incorporates various natural language processing techniques, specifically using a large-scale language model. This enables appropriate follow-up questions and comments in response to user answers.

[0194] Next, the terminal is responsible for direct interaction with the user. The terminal incorporates audio and video sensors to capture the user's voice and facial expressions with high precision. For this, high-performance cameras and microphones are used as hardware, and reliable speech recognition software is employed on the software side. The captured audio is converted into text data using the speech recognition system.

[0195] Furthermore, the user sits in front of the terminal and participates in the interview scenario. While the user answers the system's dialogue and questions, their facial expressions and tone of voice are captured by sensors. The user's emotions are analyzed in real time, and if tension or anxiety is detected, the server uses a generated AI model to create prompts and adjust the tone and content of the questions. For example, a question prompt such as, "Could you tell me a little more about your experience?" is used.

[0196] As described above, a system applying the present invention provides a fairer and less stressful interview environment compared to conventional interviews, and allows for highly accurate evaluation of the suitability between companies and job seekers.

[0197] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0198] Step 1:

[0199] The server checks the interview schedule and identifies the next interview time and date. Here, a database query is used to retrieve the schedule information and automatically set up the session at the interview start time. The input is the schedule data, and the output is the setting information for the new interview session.

[0200] Step 2:

[0201] The device activates its camera and microphone at the designated interview time, capturing the user's voice and video in real time. High-performance sensors acquire the audio and video data, which is then recorded in a file. Input is the user's speech and facial expressions, while output is audio and video data.

[0202] Step 3:

[0203] When the interview begins, the server uses a generated AI model to create an initial greeting and the purpose of the interview. The greeting, output as text, is converted into speech using speech synthesis software, and the terminal transmits this to the user. The prompt is generated in the format of "The interview will now begin. Please introduce yourself." The input is a prompt to the AI ​​model, and the output is the synthesized greeting.

[0204] Step 4:

[0205] The server selects appropriate questions from a set of questions and customizes them using a generative AI model. The resulting question text is presented to the user via the terminal and output as speech synthesis. The input consists of question data and prompts, and the output is the customized question in audio format.

[0206] Step 5:

[0207] When a user answers a question, the device records the answer as audio. This audio is then converted to text using speech recognition technology and sent to the server. The input is the user's audio answer, and the output is the transcribed answer.

[0208] Step 6:

[0209] The server uses facial recognition technology to analyze the user's facial expression data and evaluate their emotional state. Emotions are analyzed in real time, along with voice tone and response time. Inputs are voice tone and facial expression data, and output is the emotional evaluation result.

[0210] Step 7:

[0211] Based on the results of the emotional assessment, the server uses a generative AI model to adjust the tone of questions and dialogues. For example, it might generate phrases to help the user relax and incorporate them into the next dialogue. The input is the result of the emotional assessment, and the output is the adjusted dialogue.

[0212] Step 8:

[0213] The server integrates user responses and emotional data and runs an algorithm to evaluate suitability. The evaluation results are compiled into a report and provided to companies and users. The input is the user responses and emotional data, and the output is the report.

[0214] (Application Example 2)

[0215] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0216] In automated interview systems, implementing an authentication process that considers the user's emotional state is crucial for reducing the user's psychological burden and providing a less stressful environment. Furthermore, given the current need for fairer and more accurate evaluations through the appropriate use of emotional assessment, the introduction of new payment systems is required. However, existing technologies have the challenge of not being able to adequately consider the user's emotional state and thus not contributing to the smooth operation of the authentication process.

[0217] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0218] In this invention, the server includes a conversation generation means that utilizes intelligence for conducting automated interviews, a means for collecting input information using sensor devices that collect audio and video, and an optimization means that utilizes a relaxing effect in payment authentication based on the user's emotional state. This makes it possible to provide a payment authentication process with a relaxing effect that takes the user's emotional state into consideration.

[0219] "Automated interviews" are a process that utilizes artificial intelligence to automatically ask users questions and analyze their responses.

[0220] "Intelligence" refers to the ability to make judgments and take actions appropriate to a situation based on information processed by a computer.

[0221] A "conversation generation method" is a function in which the system spontaneously creates and presents questions to the user.

[0222] "Audio and video" refers to auditory and visual information obtained from the user, which is captured through sensor devices.

[0223] "Sensor equipment" refers to devices and equipment used to acquire audio and video information.

[0224] "Input information" refers to data collected for system processing, including audio, video, and other related data.

[0225] "Speech recognition means" refers to the technology or process that analyzes acquired speech data and converts it into text information.

[0226] "Character data" refers to text information converted from speech by speech recognition technology.

[0227] "Facial expression data" refers to information obtained by measuring a user's facial expressions and converting their characteristics into digital data.

[0228] "Emotion" refers to the user's instantaneous feelings and psychological state.

[0229] "Means of analyzing emotions" refers to methods and techniques for identifying and evaluating a user's emotional state.

[0230] "Persons being evaluated" refers to individuals whose fitness and other characteristics are assessed by the system.

[0231] "Organization" refers to any group or company that seeks to verify the suitability of the person being evaluated.

[0232] "Fit" is a measure that indicates the degree to which the person being evaluated matches the characteristics required by the organization.

[0233] "Evaluation means" refers to devices or systems used to evaluate the characteristics of a subject based on collected data and to determine their suitability.

[0234] "Optimization methods" refer to ways of adjusting conditions and parameters to improve the efficiency of a process or system.

[0235] "Relaxation effect" refers to the psychological or physical influence that alleviates tension in the subject.

[0236] "Information" is a general term for data that a system or process uses for processing and analysis.

[0237] A "report" refers to a written or electronic format in which evaluation results and analysis content are organized.

[0238] "Output" refers to the act or format in which a system displays or provides the results of its processing.

[0239] The system realizing this invention consists of a server, a terminal, and a user. The server requires an internet connection and functions as a platform for running programs that include an intelligence and emotion engine. The terminal functions as a sensor device that uses a camera and microphone to collect the user's voice and video. The user faces the terminal and participates in an automated interview and authentication process using voice and video.

[0240] The server uses intelligence to execute conversation generation means and generate questions to present to the user. A generative AI model is used in this process. The terminal collects audio and video in real time and sends this data to the server. The server uses speech recognition means to convert the audio data into text and further evaluates the user's emotional state using emotion analysis means. This analysis utilizes facial expression data and voice tone, and uses software such as TENSORFLOW® and OpenCV.

[0241] Based on this data, the server applies optimization measures to provide users with a relaxing experience, such as playing light background music to alleviate their tension. Finally, the server integrates this information, generates a fitness report, and outputs it to the organization.

[0242] As a concrete example, in the authentication process for electronic payments, if a user is feeling nervous in front of the payment terminal, the server will assess the situation based on the results of emotional analysis and generate and present a prompt to the user that reduces their psychological burden through light conversation such as, "The weather is lovely today, it's a perfect day for shopping."

[0243] Example of a prompt:

[0244] "Explain the mechanism for user authentication using facial recognition and voice analysis. Explain, using examples, how the user is provided with a relaxed state by the system, including specific processing steps."

[0245] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0246] Step 1:

[0247] The device activates its camera and microphone to acquire user audio and video data. This data is temporarily stored on the device and prepared for transmission to the server. The input is the user's video and audio, and the output is raw data. Specifically, the user speaks into the device, the camera captures the video, and the microphone records the audio.

[0248] Step 2:

[0249] The terminal transmits the acquired audio and video data to the server. The terminal compresses the data to reduce the transfer time. The input is the compressed audio and video data, and the output is the data received by the server. Specifically, the terminal uses its communication module to send the data to the server via the internet.

[0250] Step 3:

[0251] The server converts the received audio data into text data using speech recognition technology. This process utilizes the Google® Cloud Speech-to-Text API. The input is audio data, and the output is text data. Specifically, the server sends audio data to the API and saves the returned text to an internal database.

[0252] Step 4:

[0253] The server extracts facial expression data from video data and analyzes emotions. It uses TensorFlow to perform facial expression analysis and estimate the user's emotional state. The input is video data, and the output is emotion evaluation data. Specifically, it detects facial feature points from the video and evaluates emotions using a pre-trained model.

[0254] Step 5:

[0255] The server generates optimal dialogue to induce relaxation based on the user's emotional state. This utilizes a generative AI model. The input is emotional assessment data, and the output is a prompt designed to promote relaxation. Specifically, the generative AI creates the prompt and prepares it to be sent to the user's device.

[0256] Step 6:

[0257] The server sends a generated prompt to the terminal, which then presents it to the user via voice or on-screen display. The input is the prompt, and the output is the information presented to the user. Specifically, the terminal either plays the text using speech synthesis or displays the text on the screen.

[0258] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0259] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0260] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0261] [Second Embodiment]

[0262] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0263] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0264] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0265] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0266] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0267] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0268] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0269] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0270] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0271] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0272] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0273] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0274] This invention is an automated interview system that utilizes artificial intelligence, and its specific embodiments are described below.

[0275] In this system, the server first automatically sets up the interview session according to the interview schedule. Next, the terminal uses its connected camera and microphone to acquire audio and video data from the user in real time. At the start of the system, the terminal verbally greets the user with a respectful voice message to encourage them to begin the conversation.

[0276] The server uses an internally built-in generative model to automatically generate questions for the user, which the terminal then presents to the user verbally. These questions are customized based on the talent characteristics sought by the company. For example, when evaluating creativity, questions are prepared to elicit the user's original ideas.

[0277] When a user answers a question, the device records the audio and uses speech recognition technology to convert the content into text. Facial expressions are also captured in real time, and this data is sent to a server.

[0278] On the server, sentiment analysis and evaluation are performed based on the received text and facial expression data. The user's emotional state is estimated from changes in voice tone and facial expressions, and the candidate's logical thinking ability and creativity are analyzed. These analysis results are compared and quantified or graded to evaluate the degree of fit with the characteristics required by the company.

[0279] Next, the server performs a suitability assessment and generates a comprehensive report based on the results. This report is provided to the company's evaluators, and the user receives the interview data results and feedback. For example, the assessment report may indicate that the user was in a relaxed emotional state or demonstrated a high level of logical thinking.

[0280] In this way, the present invention can significantly improve the fairness and efficiency of interviews and can provide value in recruitment activities in many industrial fields.

[0281] The following describes the processing flow.

[0282] Step 1:

[0283] The server checks the interview schedule and automatically prepares the interview session at the appropriate date and time. This includes loading interview templates and allocating the necessary computing resources.

[0284] Step 2:

[0285] Check the status of the camera and microphone connected to the terminal. Ensure the connection is normal, test the volume and video quality, and adjust to the optimal settings for the interview.

[0286] Step 3:

[0287] The server starts the AI generation model and generates the first greeting message for the user. This message takes into account the naturalness and propriety of the conversation.

[0288] Step 4:

[0289] The terminal converts the generated greeting message into voice and plays it for the user. This enables the user to recognize the start of the interview.

[0290] Step 5:

[0291] The server randomizes the question set or customizes it based on the company's metrics and selects the first question.

[0292] Step 6:

[0293] The terminal presents the selected question to the user in voice. The user listens to the question and understands its intention.

[0294] Step 7:

[0295] The user answers the question in voice. Try to ensure the answer is consistent and provide an opportunity for re-answer if necessary.

[0296] Step 8:

[0297] The terminal records the user's answer and uses speech recognition technology to convert it into text data in real time. Check the accuracy of the conversion result and make corrections if necessary.

[0298] Step 9:

[0299] The terminal captures the user's face as video data and records changes in expression, thereby collecting data for emotion analysis.

[0300] Step 10:

[0301] The server integrates the voice data and expression data and performs emotion analysis. As a result of the analysis, the emotion state and tension level of the user are evaluated.

[0302] Step 11:

[0303] The server uses the emotion analysis result and the response content to perform a fitness evaluation. This evaluation is performed based on an algorithm that calculates the degree of match with the personnel characteristics required by the company.

[0304] Step 12:

[0305] The server generates the evaluation result as a report and notifies the company's evaluation staff. Also, the interview result and feedback are provided to the user.

[0306] Step 13:

[0307] The terminal organizes all the interview data, notifies the user of the end of the system, and safely ends the session.

[0308] (Example 1)

[0309] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0310] In the conventional personnel interview process, there are problems such as lack of fairness and efficiency, and consumption of a large amount of human resources. Also, the subjectivity in the evaluation process is strong, and it is difficult to accurately measure the compatibility between candidates and the company.

[0311] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0312] In this invention, the server includes a question and answer generation means utilizing an intelligent agent for conducting automated interviews, a means for collecting input information using a sensing device that acquires acoustic and visual data, and a recognition means for converting the collected audio information into textual information. This automates the interview process and enables objective and efficient evaluation of candidates.

[0313] An "automated interview" is an interview process that utilizes artificial intelligence technology to minimize human intervention.

[0314] An "intelligent agent" is a program or system that uses artificial intelligence to autonomously perform a specific task.

[0315] A "question and answer generation tool" is a mechanism for generating questions based on specific conditions or criteria and for eliciting answers to those questions.

[0316] A "sensing device" is hardware that has the function of acquiring external data such as audio and video.

[0317] "Recognition means" refers to technology that analyzes input audio data and converts it into corresponding text data.

[0318] "Emotion inference techniques" are technologies used to infer a user's emotions from their voice tone and facial expressions.

[0319] A "fitness quantification method" is a system that expresses the degree to which a candidate's characteristics match the characteristics sought by a company using numerical values.

[0320] A "report generation method" is a technology that aggregates interview results and evaluation data and outputs them as a report in a standardized format.

[0321] The "automatic schedule setting method" is a function that automatically sets the date, time, and order of interviews based on predetermined conditions.

[0322] "Dynamic evaluation" is a process that collects data in real time and performs evaluations based on information that changes as it occurs.

[0323] "Real-time information gathering" refers to the process of quickly acquiring data from a target in real time.

[0324] This invention is for constructing an automated interview system, which realizes an advanced interview process through the coordination of an intelligent agent, a sensing device, a recognition means, and related technologies.

[0325] First, the server has a mechanism to manage the automated interview schedule, automatically configuring the system based on the scheduled interviews. This allows the server to efficiently prepare for interviews.

[0326] Next, the device uses its connected camera and microphone to acquire the user's voice and video in real time. General-purpose APIs are used for device operation, and data is sent to the server.

[0327] Artificial intelligence (AI) models are used to generate speech synthesis technology for greetings and questions to the user. The server utilizes the generation AI model to create questions for the user and sets appropriate prompts. A specific example of a prompt could be, "Please generate questions to assess imagination."

[0328] When a user responds, the device records the audio and converts it to text using speech recognition technology. Simultaneously, it captures the user's facial expressions, and this data is aggregated on a server.

[0329] The server performs multifaceted analysis based on the aggregated data. It utilizes machine learning models to calculate the degree of fit, including sentiment inference and evaluation of logical thinking ability. The results are ultimately generated as a report and provided to the organization. Furthermore, the results are also communicated to the user as feedback.

[0330] This automated interview system can significantly improve the efficiency and fairness of interviews, demonstrating its value in various industrial sectors.

[0331] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0332] Step 1:

[0333] The server checks the interview schedule. The interview date, time, and candidate information are read from the database as input. Based on this information, the server prepares the session settings and starts the relevant modules. This completes the preparation for the automated interview process.

[0334] Step 2:

[0335] The device activates its camera and microphone to acquire audio and video data from the user. The input is the user's voice and video, which are captured in real time. The collected data is sent to a server via the network as output. The device API is utilized to ensure accurate data acquisition and transmission.

[0336] Step 3:

[0337] The terminal uses speech synthesis technology to greet the user. Text data from the server is used as input, and speech synthesis platform is used to output voice data. This allows the user to recognize the start of the process.

[0338] Step 4:

[0339] The server generates questions using a generation AI model. The input here is a prompt based on the characteristics requested by the company. An example prompt is "Generate questions that stimulate creativity." As a result, the generated questions are output in text format and sent to the terminal.

[0340] Step 5:

[0341] The terminal presents the user with a generated question. The input is text data sent from the server, which is then processed again using speech synthesis and output as audio data. The user then answers this audio question.

[0342] Step 6:

[0343] The device records the user's responses, and the audio data is converted into text data using recognition technology. The user's audio data is used as input, and the converted text is output. In addition, facial expression data is also captured and sent to the server.

[0344] Step 7:

[0345] The server analyzes the collected text and facial expression data. The input consists of speech-recognized text and facial expression data, and data calculations are performed to evaluate emotion inference and logical thinking ability. As a result of the analysis, the degree of fit is quantified and output as data for organizational evaluation.

[0346] Step 8:

[0347] The server aggregates the analysis results and generates a report. The input consists of evaluated fit and analysis information, and a report is output based on this information in a standard format. This report is provided to the organization's evaluators, and feedback is sent to the user.

[0348] (Application Example 1)

[0349] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0350] Traditional interview methods are constrained by time and location, resulting in inefficient and sometimes unobjective interview processes. Furthermore, subjective evaluations by interviewers can be biased, making it difficult to accurately assess the applicant's suitability for the organization's talent requirements. Additionally, immediate feedback after interviews is often lacking, hindering applicants from having opportunities for self-improvement.

[0351] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0352] In this invention, the server includes a question-and-answer means based on conversation generation, a means for acquiring information using an input device that records audio and video, and a recognition means for converting audio information into text information. This makes it possible to dynamically and objectively evaluate the compatibility between applicants and organizations.

[0353] "Conversation generation" is the process of automatically conducting question-and-answer sessions using artificial intelligence.

[0354] A "question and answer system" is a function that allows the system to automatically generate questions and receive answers from users.

[0355] An "input device" is a device used to acquire data such as audio and video, and includes sensors, microphones, and cameras.

[0356] "Means of acquiring information" refers to the process of collecting audio and video data through digital devices.

[0357] "Recognition means" refers to the process of converting acquired audio into text, and utilizes speech recognition technology.

[0358] "Facial expression information" refers to the digital data recording of the user's facial movements.

[0359] "Means for analyzing emotions" refers to technologies that estimate a user's emotional state from facial expression information and voice data.

[0360] "Methods for evaluating consistency" refer to the process of comparing the characteristics of applicants with the requirements of the organization and quantifying the degree of agreement between them.

[0361] "Smart devices" refer to mobile terminals with advanced information processing capabilities, such as mobile phones, tablets, and wearable devices.

[0362] A "report with feedback" is a document that includes an evaluation based on the interview results, intended to communicate areas for improvement to the user.

[0363] "Methods for dynamically acquiring data" refer to functions that continuously collect information that changes in real time.

[0364] In the system that implements this application, a server plays a primary role. The server generates questions to present to the user using a generative AI model and sends them to the terminal. The terminal, as part of a smart device, uses a microphone and camera to collect audio and video data from the user. Using speech recognition technology, the terminal converts the audio into text data in real time.

[0365] The server analyzes emotions based on acquired text data and facial expression information. This analysis uses proprietary emotion analysis software to estimate the user's emotional state. Next, the analyzed data is used to evaluate the suitability of the applicant to the organization, and the results are quantified or graded. This evaluation result is generated as a report with feedback and sent to both the user and the organization.

[0366] As a specific example of the system, we assume that the terminal is a smartphone. In this case, the smartphone's camera and microphone are utilized, and the user participates in the interview process via the internet. The user can flexibly respond to the questions received through the terminal from anywhere, such as their home or workplace.

[0367] An example of a prompt would be, "Generate in-depth questions about the user's project experience, and then perform a sentiment assessment based on the responses." By using this prompt, the generating AI model can provide high-quality questions tailored to the purpose, creating a foundation for thoroughly evaluating the applicant's characteristics.

[0368] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0369] Step 1:

[0370] The server uses a generative AI model to generate questions based on a prompt. This prompt might be something like, "Generate in-depth questions about the user's project experience." The input is the prompt, and the output is the questions presented to the user. Natural language generation is performed within the server by the AI ​​model, resulting in customized questions.

[0371] Step 2:

[0372] The server sends the generated question to the terminal. The terminal presents this to the user as an audio message. The input here is the question text from the server, and the output is the audio the user hears. The terminal's text-to-speech function converts the text into speech, which is then read aloud to the user.

[0373] Step 3:

[0374] The user answers the presented questions via voice through the device. The input is the user's voice response, and the output is the voice data recorded directly on the device. The user provides voice input, and the device collects the data.

[0375] Step 4:

[0376] The terminal converts the user's voice into text data. It uses speech recognition software to convert the input voice data into text information. The output is sent to the server as text data.

[0377] Step 5:

[0378] The device uses a camera to acquire user facial expression information in real time. The input is live video data, and the output generates still images and dynamic facial expression data necessary for facial expression analysis. The camera captures changes in the user's facial expressions, and this information is transferred to the server as processed data.

[0379] Step 6:

[0380] The server analyzes emotions and evaluates suitability based on text data and facial expression information. The input is text data and facial expression data, and the output is an evaluation result indicating the compatibility between the applicant and the organization. The server uses algorithms to perform data calculations and generate an integrated evaluation score.

[0381] Step 7:

[0382] The server generates a report with feedback based on the evaluation results and sends it to the user and the organization. The input is the evaluation results, and the output is the completed report. The server supports the user's next steps by formatting the data and generating an easy-to-read report.

[0383] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0384] This invention is an automated interview system using artificial intelligence and an emotion engine, and its specific embodiments are described below.

[0385] In this system, the server first automatically sets up the interview session according to the interview schedule. At the designated time, the terminal starts capturing the user's audio and video in real time through the connected camera and microphone.

[0386] Once the interview begins, the server uses a conversation generation model to automatically generate an initial greeting and an explanation of the purpose for the user. The terminal then communicates this to the user verbally, allowing the user to recognize the start of the interview.

[0387] Next, the server generates a set of questions, selects a customized initial question, and presents it to the user through the terminal. The user's responses are recorded on the terminal and converted to text using speech recognition technology. In parallel with this audio data, the terminal also captures the user's facial expressions with high accuracy to collect data necessary for sentiment analysis.

[0388] A key feature of this system is its emotion engine, which analyzes voice tone, facial expressions, and even the user's response time to assess the user's emotional state in real time. For example, if tension is detected, the server adjusts the content and tone of the questions asked by the system to the user to encourage a more relaxed response.

[0389] The server integrates this sentiment data with user responses and calculates the degree of fit with the characteristics the company is looking for. The evaluation results are generated as a report and provided to the company, and the user is notified with feedback.

[0390] For example, if a user temporarily stops responding due to nervousness, the system will detect this and encourage relaxation by inserting easy-to-use random jokes or light conversation, then promptly ask questions again to allow the user to perform at their best.

[0391] This embodiment provides a fairer and less stressful environment compared to conventional interviews, while also enabling more accurate talent evaluation.

[0392] The following describes the processing flow.

[0393] Step 1:

[0394] The server checks the interview schedule and prepares to set up the interview session at the specified date and time. It loads the interview template and required question list and verifies that it is ready.

[0395] Step 2:

[0396] The device tests the connection status of the camera and microphone and checks their performance. It checks the clarity of the audio and the quality of the video and adjusts them to the optimal state.

[0397] Step 3:

[0398] The server uses AI to generate an initial message that greets the user and explains the interview process and purpose. This message is designed to help the user feel at ease during the interview.

[0399] Step 4:

[0400] The device plays an initial message as audio to the user, informing them that the interview is about to begin. This helps the user prepare themselves mentally.

[0401] Step 5:

[0402] The server selects the first question and generates questions based on key themes in the interview process. These questions are aligned with the characteristics of the candidate the company is seeking.

[0403] Step 6:

[0404] The device presents the user with selected questions via voice, and the user responds accordingly. A feature that detects the user's level of anxiety is also utilized during the response process.

[0405] Step 7:

[0406] The device records the user's response and begins the process of converting the audio into text data in real time. At the same time, the user's facial expressions are also captured by the camera.

[0407] Step 8:

[0408] The server receives voice text and facial expression data, and uses an emotion engine to analyze the user's emotional state. This allows it to determine the user's level of tension or excitement.

[0409] Step 9:

[0410] Based on the analysis results, the server dynamically changes the difficulty and tone of questions if necessary to adjust the flow of the conversation. For example, it might add friendly topics to ease tension.

[0411] Step 10:

[0412] The server integrates all the data and evaluates the degree of relevance based on the user's responses and emotional state. This determines how well the user matches the company's needs.

[0413] Step 11:

[0414] The server generates evaluation results in report format and sends them to the company's system for the evaluators. At the same time, it sends the interview evaluation results and feedback to the user.

[0415] Step 12:

[0416] The terminal confirms that all processing is complete, saves the interview data, and then safely shuts down the system. The user is notified that the interview has ended, and it is clearly indicated that the interaction was completed safely.

[0417] (Example 2)

[0418] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0419] Traditional interview systems have suffered from significant influence from the interviewer's subjectivity and the interview environment, making fair and consistent evaluation difficult. Furthermore, it was challenging to appropriately analyze and respond to applicants' emotions and reactions during the interview. Therefore, accurately assessing applicants' actual abilities and aptitudes proved difficult.

[0420] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0421] In this invention, the server includes a dialogue generation means utilizing artificial intelligence for conducting automated interviews, a means for collecting data using a sensor device for acquiring audio and video, and a speech recognition means for converting the collected audio information into text information. This enables the analysis of applicants' emotions and responses during interviews and allows for appropriate responses in real time. This makes it possible to achieve fair and consistent interview evaluations and provide valuable information for both companies and applicants.

[0422] "Automated interviews" are an interview method that uses artificial intelligence to communicate with job seekers without the need for human interviewers.

[0423] "Dialogue generation means" refers to a technology that uses artificial intelligence to automatically generate questions and responses for effective communication with job seekers.

[0424] A "sensor device" is a hardware device used to acquire environmental information such as sound and video.

[0425] "Speech recognition means" refers to processing technology for converting acquired speech information into text information and digitizing it.

[0426] "Facial expression information" refers to digital data acquired from the facial movements and characteristics of job applicants, and is used for emotional analysis.

[0427] "Emotion" refers to a person's psychological state and is a part of the feelings that can be analyzed from tone of voice and facial expressions.

[0428] "Suitability" is a measure used to evaluate how well a job seeker matches the characteristics and abilities that a company is looking for.

[0429] A "report" is a document that organizes and summarizes the interview results and analysis, and is provided to the company.

[0430] A "generative AI model" is an artificial intelligence technology that generates intelligent responses or content based on input data.

[0431] "Control means" refers to functions and technologies used to adjust the operation and response of a system.

[0432] This invention relates to an automated interview system that uses artificial intelligence to manage the entire interview process and enable more objective and effective evaluation. The system mainly consists of three elements: a server, a terminal, and a user.

[0433] First, the server handles the system's primary processing. The server generates interview dialogues using a generative AI model. These dialogues are dynamically updated based on user responses. This generative AI model incorporates various natural language processing techniques, specifically using a large-scale language model. This enables appropriate follow-up questions and comments in response to user answers.

[0434] Next, the terminal is responsible for direct interaction with the user. The terminal incorporates audio and video sensors to capture the user's voice and facial expressions with high precision. For this, high-performance cameras and microphones are used as hardware, and reliable speech recognition software is employed on the software side. The captured audio is converted into text data using the speech recognition system.

[0435] Furthermore, the user sits in front of the terminal and participates in the interview scenario. While the user answers the system's dialogue and questions, their facial expressions and tone of voice are captured by sensors. The user's emotions are analyzed in real time, and if tension or anxiety is detected, the server uses a generated AI model to create prompts and adjust the tone and content of the questions. For example, a question prompt such as, "Could you tell me a little more about your experience?" is used.

[0436] As described above, a system applying the present invention provides a fairer and less stressful interview environment compared to conventional interviews, and allows for highly accurate evaluation of the suitability between companies and job seekers.

[0437] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0438] Step 1:

[0439] The server checks the interview schedule and identifies the next interview time and date. Here, a database query is used to retrieve the schedule information and automatically set up the session at the interview start time. The input is the schedule data, and the output is the setting information for the new interview session.

[0440] Step 2:

[0441] The device activates its camera and microphone at the designated interview time, capturing the user's voice and video in real time. High-performance sensors acquire the audio and video data, which is then recorded in a file. Input is the user's speech and facial expressions, while output is audio and video data.

[0442] Step 3:

[0443] When the interview begins, the server uses a generated AI model to create an initial greeting and the purpose of the interview. The greeting, output as text, is converted into speech using speech synthesis software, and the terminal transmits this to the user. The prompt is generated in the format of "The interview will now begin. Please introduce yourself." The input is a prompt to the AI ​​model, and the output is the synthesized greeting.

[0444] Step 4:

[0445] The server selects appropriate questions from a set of questions and customizes them using a generative AI model. The resulting question text is presented to the user via the terminal and output as speech synthesis. The input consists of question data and prompts, and the output is the customized question in audio format.

[0446] Step 5:

[0447] When a user answers a question, the device records the answer as audio. This audio is then converted to text using speech recognition technology and sent to the server. The input is the user's audio answer, and the output is the transcribed answer.

[0448] Step 6:

[0449] The server uses facial recognition technology to analyze the user's facial expression data and evaluate their emotional state. Emotions are analyzed in real time, along with voice tone and response time. Inputs are voice tone and facial expression data, and output is the emotional evaluation result.

[0450] Step 7:

[0451] Based on the results of the emotional assessment, the server uses a generative AI model to adjust the tone of questions and dialogues. For example, it might generate phrases to help the user relax and incorporate them into the next dialogue. The input is the result of the emotional assessment, and the output is the adjusted dialogue.

[0452] Step 8:

[0453] The server integrates user responses and emotional data and runs an algorithm to evaluate suitability. The evaluation results are compiled into a report and provided to companies and users. The input is the user responses and emotional data, and the output is the report.

[0454] (Application Example 2)

[0455] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0456] In automated interview systems, implementing an authentication process that considers the user's emotional state is crucial for reducing the user's psychological burden and providing a less stressful environment. Furthermore, given the current need for fairer and more accurate evaluations through the appropriate use of emotional assessment, the introduction of new payment systems is required. However, existing technologies have the challenge of not being able to adequately consider the user's emotional state and thus not contributing to the smooth operation of the authentication process.

[0457] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0458] In this invention, the server includes a conversation generation means that utilizes intelligence for conducting automated interviews, a means for collecting input information using sensor devices that collect audio and video, and an optimization means that utilizes a relaxing effect in payment authentication based on the user's emotional state. This makes it possible to provide a payment authentication process with a relaxing effect that takes the user's emotional state into consideration.

[0459] "Automated interviews" are a process that utilizes artificial intelligence to automatically ask users questions and analyze their responses.

[0460] "Intelligence" refers to the ability to make judgments and take actions appropriate to a situation based on information processed by a computer.

[0461] A "conversation generation method" is a function in which the system spontaneously creates and presents questions to the user.

[0462] "Audio and video" refers to auditory and visual information obtained from the user, which is captured through sensor devices.

[0463] "Sensor equipment" refers to devices and equipment used to acquire audio and video information.

[0464] "Input information" refers to data collected for system processing, including audio, video, and other related data.

[0465] "Speech recognition means" refers to the technology or process that analyzes acquired speech data and converts it into text information.

[0466] "Character data" refers to text information converted from speech by speech recognition technology.

[0467] "Facial expression data" refers to information obtained by measuring a user's facial expressions and converting their characteristics into digital data.

[0468] "Emotion" refers to the user's instantaneous feelings and psychological state.

[0469] "Means of analyzing emotions" refers to methods and techniques for identifying and evaluating a user's emotional state.

[0470] "Persons being evaluated" refers to individuals whose fitness and other characteristics are assessed by the system.

[0471] "Organization" refers to any group or company that seeks to verify the suitability of the person being evaluated.

[0472] "Fit" is a measure that indicates the degree to which the person being evaluated matches the characteristics required by the organization.

[0473] "Evaluation means" refers to devices or systems used to evaluate the characteristics of a subject based on collected data and to determine their suitability.

[0474] "Optimization methods" refer to ways of adjusting conditions and parameters to improve the efficiency of a process or system.

[0475] "Relaxation effect" refers to the psychological or physical influence that alleviates tension in the subject.

[0476] "Information" is a general term for data that a system or process uses for processing and analysis.

[0477] A "report" refers to a written or electronic format in which evaluation results and analysis content are organized.

[0478] "Output" refers to the act or format in which a system displays or provides the results of its processing.

[0479] The system realizing this invention consists of a server, a terminal, and a user. The server requires an internet connection and functions as a platform for running programs that include an intelligence and emotion engine. The terminal functions as a sensor device that uses a camera and microphone to collect the user's voice and video. The user faces the terminal and participates in an automated interview and authentication process using voice and video.

[0480] The server uses intelligence to execute conversation generation mechanisms and generate questions to present to the user. This process employs a generative AI model. The terminal collects audio and video in real time and sends this data to the server. The server uses speech recognition mechanisms to convert the audio data into text and further evaluates the user's emotional state using emotion analysis mechanisms. This analysis utilizes facial expression data and voice tone, employing software such as TensorFlow and OpenCV.

[0481] Based on this data, the server applies optimization measures to provide users with a relaxing experience, such as playing light background music to alleviate their tension. Finally, the server integrates this information, generates a fitness report, and outputs it to the organization.

[0482] As a concrete example, in the authentication process for electronic payments, if a user is feeling nervous in front of the payment terminal, the server will assess the situation based on the results of emotional analysis and generate and present a prompt to the user that reduces their psychological burden through light conversation such as, "The weather is lovely today, it's a perfect day for shopping."

[0483] Example of a prompt:

[0484] "Explain the mechanism for user authentication using facial recognition and voice analysis. Explain, using examples, how the user is provided with a relaxed state by the system, including specific processing steps."

[0485] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0486] Step 1:

[0487] The device activates its camera and microphone to acquire user audio and video data. This data is temporarily stored on the device and prepared for transmission to the server. The input is the user's video and audio, and the output is raw data. Specifically, the user speaks into the device, the camera captures the video, and the microphone records the audio.

[0488] Step 2:

[0489] The terminal transmits the acquired audio and video data to the server. The terminal compresses the data to reduce the transfer time. The input is the compressed audio and video data, and the output is the data received by the server. Specifically, the terminal uses its communication module to send the data to the server via the internet.

[0490] Step 3:

[0491] The server converts the received audio data into text data using speech recognition technology. This process utilizes the Google Cloud Speech-to-Text API. The input is audio data, and the output is text data. Specifically, the server sends audio data to the API and saves the returned text to an internal database.

[0492] Step 4:

[0493] The server extracts facial expression data from video data and analyzes emotions. It uses TensorFlow to perform facial expression analysis and estimate the user's emotional state. The input is video data, and the output is emotion evaluation data. Specifically, it detects facial feature points from the video and evaluates emotions using a pre-trained model.

[0494] Step 5:

[0495] The server generates optimal dialogue to induce relaxation based on the user's emotional state. This utilizes a generative AI model. The input is emotional assessment data, and the output is a prompt designed to promote relaxation. Specifically, the generative AI creates the prompt and prepares it to be sent to the user's device.

[0496] Step 6:

[0497] The server sends a generated prompt to the terminal, which then presents it to the user via voice or on-screen display. The input is the prompt, and the output is the information presented to the user. Specifically, the terminal either plays the text using speech synthesis or displays the text on the screen.

[0498] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0499] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0500] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0501] [Third Embodiment]

[0502] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0503] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0504] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0505] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0506] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0507] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0508] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0509] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0510] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0511] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0512] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0513] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0514] This invention is an automated interview system that utilizes artificial intelligence, and its specific embodiments are described below.

[0515] In this system, the server first automatically sets up the interview session according to the interview schedule. Next, the terminal uses its connected camera and microphone to acquire audio and video data from the user in real time. At the start of the system, the terminal verbally greets the user with a respectful voice message to encourage them to begin the conversation.

[0516] The server uses an internally built-in generative model to automatically generate questions for the user, which the terminal then presents to the user verbally. These questions are customized based on the talent characteristics sought by the company. For example, when evaluating creativity, questions are prepared to elicit the user's original ideas.

[0517] When a user answers a question, the device records the audio and uses speech recognition technology to convert the content into text. Facial expressions are also captured in real time, and this data is sent to a server.

[0518] On the server, sentiment analysis and evaluation are performed based on the received text and facial expression data. The user's emotional state is estimated from changes in voice tone and facial expressions, and the candidate's logical thinking ability and creativity are analyzed. These analysis results are compared and quantified or graded to evaluate the degree of fit with the characteristics required by the company.

[0519] Next, the server performs a suitability assessment and generates a comprehensive report based on the results. This report is provided to the company's evaluators, and the user receives the interview data results and feedback. For example, the assessment report may indicate that the user was in a relaxed emotional state or demonstrated a high level of logical thinking.

[0520] In this way, the present invention can significantly improve the fairness and efficiency of interviews and can provide value in recruitment activities in many industrial fields.

[0521] The following describes the processing flow.

[0522] Step 1:

[0523] The server checks the interview schedule and automatically prepares the interview session at the appropriate date and time. This includes loading interview templates and allocating the necessary computing resources.

[0524] Step 2:

[0525] Check the status of the camera and microphone connected to the device. Confirm that the connection is working correctly, test the volume and video quality, and adjust the settings to the optimal level for conducting the interview.

[0526] Step 3:

[0527] The server activates an AI-generated model to create an initial greeting message for the user. This message takes into account the naturalness and politeness of the conversation.

[0528] Step 4:

[0529] The device converts the generated greeting message into audio and plays it for the user. This allows the user to recognize the start of the interview.

[0530] Step 5:

[0531] The server randomizes or customizes the question set based on company metrics and selects the first question.

[0532] Step 6:

[0533] The device presents the selected question to the user verbally. The user listens to the question and understands its intent.

[0534] Step 7:

[0535] Users will answer questions verbally. Efforts will be made to ensure consistency in answers, and opportunities for re-answering will be provided as needed.

[0536] Step 8:

[0537] The device records the user's responses and converts them into text data in real time using speech recognition technology. The accuracy of the conversion results is checked, and corrections are made as needed.

[0538] Step 9:

[0539] The device captures the user's face as video data and records changes in facial expressions. This allows for the collection of data for emotion analysis.

[0540] Step 10:

[0541] The server integrates voice data and facial expression data to perform emotion analysis. Based on the analysis, it evaluates the user's emotional state and level of tension.

[0542] Step 11:

[0543] The server uses the sentiment analysis results and responses to perform a suitability assessment. This assessment is based on an algorithm that calculates the degree of match with the talent characteristics sought by the company.

[0544] Step 12:

[0545] The server generates an evaluation report and notifies the company's evaluators. It also provides users with interview results and feedback.

[0546] Step 13:

[0547] The terminal organizes all interview data, notifies the user that the system is ending, and safely terminates the session.

[0548] (Example 1)

[0549] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0550] Traditional HR interview processes suffer from a lack of fairness and efficiency, and consume a significant amount of human resources. Furthermore, the evaluation process is highly subjective, making it difficult to accurately measure the suitability of candidates for the company.

[0551] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0552] In this invention, the server includes a question and answer generation means utilizing an intelligent agent for conducting automated interviews, a means for collecting input information using a sensing device that acquires acoustic and visual data, and a recognition means for converting the collected audio information into textual information. This automates the interview process and enables objective and efficient evaluation of candidates.

[0553] An "automated interview" is an interview process that utilizes artificial intelligence technology to minimize human intervention.

[0554] An "intelligent agent" is a program or system that uses artificial intelligence to autonomously perform a specific task.

[0555] A "question and answer generation tool" is a mechanism for generating questions based on specific conditions or criteria and for eliciting answers to those questions.

[0556] A "sensing device" is hardware that has the function of acquiring external data such as audio and video.

[0557] "Recognition means" refers to technology that analyzes input audio data and converts it into corresponding text data.

[0558] "Emotion inference techniques" are technologies used to infer a user's emotions from their voice tone and facial expressions.

[0559] A "fitness quantification method" is a system that expresses the degree to which a candidate's characteristics match the characteristics sought by a company using numerical values.

[0560] A "report generation method" is a technology that aggregates interview results and evaluation data and outputs them as a report in a standardized format.

[0561] The "automatic schedule setting method" is a function that automatically sets the date, time, and order of interviews based on predetermined conditions.

[0562] "Dynamic evaluation" is a process that collects data in real time and performs evaluations based on information that changes as it occurs.

[0563] "Real-time information gathering" refers to the process of quickly acquiring data from a target in real time.

[0564] This invention is for constructing an automated interview system, which realizes an advanced interview process through the coordination of an intelligent agent, a sensing device, a recognition means, and related technologies.

[0565] First, the server has a mechanism to manage the automated interview schedule, automatically configuring the system based on the scheduled interviews. This allows the server to efficiently prepare for interviews.

[0566] Next, the device uses its connected camera and microphone to acquire the user's voice and video in real time. General-purpose APIs are used for device operation, and data is sent to the server.

[0567] Artificial intelligence (AI) models are used to generate speech synthesis technology for greetings and questions to the user. The server utilizes the generation AI model to create questions for the user and sets appropriate prompts. A specific example of a prompt could be, "Please generate questions to assess imagination."

[0568] When a user responds, the device records the audio and converts it to text using speech recognition technology. Simultaneously, it captures the user's facial expressions, and this data is aggregated on a server.

[0569] The server performs multifaceted analysis based on the aggregated data. It utilizes machine learning models to calculate the degree of fit, including sentiment inference and evaluation of logical thinking ability. The results are ultimately generated as a report and provided to the organization. Furthermore, the results are also communicated to the user as feedback.

[0570] This automated interview system can significantly improve the efficiency and fairness of interviews, demonstrating its value in various industrial sectors.

[0571] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0572] Step 1:

[0573] The server checks the interview schedule. The interview date, time, and candidate information are read from the database as input. Based on this information, the server prepares the session settings and starts the relevant modules. This completes the preparation for the automated interview process.

[0574] Step 2:

[0575] The device activates its camera and microphone to acquire audio and video data from the user. The input is the user's voice and video, which are captured in real time. The collected data is sent to a server via the network as output. The device API is utilized to ensure accurate data acquisition and transmission.

[0576] Step 3:

[0577] The terminal uses speech synthesis technology to greet the user. Text data from the server is used as input, and speech synthesis platform is used to output voice data. This allows the user to recognize the start of the process.

[0578] Step 4:

[0579] The server generates questions using a generation AI model. The input here is a prompt based on the characteristics requested by the company. An example prompt is "Generate questions that stimulate creativity." As a result, the generated questions are output in text format and sent to the terminal.

[0580] Step 5:

[0581] The terminal presents the user with a generated question. The input is text data sent from the server, which is then processed again using speech synthesis and output as audio data. The user then answers this audio question.

[0582] Step 6:

[0583] The device records the user's responses, and the audio data is converted into text data using recognition technology. The user's audio data is used as input, and the converted text is output. In addition, facial expression data is also captured and sent to the server.

[0584] Step 7:

[0585] The server analyzes the collected text and facial expression data. The input consists of speech-recognized text and facial expression data, and data calculations are performed to evaluate emotion inference and logical thinking ability. As a result of the analysis, the degree of fit is quantified and output as data for organizational evaluation.

[0586] Step 8:

[0587] The server aggregates the analysis results and generates a report. The input consists of evaluated fit and analysis information, and a report is output based on this information in a standard format. This report is provided to the organization's evaluators, and feedback is sent to the user.

[0588] (Application Example 1)

[0589] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0590] Traditional interview methods are constrained by time and location, resulting in inefficient and sometimes unobjective interview processes. Furthermore, subjective evaluations by interviewers can be biased, making it difficult to accurately assess the applicant's suitability for the organization's talent requirements. Additionally, immediate feedback after interviews is often lacking, hindering applicants from having opportunities for self-improvement.

[0591] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0592] In this invention, the server includes a question-and-answer means based on conversation generation, a means for acquiring information using an input device that records audio and video, and a recognition means for converting audio information into text information. This makes it possible to dynamically and objectively evaluate the compatibility between applicants and organizations.

[0593] "Conversation generation" is the process of automatically conducting question-and-answer sessions using artificial intelligence.

[0594] A "question and answer system" is a function that allows the system to automatically generate questions and receive answers from users.

[0595] An "input device" is a device used to acquire data such as audio and video, and includes sensors, microphones, and cameras.

[0596] "Means of acquiring information" refers to the process of collecting audio and video data through digital devices.

[0597] "Recognition means" refers to the process of converting acquired audio into text, and utilizes speech recognition technology.

[0598] "Facial expression information" refers to the digital data recording of the user's facial movements.

[0599] "Means for analyzing emotions" refers to technologies that estimate a user's emotional state from facial expression information and voice data.

[0600] "Methods for evaluating consistency" refer to the process of comparing the characteristics of applicants with the requirements of the organization and quantifying the degree of agreement between them.

[0601] "Smart devices" refer to mobile terminals with advanced information processing capabilities, such as mobile phones, tablets, and wearable devices.

[0602] A "report with feedback" is a document that includes an evaluation based on the interview results, intended to communicate areas for improvement to the user.

[0603] "Methods for dynamically acquiring data" refer to functions that continuously collect information that changes in real time.

[0604] In the system that implements this application, a server plays a primary role. The server generates questions to present to the user using a generative AI model and sends them to the terminal. The terminal, as part of a smart device, uses a microphone and camera to collect audio and video data from the user. Using speech recognition technology, the terminal converts the audio into text data in real time.

[0605] The server analyzes emotions based on acquired text data and facial expression information. This analysis uses proprietary emotion analysis software to estimate the user's emotional state. Next, the analyzed data is used to evaluate the suitability of the applicant to the organization, and the results are quantified or graded. This evaluation result is generated as a report with feedback and sent to both the user and the organization.

[0606] As a specific example of the system, we assume that the terminal is a smartphone. In this case, the smartphone's camera and microphone are utilized, and the user participates in the interview process via the internet. The user can flexibly respond to the questions received through the terminal from anywhere, such as their home or workplace.

[0607] An example of a prompt would be, "Generate in-depth questions about the user's project experience, and then perform a sentiment assessment based on the responses." By using this prompt, the generating AI model can provide high-quality questions tailored to the purpose, creating a foundation for thoroughly evaluating the applicant's characteristics.

[0608] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0609] Step 1:

[0610] The server uses a generative AI model to generate questions based on a prompt. This prompt might be something like, "Generate in-depth questions about the user's project experience." The input is the prompt, and the output is the questions presented to the user. Natural language generation is performed within the server by the AI ​​model, resulting in customized questions.

[0611] Step 2:

[0612] The server sends the generated question to the terminal. The terminal presents this to the user as an audio message. The input here is the question text from the server, and the output is the audio the user hears. The terminal's text-to-speech function converts the text into speech, which is then read aloud to the user.

[0613] Step 3:

[0614] The user answers the presented questions via voice through the device. The input is the user's voice response, and the output is the voice data recorded directly on the device. The user provides voice input, and the device collects the data.

[0615] Step 4:

[0616] The terminal converts the user's voice into text data. It uses speech recognition software to convert the input voice data into text information. The output is sent to the server as text data.

[0617] Step 5:

[0618] The device uses a camera to acquire user facial expression information in real time. The input is live video data, and the output generates still images and dynamic facial expression data necessary for facial expression analysis. The camera captures changes in the user's facial expressions, and this information is transferred to the server as processed data.

[0619] Step 6:

[0620] The server analyzes emotions and evaluates suitability based on text data and facial expression information. The input is text data and facial expression data, and the output is an evaluation result indicating the compatibility between the applicant and the organization. The server uses algorithms to perform data calculations and generate an integrated evaluation score.

[0621] Step 7:

[0622] The server generates a report with feedback based on the evaluation results and sends it to the user and the organization. The input is the evaluation results, and the output is the completed report. The server supports the user's next steps by formatting the data and generating an easy-to-read report.

[0623] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0624] This invention is an automated interview system using artificial intelligence and an emotion engine, and its specific embodiments are described below.

[0625] In this system, the server first automatically sets up the interview session according to the interview schedule. At the designated time, the terminal starts capturing the user's audio and video in real time through the connected camera and microphone.

[0626] Once the interview begins, the server uses a conversation generation model to automatically generate an initial greeting and an explanation of the purpose for the user. The terminal then communicates this to the user verbally, allowing the user to recognize the start of the interview.

[0627] Next, the server generates a set of questions, selects a customized initial question, and presents it to the user through the terminal. The user's responses are recorded on the terminal and converted to text using speech recognition technology. In parallel with this audio data, the terminal also captures the user's facial expressions with high accuracy to collect data necessary for sentiment analysis.

[0628] A key feature of this system is its emotion engine, which analyzes voice tone, facial expressions, and even the user's response time to assess the user's emotional state in real time. For example, if tension is detected, the server adjusts the content and tone of the questions asked by the system to the user to encourage a more relaxed response.

[0629] The server integrates this sentiment data with user responses and calculates the degree of fit with the characteristics the company is looking for. The evaluation results are generated as a report and provided to the company, and the user is notified with feedback.

[0630] For example, if a user temporarily stops responding due to nervousness, the system will detect this and encourage relaxation by inserting easy-to-use random jokes or light conversation, then promptly ask questions again to allow the user to perform at their best.

[0631] This embodiment provides a fairer and less stressful environment compared to conventional interviews, while also enabling more accurate talent evaluation.

[0632] The following describes the processing flow.

[0633] Step 1:

[0634] The server checks the interview schedule and prepares to set up the interview session at the specified date and time. It loads the interview template and required question list and verifies that it is ready.

[0635] Step 2:

[0636] The device tests the connection status of the camera and microphone and checks their performance. It checks the clarity of the audio and the quality of the video and adjusts them to the optimal state.

[0637] Step 3:

[0638] The server uses AI to generate an initial message that greets the user and explains the interview process and purpose. This message is designed to help the user feel at ease during the interview.

[0639] Step 4:

[0640] The device plays an initial message as audio to the user, informing them that the interview is about to begin. This helps the user prepare themselves mentally.

[0641] Step 5:

[0642] The server selects the first question and generates questions based on key themes in the interview process. These questions are aligned with the characteristics of the candidate the company is seeking.

[0643] Step 6:

[0644] The device presents the user with selected questions via voice, and the user responds accordingly. A feature that detects the user's level of anxiety is also utilized during the response process.

[0645] Step 7:

[0646] The device records the user's response and begins the process of converting the audio into text data in real time. At the same time, the user's facial expressions are also captured by the camera.

[0647] Step 8:

[0648] The server receives voice text and facial expression data, and uses an emotion engine to analyze the user's emotional state. This allows it to determine the user's level of tension or excitement.

[0649] Step 9:

[0650] Based on the analysis results, the server dynamically changes the difficulty and tone of questions if necessary to adjust the flow of the conversation. For example, it might add friendly topics to ease tension.

[0651] Step 10:

[0652] The server integrates all the data and evaluates the degree of relevance based on the user's responses and emotional state. This determines how well the user matches the company's needs.

[0653] Step 11:

[0654] The server generates evaluation results in report format and sends them to the company's system for the evaluators. At the same time, it sends the interview evaluation results and feedback to the user.

[0655] Step 12:

[0656] The terminal confirms that all processing is complete, saves the interview data, and then safely shuts down the system. The user is notified that the interview has ended, and it is clearly indicated that the interaction was completed safely.

[0657] (Example 2)

[0658] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0659] Traditional interview systems have suffered from significant influence from the interviewer's subjectivity and the interview environment, making fair and consistent evaluation difficult. Furthermore, it was challenging to appropriately analyze and respond to applicants' emotions and reactions during the interview. Therefore, accurately assessing applicants' actual abilities and aptitudes proved difficult.

[0660] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0661] In this invention, the server includes a dialogue generation means utilizing artificial intelligence for conducting automated interviews, a means for collecting data using a sensor device for acquiring audio and video, and a speech recognition means for converting the collected audio information into text information. This enables the analysis of applicants' emotions and responses during interviews and allows for appropriate responses in real time. This makes it possible to achieve fair and consistent interview evaluations and provide valuable information for both companies and applicants.

[0662] "Automated interviews" are an interview method that uses artificial intelligence to communicate with job seekers without the need for human interviewers.

[0663] "Dialogue generation means" refers to a technology that uses artificial intelligence to automatically generate questions and responses for effective communication with job seekers.

[0664] A "sensor device" is a hardware device used to acquire environmental information such as sound and video.

[0665] "Speech recognition means" refers to processing technology for converting acquired speech information into text information and digitizing it.

[0666] "Facial expression information" refers to digital data acquired from the facial movements and characteristics of job applicants, and is used for emotional analysis.

[0667] "Emotion" refers to a person's psychological state and is a part of the feelings that can be analyzed from tone of voice and facial expressions.

[0668] "Suitability" is a measure used to evaluate how well a job seeker matches the characteristics and abilities that a company is looking for.

[0669] A "report" is a document that organizes and summarizes the interview results and analysis, and is provided to the company.

[0670] A "generative AI model" is an artificial intelligence technology that generates intelligent responses or content based on input data.

[0671] "Control means" refers to functions and technologies used to adjust the operation and response of a system.

[0672] This invention relates to an automated interview system that uses artificial intelligence to manage the entire interview process and enable more objective and effective evaluation. The system mainly consists of three elements: a server, a terminal, and a user.

[0673] First, the server handles the system's primary processing. The server generates interview dialogues using a generative AI model. These dialogues are dynamically updated based on user responses. This generative AI model incorporates various natural language processing techniques, specifically using a large-scale language model. This enables appropriate follow-up questions and comments in response to user answers.

[0674] Next, the terminal is responsible for direct interaction with the user. The terminal incorporates audio and video sensors to capture the user's voice and facial expressions with high precision. For this, high-performance cameras and microphones are used as hardware, and reliable speech recognition software is employed on the software side. The captured audio is converted into text data using the speech recognition system.

[0675] Furthermore, the user sits in front of the terminal and participates in the interview scenario. While the user answers the system's dialogue and questions, their facial expressions and tone of voice are captured by sensors. The user's emotions are analyzed in real time, and if tension or anxiety is detected, the server uses a generated AI model to create prompts and adjust the tone and content of the questions. For example, a question prompt such as, "Could you tell me a little more about your experience?" is used.

[0676] As described above, a system applying the present invention provides a fairer and less stressful interview environment compared to conventional interviews, and allows for highly accurate evaluation of the suitability between companies and job seekers.

[0677] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0678] Step 1:

[0679] The server checks the interview schedule and identifies the next interview time and date. Here, a database query is used to retrieve the schedule information and automatically set up the session at the interview start time. The input is the schedule data, and the output is the setting information for the new interview session.

[0680] Step 2:

[0681] The device activates its camera and microphone at the designated interview time, capturing the user's voice and video in real time. High-performance sensors acquire the audio and video data, which is then recorded in a file. Input is the user's speech and facial expressions, while output is audio and video data.

[0682] Step 3:

[0683] When the interview begins, the server uses a generated AI model to create an initial greeting and the purpose of the interview. The greeting, output as text, is converted into speech using speech synthesis software, and the terminal transmits this to the user. The prompt is generated in the format of "The interview will now begin. Please introduce yourself." The input is a prompt to the AI ​​model, and the output is the synthesized greeting.

[0684] Step 4:

[0685] The server selects appropriate questions from a set of questions and customizes them using a generative AI model. The resulting question text is presented to the user via the terminal and output as speech synthesis. The input consists of question data and prompts, and the output is the customized question in audio format.

[0686] Step 5:

[0687] When a user answers a question, the device records the answer as audio. This audio is then converted to text using speech recognition technology and sent to the server. The input is the user's audio answer, and the output is the transcribed answer.

[0688] Step 6:

[0689] The server uses facial recognition technology to analyze the user's facial expression data and evaluate their emotional state. Emotions are analyzed in real time, along with voice tone and response time. Inputs are voice tone and facial expression data, and output is the emotional evaluation result.

[0690] Step 7:

[0691] Based on the results of the emotional assessment, the server uses a generative AI model to adjust the tone of questions and dialogues. For example, it might generate phrases to help the user relax and incorporate them into the next dialogue. The input is the result of the emotional assessment, and the output is the adjusted dialogue.

[0692] Step 8:

[0693] The server integrates user responses and emotional data and runs an algorithm to evaluate suitability. The evaluation results are compiled into a report and provided to companies and users. The input is the user responses and emotional data, and the output is the report.

[0694] (Application Example 2)

[0695] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0696] In automated interview systems, implementing an authentication process that considers the user's emotional state is crucial for reducing the user's psychological burden and providing a less stressful environment. Furthermore, given the current need for fairer and more accurate evaluations through the appropriate use of emotional assessment, the introduction of new payment systems is required. However, existing technologies have the challenge of not being able to adequately consider the user's emotional state and thus not contributing to the smooth operation of the authentication process.

[0697] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0698] In this invention, the server includes a conversation generation means that utilizes intelligence for conducting automated interviews, a means for collecting input information using sensor devices that collect audio and video, and an optimization means that utilizes a relaxing effect in payment authentication based on the user's emotional state. This makes it possible to provide a payment authentication process with a relaxing effect that takes the user's emotional state into consideration.

[0699] "Automated interviews" are a process that utilizes artificial intelligence to automatically ask users questions and analyze their responses.

[0700] "Intelligence" refers to the ability to make judgments and take actions appropriate to a situation based on information processed by a computer.

[0701] A "conversation generation method" is a function in which the system spontaneously creates and presents questions to the user.

[0702] "Audio and video" refers to auditory and visual information obtained from the user, which is captured through sensor devices.

[0703] "Sensor equipment" refers to devices and equipment used to acquire audio and video information.

[0704] "Input information" refers to data collected for system processing, including audio, video, and other related data.

[0705] "Speech recognition means" refers to the technology or process that analyzes acquired speech data and converts it into text information.

[0706] "Character data" refers to text information converted from speech by speech recognition technology.

[0707] "Facial expression data" refers to information obtained by measuring a user's facial expressions and converting their characteristics into digital data.

[0708] "Emotion" refers to the user's instantaneous feelings and psychological state.

[0709] "Means of analyzing emotions" refers to methods and techniques for identifying and evaluating a user's emotional state.

[0710] "Persons being evaluated" refers to individuals whose fitness and other characteristics are assessed by the system.

[0711] "Organization" refers to any group or company that seeks to verify the suitability of the person being evaluated.

[0712] "Fit" is a measure that indicates the degree to which the person being evaluated matches the characteristics required by the organization.

[0713] "Evaluation means" refers to devices or systems used to evaluate the characteristics of a subject based on collected data and to determine their suitability.

[0714] "Optimization methods" refer to ways of adjusting conditions and parameters to improve the efficiency of a process or system.

[0715] "Relaxation effect" refers to the psychological or physical influence that alleviates tension in the subject.

[0716] "Information" is a general term for data that a system or process uses for processing and analysis.

[0717] A "report" refers to a written or electronic format in which evaluation results and analysis content are organized.

[0718] "Output" refers to the act or format in which a system displays or provides the results of its processing.

[0719] The system realizing this invention consists of a server, a terminal, and a user. The server requires an internet connection and functions as a platform for running programs that include an intelligence and emotion engine. The terminal functions as a sensor device that uses a camera and microphone to collect the user's voice and video. The user faces the terminal and participates in an automated interview and authentication process using voice and video.

[0720] The server uses intelligence to execute conversation generation mechanisms and generate questions to present to the user. This process employs a generative AI model. The terminal collects audio and video in real time and sends this data to the server. The server uses speech recognition mechanisms to convert the audio data into text and further evaluates the user's emotional state using emotion analysis mechanisms. This analysis utilizes facial expression data and voice tone, employing software such as TensorFlow and OpenCV.

[0721] Based on this data, the server applies optimization measures to provide users with a relaxing experience, such as playing light background music to alleviate their tension. Finally, the server integrates this information, generates a fitness report, and outputs it to the organization.

[0722] As a concrete example, in the authentication process for electronic payments, if a user is feeling nervous in front of the payment terminal, the server will assess the situation based on the results of emotional analysis and generate and present a prompt to the user that reduces their psychological burden through light conversation such as, "The weather is lovely today, it's a perfect day for shopping."

[0723] Example of a prompt:

[0724] "Explain the mechanism for user authentication using facial recognition and voice analysis. Explain, using examples, how the user is provided with a relaxed state by the system, including specific processing steps."

[0725] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0726] Step 1:

[0727] The device activates its camera and microphone to acquire user audio and video data. This data is temporarily stored on the device and prepared for transmission to the server. The input is the user's video and audio, and the output is raw data. Specifically, the user speaks into the device, the camera captures the video, and the microphone records the audio.

[0728] Step 2:

[0729] The terminal transmits the acquired audio and video data to the server. The terminal compresses the data to reduce the transfer time. The input is the compressed audio and video data, and the output is the data received by the server. Specifically, the terminal uses its communication module to send the data to the server via the internet.

[0730] Step 3:

[0731] The server converts the received audio data into text data using speech recognition technology. This process utilizes the Google Cloud Speech-to-Text API. The input is audio data, and the output is text data. Specifically, the server sends audio data to the API and saves the returned text to an internal database.

[0732] Step 4:

[0733] The server extracts facial expression data from video data and analyzes emotions. It uses TensorFlow to perform facial expression analysis and estimate the user's emotional state. The input is video data, and the output is emotion evaluation data. Specifically, it detects facial feature points from the video and evaluates emotions using a pre-trained model.

[0734] Step 5:

[0735] The server generates optimal dialogue to induce relaxation based on the user's emotional state. This utilizes a generative AI model. The input is emotional assessment data, and the output is a prompt designed to promote relaxation. Specifically, the generative AI creates the prompt and prepares it to be sent to the user's device.

[0736] Step 6:

[0737] The server sends a generated prompt to the terminal, which then presents it to the user via voice or on-screen display. The input is the prompt, and the output is the information presented to the user. Specifically, the terminal either plays the text using speech synthesis or displays the text on the screen.

[0738] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0739] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0740] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0741] [Fourth Embodiment]

[0742] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0743] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0744] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0745] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0746] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0747] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0748] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0749] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0750] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0751] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0752] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0753] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0754] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0755] This invention is an automated interview system that utilizes artificial intelligence, and its specific embodiments are described below.

[0756] In this system, the server first automatically sets up the interview session according to the interview schedule. Next, the terminal uses its connected camera and microphone to acquire audio and video data from the user in real time. At the start of the system, the terminal verbally greets the user with a respectful voice message to encourage them to begin the conversation.

[0757] The server uses an internally built-in generative model to automatically generate questions for the user, which the terminal then presents to the user verbally. These questions are customized based on the talent characteristics sought by the company. For example, when evaluating creativity, questions are prepared to elicit the user's original ideas.

[0758] When a user answers a question, the device records the audio and uses speech recognition technology to convert the content into text. Facial expressions are also captured in real time, and this data is sent to a server.

[0759] On the server, sentiment analysis and evaluation are performed based on the received text and facial expression data. The user's emotional state is estimated from changes in voice tone and facial expressions, and the candidate's logical thinking ability and creativity are analyzed. These analysis results are compared and quantified or graded to evaluate the degree of fit with the characteristics required by the company.

[0760] Next, the server performs a suitability assessment and generates a comprehensive report based on the results. This report is provided to the company's evaluators, and the user receives the interview data results and feedback. For example, the assessment report may indicate that the user was in a relaxed emotional state or demonstrated a high level of logical thinking.

[0761] In this way, the present invention can significantly improve the fairness and efficiency of interviews and can provide value in recruitment activities in many industrial fields.

[0762] The following describes the processing flow.

[0763] Step 1:

[0764] The server checks the interview schedule and automatically prepares the interview session at the appropriate date and time. This includes loading interview templates and allocating the necessary computing resources.

[0765] Step 2:

[0766] Check the status of the camera and microphone connected to the device. Confirm that the connection is working correctly, test the volume and video quality, and adjust the settings to the optimal level for conducting the interview.

[0767] Step 3:

[0768] The server activates an AI-generated model to create an initial greeting message for the user. This message takes into account the naturalness and politeness of the conversation.

[0769] Step 4:

[0770] The device converts the generated greeting message into audio and plays it for the user. This allows the user to recognize the start of the interview.

[0771] Step 5:

[0772] The server randomizes or customizes the question set based on company metrics and selects the first question.

[0773] Step 6:

[0774] The device presents the selected question to the user verbally. The user listens to the question and understands its intent.

[0775] Step 7:

[0776] Users will answer questions verbally. Efforts will be made to ensure consistency in answers, and opportunities for re-answering will be provided as needed.

[0777] Step 8:

[0778] The device records the user's responses and converts them into text data in real time using speech recognition technology. The accuracy of the conversion results is checked, and corrections are made as needed.

[0779] Step 9:

[0780] The device captures the user's face as video data and records changes in facial expressions. This allows for the collection of data for emotion analysis.

[0781] Step 10:

[0782] The server integrates voice data and facial expression data to perform emotion analysis. Based on the analysis, it evaluates the user's emotional state and level of tension.

[0783] Step 11:

[0784] The server uses the sentiment analysis results and responses to perform a suitability assessment. This assessment is based on an algorithm that calculates the degree of match with the talent characteristics sought by the company.

[0785] Step 12:

[0786] The server generates an evaluation report and notifies the company's evaluators. It also provides users with interview results and feedback.

[0787] Step 13:

[0788] The terminal organizes all interview data, notifies the user that the system is ending, and safely terminates the session.

[0789] (Example 1)

[0790] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0791] Traditional HR interview processes suffer from a lack of fairness and efficiency, and consume a significant amount of human resources. Furthermore, the evaluation process is highly subjective, making it difficult to accurately measure the suitability of candidates for the company.

[0792] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0793] In this invention, the server includes a question and answer generation means utilizing an intelligent agent for conducting automated interviews, a means for collecting input information using a sensing device that acquires acoustic and visual data, and a recognition means for converting the collected audio information into textual information. This automates the interview process and enables objective and efficient evaluation of candidates.

[0794] An "automated interview" is an interview process that utilizes artificial intelligence technology to minimize human intervention.

[0795] An "intelligent agent" is a program or system that uses artificial intelligence to autonomously perform a specific task.

[0796] A "question and answer generation tool" is a mechanism for generating questions based on specific conditions or criteria and for eliciting answers to those questions.

[0797] A "sensing device" is hardware that has the function of acquiring external data such as audio and video.

[0798] "Recognition means" refers to technology that analyzes input audio data and converts it into corresponding text data.

[0799] "Emotion inference techniques" are technologies used to infer a user's emotions from their voice tone and facial expressions.

[0800] A "fitness quantification method" is a system that expresses the degree to which a candidate's characteristics match the characteristics sought by a company using numerical values.

[0801] A "report generation method" is a technology that aggregates interview results and evaluation data and outputs them as a report in a standardized format.

[0802] The "automatic schedule setting method" is a function that automatically sets the date, time, and order of interviews based on predetermined conditions.

[0803] "Dynamic evaluation" is a process that collects data in real time and performs evaluations based on information that changes as it occurs.

[0804] "Real-time information gathering" refers to the process of quickly acquiring data from a target in real time.

[0805] This invention is for constructing an automated interview system, which realizes an advanced interview process through the coordination of an intelligent agent, a sensing device, a recognition means, and related technologies.

[0806] First, the server has a mechanism to manage the automated interview schedule, automatically configuring the system based on the scheduled interviews. This allows the server to efficiently prepare for interviews.

[0807] Next, the device uses its connected camera and microphone to acquire the user's voice and video in real time. General-purpose APIs are used for device operation, and data is sent to the server.

[0808] Artificial intelligence (AI) models are used to generate speech synthesis technology for greetings and questions to the user. The server utilizes the generation AI model to create questions for the user and sets appropriate prompts. A specific example of a prompt could be, "Please generate questions to assess imagination."

[0809] When a user responds, the device records the audio and converts it to text using speech recognition technology. Simultaneously, it captures the user's facial expressions, and this data is aggregated on a server.

[0810] The server performs multifaceted analysis based on the aggregated data. It utilizes machine learning models to calculate the degree of fit, including sentiment inference and evaluation of logical thinking ability. The results are ultimately generated as a report and provided to the organization. Furthermore, the results are also communicated to the user as feedback.

[0811] This automated interview system can significantly improve the efficiency and fairness of interviews, demonstrating its value in various industrial sectors.

[0812] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0813] Step 1:

[0814] The server checks the interview schedule. The interview date, time, and candidate information are read from the database as input. Based on this information, the server prepares the session settings and starts the relevant modules. This completes the preparation for the automated interview process.

[0815] Step 2:

[0816] The device activates its camera and microphone to acquire audio and video data from the user. The input is the user's voice and video, which are captured in real time. The collected data is sent to a server via the network as output. The device API is utilized to ensure accurate data acquisition and transmission.

[0817] Step 3:

[0818] The terminal uses speech synthesis technology to greet the user. Text data from the server is used as input, and speech synthesis platform is used to output voice data. This allows the user to recognize the start of the process.

[0819] Step 4:

[0820] The server generates questions using a generation AI model. The input here is a prompt based on the characteristics requested by the company. An example prompt is "Generate questions that stimulate creativity." As a result, the generated questions are output in text format and sent to the terminal.

[0821] Step 5:

[0822] The terminal presents the user with a generated question. The input is text data sent from the server, which is then processed again using speech synthesis and output as audio data. The user then answers this audio question.

[0823] Step 6:

[0824] The device records the user's responses, and the audio data is converted into text data using recognition technology. The user's audio data is used as input, and the converted text is output. In addition, facial expression data is also captured and sent to the server.

[0825] Step 7:

[0826] The server analyzes the collected text and facial expression data. The input consists of speech-recognized text and facial expression data, and data calculations are performed to evaluate emotion inference and logical thinking ability. As a result of the analysis, the degree of fit is quantified and output as data for organizational evaluation.

[0827] Step 8:

[0828] The server aggregates the analysis results and generates a report. The input consists of evaluated fit and analysis information, and a report is output based on this information in a standard format. This report is provided to the organization's evaluators, and feedback is sent to the user.

[0829] (Application Example 1)

[0830] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0831] Traditional interview methods are constrained by time and location, resulting in inefficient and sometimes unobjective interview processes. Furthermore, subjective evaluations by interviewers can be biased, making it difficult to accurately assess the applicant's suitability for the organization's talent requirements. Additionally, immediate feedback after interviews is often lacking, hindering applicants from having opportunities for self-improvement.

[0832] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0833] In this invention, the server includes a question-and-answer means based on conversation generation, a means for acquiring information using an input device that records audio and video, and a recognition means for converting audio information into text information. This makes it possible to dynamically and objectively evaluate the compatibility between applicants and organizations.

[0834] "Conversation generation" is the process of automatically conducting question-and-answer sessions using artificial intelligence.

[0835] A "question and answer system" is a function that allows the system to automatically generate questions and receive answers from users.

[0836] An "input device" is a device used to acquire data such as audio and video, and includes sensors, microphones, and cameras.

[0837] "Means of acquiring information" refers to the process of collecting audio and video data through digital devices.

[0838] "Recognition means" refers to the process of converting acquired audio into text, and utilizes speech recognition technology.

[0839] "Facial expression information" refers to the digital data recording of the user's facial movements.

[0840] "Means for analyzing emotions" refers to technologies that estimate a user's emotional state from facial expression information and voice data.

[0841] "Methods for evaluating consistency" refer to the process of comparing the characteristics of applicants with the requirements of the organization and quantifying the degree of agreement between them.

[0842] "Smart devices" refer to mobile terminals with advanced information processing capabilities, such as mobile phones, tablets, and wearable devices.

[0843] A "report with feedback" is a document that includes an evaluation based on the interview results, intended to communicate areas for improvement to the user.

[0844] "Methods for dynamically acquiring data" refer to functions that continuously collect information that changes in real time.

[0845] In the system that implements this application, a server plays a primary role. The server generates questions to present to the user using a generative AI model and sends them to the terminal. The terminal, as part of a smart device, uses a microphone and camera to collect audio and video data from the user. Using speech recognition technology, the terminal converts the audio into text data in real time.

[0846] The server analyzes emotions based on acquired text data and facial expression information. This analysis uses proprietary emotion analysis software to estimate the user's emotional state. Next, the analyzed data is used to evaluate the suitability of the applicant to the organization, and the results are quantified or graded. This evaluation result is generated as a report with feedback and sent to both the user and the organization.

[0847] As a specific example of the system, we assume that the terminal is a smartphone. In this case, the smartphone's camera and microphone are utilized, and the user participates in the interview process via the internet. The user can flexibly respond to the questions received through the terminal from anywhere, such as their home or workplace.

[0848] An example of a prompt would be, "Generate in-depth questions about the user's project experience, and then perform a sentiment assessment based on the responses." By using this prompt, the generating AI model can provide high-quality questions tailored to the purpose, creating a foundation for thoroughly evaluating the applicant's characteristics.

[0849] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0850] Step 1:

[0851] The server uses a generative AI model to generate questions based on a prompt. This prompt might be something like, "Generate in-depth questions about the user's project experience." The input is the prompt, and the output is the questions presented to the user. Natural language generation is performed within the server by the AI ​​model, resulting in customized questions.

[0852] Step 2:

[0853] The server sends the generated question to the terminal. The terminal presents this to the user as an audio message. The input here is the question text from the server, and the output is the audio the user hears. The terminal's text-to-speech function converts the text into speech, which is then read aloud to the user.

[0854] Step 3:

[0855] The user answers the presented questions via voice through the device. The input is the user's voice response, and the output is the voice data recorded directly on the device. The user provides voice input, and the device collects the data.

[0856] Step 4:

[0857] The terminal converts the user's voice into text data. It uses speech recognition software to convert the input voice data into text information. The output is sent to the server as text data.

[0858] Step 5:

[0859] The device uses a camera to acquire user facial expression information in real time. The input is live video data, and the output generates still images and dynamic facial expression data necessary for facial expression analysis. The camera captures changes in the user's facial expressions, and this information is transferred to the server as processed data.

[0860] Step 6:

[0861] The server analyzes emotions and evaluates suitability based on text data and facial expression information. The input is text data and facial expression data, and the output is an evaluation result indicating the compatibility between the applicant and the organization. The server uses algorithms to perform data calculations and generate an integrated evaluation score.

[0862] Step 7:

[0863] The server generates a report with feedback based on the evaluation results and sends it to the user and the organization. The input is the evaluation results, and the output is the completed report. The server supports the user's next steps by formatting the data and generating an easy-to-read report.

[0864] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0865] This invention is an automated interview system using artificial intelligence and an emotion engine, and its specific embodiments are described below.

[0866] In this system, the server first automatically sets up the interview session according to the interview schedule. At the designated time, the terminal starts capturing the user's audio and video in real time through the connected camera and microphone.

[0867] Once the interview begins, the server uses a conversation generation model to automatically generate an initial greeting and an explanation of the purpose for the user. The terminal then communicates this to the user verbally, allowing the user to recognize the start of the interview.

[0868] Next, the server generates a set of questions, selects a customized initial question, and presents it to the user through the terminal. The user's responses are recorded on the terminal and converted to text using speech recognition technology. In parallel with this audio data, the terminal also captures the user's facial expressions with high accuracy to collect data necessary for sentiment analysis.

[0869] A key feature of this system is its emotion engine, which analyzes voice tone, facial expressions, and even the user's response time to assess the user's emotional state in real time. For example, if tension is detected, the server adjusts the content and tone of the questions asked by the system to the user to encourage a more relaxed response.

[0870] The server integrates this sentiment data with user responses and calculates the degree of fit with the characteristics the company is looking for. The evaluation results are generated as a report and provided to the company, and the user is notified with feedback.

[0871] For example, if a user temporarily stops responding due to nervousness, the system will detect this and encourage relaxation by inserting easy-to-use random jokes or light conversation, then promptly ask questions again to allow the user to perform at their best.

[0872] This embodiment provides a fairer and less stressful environment compared to conventional interviews, while also enabling more accurate talent evaluation.

[0873] The following describes the processing flow.

[0874] Step 1:

[0875] The server checks the interview schedule and prepares to set up the interview session at the specified date and time. It loads the interview template and required question list and verifies that it is ready.

[0876] Step 2:

[0877] The device tests the connection status of the camera and microphone and checks their performance. It checks the clarity of the audio and the quality of the video and adjusts them to the optimal state.

[0878] Step 3:

[0879] The server uses AI to generate an initial message that greets the user and explains the interview process and purpose. This message is designed to help the user feel at ease during the interview.

[0880] Step 4:

[0881] The device plays an initial message as audio to the user, informing them that the interview is about to begin. This helps the user prepare themselves mentally.

[0882] Step 5:

[0883] The server selects the first question and generates questions based on key themes in the interview process. These questions are aligned with the characteristics of the candidate the company is seeking.

[0884] Step 6:

[0885] The device presents the user with selected questions via voice, and the user responds accordingly. A feature that detects the user's level of anxiety is also utilized during the response process.

[0886] Step 7:

[0887] The device records the user's response and begins the process of converting the audio into text data in real time. At the same time, the user's facial expressions are also captured by the camera.

[0888] Step 8:

[0889] The server receives voice text and facial expression data, and uses an emotion engine to analyze the user's emotional state. This allows it to determine the user's level of tension or excitement.

[0890] Step 9:

[0891] Based on the analysis results, the server dynamically changes the difficulty and tone of questions if necessary to adjust the flow of the conversation. For example, it might add friendly topics to ease tension.

[0892] Step 10:

[0893] The server integrates all the data and evaluates the degree of relevance based on the user's responses and emotional state. This determines how well the user matches the company's needs.

[0894] Step 11:

[0895] The server generates evaluation results in report format and sends them to the company's system for the evaluators. At the same time, it sends the interview evaluation results and feedback to the user.

[0896] Step 12:

[0897] The terminal confirms that all processing is complete, saves the interview data, and then safely shuts down the system. The user is notified that the interview has ended, and it is clearly indicated that the interaction was completed safely.

[0898] (Example 2)

[0899] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0900] Traditional interview systems have suffered from significant influence from the interviewer's subjectivity and the interview environment, making fair and consistent evaluation difficult. Furthermore, it was challenging to appropriately analyze and respond to applicants' emotions and reactions during the interview. Therefore, accurately assessing applicants' actual abilities and aptitudes proved difficult.

[0901] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0902] In this invention, the server includes a dialogue generation means utilizing artificial intelligence for conducting automated interviews, a means for collecting data using a sensor device for acquiring audio and video, and a speech recognition means for converting the collected audio information into text information. This enables the analysis of applicants' emotions and responses during interviews and allows for appropriate responses in real time. This makes it possible to achieve fair and consistent interview evaluations and provide valuable information for both companies and applicants.

[0903] "Automated interviews" are an interview method that uses artificial intelligence to communicate with job seekers without the need for human interviewers.

[0904] "Dialogue generation means" refers to a technology that uses artificial intelligence to automatically generate questions and responses for effective communication with job seekers.

[0905] A "sensor device" is a hardware device used to acquire environmental information such as sound and video.

[0906] "Speech recognition means" refers to processing technology for converting acquired speech information into text information and digitizing it.

[0907] "Facial expression information" refers to digital data acquired from the facial movements and characteristics of job applicants, and is used for emotional analysis.

[0908] "Emotion" refers to a person's psychological state and is a part of the feelings that can be analyzed from tone of voice and facial expressions.

[0909] "Suitability" is a measure used to evaluate how well a job seeker matches the characteristics and abilities that a company is looking for.

[0910] A "report" is a document that organizes and summarizes the interview results and analysis, and is provided to the company.

[0911] A "generative AI model" is an artificial intelligence technology that generates intelligent responses or content based on input data.

[0912] "Control means" refers to functions and technologies used to adjust the operation and response of a system.

[0913] This invention relates to an automated interview system that uses artificial intelligence to manage the entire interview process and enable more objective and effective evaluation. The system mainly consists of three elements: a server, a terminal, and a user.

[0914] First, the server handles the system's primary processing. The server generates interview dialogues using a generative AI model. These dialogues are dynamically updated based on user responses. This generative AI model incorporates various natural language processing techniques, specifically using a large-scale language model. This enables appropriate follow-up questions and comments in response to user answers.

[0915] Next, the terminal is responsible for direct interaction with the user. The terminal incorporates audio and video sensors to capture the user's voice and facial expressions with high precision. For this, high-performance cameras and microphones are used as hardware, and reliable speech recognition software is employed on the software side. The captured audio is converted into text data using the speech recognition system.

[0916] Furthermore, the user sits in front of the terminal and participates in the interview scenario. While the user answers the system's dialogue and questions, their facial expressions and tone of voice are captured by sensors. The user's emotions are analyzed in real time, and if tension or anxiety is detected, the server uses a generated AI model to create prompts and adjust the tone and content of the questions. For example, a question prompt such as, "Could you tell me a little more about your experience?" is used.

[0917] As described above, a system applying the present invention provides a fairer and less stressful interview environment compared to conventional interviews, and allows for highly accurate evaluation of the suitability between companies and job seekers.

[0918] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0919] Step 1:

[0920] The server checks the interview schedule and identifies the next interview time and date. Here, a database query is used to retrieve the schedule information and automatically set up the session at the interview start time. The input is the schedule data, and the output is the setting information for the new interview session.

[0921] Step 2:

[0922] The device activates its camera and microphone at the designated interview time, capturing the user's voice and video in real time. High-performance sensors acquire the audio and video data, which is then recorded in a file. Input is the user's speech and facial expressions, while output is audio and video data.

[0923] Step 3:

[0924] When the interview begins, the server uses a generated AI model to create an initial greeting and the purpose of the interview. The greeting, output as text, is converted into speech using speech synthesis software, and the terminal transmits this to the user. The prompt is generated in the format of "The interview will now begin. Please introduce yourself." The input is a prompt to the AI ​​model, and the output is the synthesized greeting.

[0925] Step 4:

[0926] The server selects appropriate questions from a set of questions and customizes them using a generative AI model. The resulting question text is presented to the user via the terminal and output as speech synthesis. The input consists of question data and prompts, and the output is the customized question in audio format.

[0927] Step 5:

[0928] When a user answers a question, the device records the answer as audio. This audio is then converted to text using speech recognition technology and sent to the server. The input is the user's audio answer, and the output is the transcribed answer.

[0929] Step 6:

[0930] The server uses facial recognition technology to analyze the user's facial expression data and evaluate their emotional state. Emotions are analyzed in real time, along with voice tone and response time. Inputs are voice tone and facial expression data, and output is the emotional evaluation result.

[0931] Step 7:

[0932] Based on the results of the emotional assessment, the server uses a generative AI model to adjust the tone of questions and dialogues. For example, it might generate phrases to help the user relax and incorporate them into the next dialogue. The input is the result of the emotional assessment, and the output is the adjusted dialogue.

[0933] Step 8:

[0934] The server integrates user responses and emotional data and runs an algorithm to evaluate suitability. The evaluation results are compiled into a report and provided to companies and users. The input is the user responses and emotional data, and the output is the report.

[0935] (Application Example 2)

[0936] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0937] In automated interview systems, implementing an authentication process that considers the user's emotional state is crucial for reducing the user's psychological burden and providing a less stressful environment. Furthermore, given the current need for fairer and more accurate evaluations through the appropriate use of emotional assessment, the introduction of new payment systems is required. However, existing technologies have the challenge of not being able to adequately consider the user's emotional state and thus not contributing to the smooth operation of the authentication process.

[0938] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0939] In this invention, the server includes a conversation generation means that utilizes intelligence for conducting automated interviews, a means for collecting input information using sensor devices that collect audio and video, and an optimization means that utilizes a relaxing effect in payment authentication based on the user's emotional state. This makes it possible to provide a payment authentication process with a relaxing effect that takes the user's emotional state into consideration.

[0940] "Automated interviews" are a process that utilizes artificial intelligence to automatically ask users questions and analyze their responses.

[0941] "Intelligence" refers to the ability to make judgments and take actions appropriate to a situation based on information processed by a computer.

[0942] A "conversation generation method" is a function in which the system spontaneously creates and presents questions to the user.

[0943] "Audio and video" refers to auditory and visual information obtained from the user, which is captured through sensor devices.

[0944] "Sensor equipment" refers to devices and equipment used to acquire audio and video information.

[0945] "Input information" refers to data collected for system processing, including audio, video, and other related data.

[0946] "Speech recognition means" refers to the technology or process that analyzes acquired speech data and converts it into text information.

[0947] "Character data" refers to text information converted from speech by speech recognition technology.

[0948] "Facial expression data" refers to information obtained by measuring a user's facial expressions and converting their characteristics into digital data.

[0949] "Emotion" refers to the user's instantaneous feelings and psychological state.

[0950] "Means of analyzing emotions" refers to methods and techniques for identifying and evaluating a user's emotional state.

[0951] "Persons being evaluated" refers to individuals whose fitness and other characteristics are assessed by the system.

[0952] "Organization" refers to any group or company that seeks to verify the suitability of the person being evaluated.

[0953] "Fit" is a measure that indicates the degree to which the person being evaluated matches the characteristics required by the organization.

[0954] "Evaluation means" refers to devices or systems used to evaluate the characteristics of a subject based on collected data and to determine their suitability.

[0955] "Optimization methods" refer to ways of adjusting conditions and parameters to improve the efficiency of a process or system.

[0956] "Relaxation effect" refers to the psychological or physical influence that alleviates tension in the subject.

[0957] "Information" is a general term for data that a system or process uses for processing and analysis.

[0958] A "report" refers to a written or electronic format in which evaluation results and analysis content are organized.

[0959] "Output" refers to the act or format in which a system displays or provides the results of its processing.

[0960] The system realizing this invention consists of a server, a terminal, and a user. The server requires an internet connection and functions as a platform for running programs that include an intelligence and emotion engine. The terminal functions as a sensor device that uses a camera and microphone to collect the user's voice and video. The user faces the terminal and participates in an automated interview and authentication process using voice and video.

[0961] The server uses intelligence to execute conversation generation mechanisms and generate questions to present to the user. This process employs a generative AI model. The terminal collects audio and video in real time and sends this data to the server. The server uses speech recognition mechanisms to convert the audio data into text and further evaluates the user's emotional state using emotion analysis mechanisms. This analysis utilizes facial expression data and voice tone, employing software such as TensorFlow and OpenCV.

[0962] Based on this data, the server applies optimization measures to provide users with a relaxing experience, such as playing light background music to alleviate their tension. Finally, the server integrates this information, generates a fitness report, and outputs it to the organization.

[0963] As a concrete example, in the authentication process for electronic payments, if a user is feeling nervous in front of the payment terminal, the server will assess the situation based on the results of emotional analysis and generate and present a prompt to the user that reduces their psychological burden through light conversation such as, "The weather is lovely today, it's a perfect day for shopping."

[0964] Example of a prompt:

[0965] "Explain the mechanism for user authentication using facial recognition and voice analysis. Explain, using examples, how the user is provided with a relaxed state by the system, including specific processing steps."

[0966] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0967] Step 1:

[0968] The device activates its camera and microphone to acquire user audio and video data. This data is temporarily stored on the device and prepared for transmission to the server. The input is the user's video and audio, and the output is raw data. Specifically, the user speaks into the device, the camera captures the video, and the microphone records the audio.

[0969] Step 2:

[0970] The terminal transmits the acquired audio and video data to the server. The terminal compresses the data to reduce the transfer time. The input is the compressed audio and video data, and the output is the data received by the server. Specifically, the terminal uses its communication module to send the data to the server via the internet.

[0971] Step 3:

[0972] The server converts the received audio data into text data using speech recognition technology. This process utilizes the Google Cloud Speech-to-Text API. The input is audio data, and the output is text data. Specifically, the server sends audio data to the API and saves the returned text to an internal database.

[0973] Step 4:

[0974] The server extracts facial expression data from video data and analyzes emotions. It uses TensorFlow to perform facial expression analysis and estimate the user's emotional state. The input is video data, and the output is emotion evaluation data. Specifically, it detects facial feature points from the video and evaluates emotions using a pre-trained model.

[0975] Step 5:

[0976] The server generates optimal dialogue to induce relaxation based on the user's emotional state. This utilizes a generative AI model. The input is emotional assessment data, and the output is a prompt designed to promote relaxation. Specifically, the generative AI creates the prompt and prepares it to be sent to the user's device.

[0977] Step 6:

[0978] The server sends a generated prompt to the terminal, which then presents it to the user via voice or on-screen display. The input is the prompt, and the output is the information presented to the user. Specifically, the terminal either plays the text using speech synthesis or displays the text on the screen.

[0979] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0980] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0981] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0982] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0983] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0984] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0985] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0986] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0987] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0988] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0989] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0990] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0991] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0992] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0993] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0994] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0995] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0996] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0997] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0998] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0999] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1000] The following is further disclosed regarding the embodiments described above.

[1001] (Claim 1)

[1002] A conversation generation method utilizing artificial intelligence for automated interviews,

[1003] A means for collecting input information using a sensor device that captures audio and video,

[1004] A speech recognition means that converts collected audio data into text data,

[1005] A method for analyzing emotions from text data and facial expression data,

[1006] A method for evaluating the suitability of applicants and companies based on analyzed emotions and responses,

[1007] A means of generating and outputting this data as a report,

[1008] A system that includes this.

[1009] (Claim 2)

[1010] The system according to claim 1, which uses an algorithm to compare pre-set criteria with analyzed data in order to evaluate the degree of fit.

[1011] (Claim 3)

[1012] The system according to claim 1, wherein an audio and video sensor device collects data in real time and dynamically evaluates the emotional state based on this data.

[1013] "Example 1"

[1014] (Claim 1)

[1015] A question-and-answer generation method utilizing an intelligent agent for conducting automated interviews,

[1016] A means for collecting input information using a sensing device that acquires acoustic data and visual data,

[1017] A recognition means for converting collected audio information into text information,

[1018] A means of inferring emotions from textual information and facial change information,

[1019] A method for quantifying the suitability of a candidate to an organization based on inferred emotions and responses,

[1020] Means for generating and displaying this information as a report,

[1021] A method for automatically scheduling interviews,

[1022] A means of analyzing emotional states and logical thinking ability,

[1023] A system that includes this.

[1024] (Claim 2)

[1025] The system according to claim 1, which uses a calculation method that compares pre-set standards with analyzed information in order to quantify the degree of fit.

[1026] (Claim 3)

[1027] The system according to claim 1, wherein an acoustic and visual sensing device instantly collects information and dynamically evaluates the emotional state based on this information.

[1028] "Application Example 1"

[1029] (Claim 1)

[1030] A question-and-answer method based on conversation generation,

[1031] A means for acquiring information using an input device that records audio and video,

[1032] A recognition means for converting audio information into text information,

[1033] A means for analyzing emotions from textual information and facial expression information,

[1034] A means of evaluating the alignment between applicants and the organization based on analyzed emotions and response content,

[1035] A means of continuously acquiring information on smart devices,

[1036] A means of generating and outputting the analyzed data as a report with feedback,

[1037] A system that includes this.

[1038] (Claim 2)

[1039] The system according to claim 1, which uses a calculation method that references pre-set criteria and analyzed information in order to evaluate consistency.

[1040] (Claim 3)

[1041] The system according to claim 1, wherein an audio and video input device dynamically acquires data and evaluates the emotional state based on this data.

[1042] "Example 2 of combining an emotion engine"

[1043] (Claim 1)

[1044] A dialogue generation method utilizing artificial intelligence for automated interviews,

[1045] A means for collecting data using a sensor device for acquiring sound and video,

[1046] A speech recognition means that converts collected audio information into text information,

[1047] A means of analyzing emotions from textual information and facial expression information,

[1048] A means of evaluating the suitability of job seekers and organizations based on analyzed emotions and responses,

[1049] A means of generating and outputting a report based on the assessment of suitability,

[1050] A control means that adjusts the content and tone of the response using a generative AI model,

[1051] A system that includes this.

[1052] (Claim 2)

[1053] The system according to claim 1, which uses an algorithm that compares pre-set criteria with analyzed information in order to evaluate suitability.

[1054] (Claim 3)

[1055] The system according to claim 1, wherein voice and video sensor devices dynamically collect information and evaluate emotional states in real time based on the information.

[1056] "Application example 2 when combining with an emotional engine"

[1057] (Claim 1)

[1058] An intelligent conversation generation method for conducting automated interviews,

[1059] A means for collecting input information using sensor equipment that collects audio and video,

[1060] A speech recognition means that converts collected speech data into text data,

[1061] A means for analyzing emotions from text data and facial expression data,

[1062] A means of evaluating the degree of fit between the evaluator and the organization based on the analyzed emotions and responses,

[1063] An optimization method that utilizes relaxation effects in payment authentication based on the user's emotional state,

[1064] A means of generating and outputting this information as a report,

[1065] A system that includes this.

[1066] (Claim 2)

[1067] The system according to claim 1, which uses a calculation procedure to compare pre-set criteria with analyzed information in order to evaluate the degree of fit.

[1068] (Claim 3)

[1069] The system according to claim 1, wherein audio and video sensor devices collect information in real time and dynamically evaluate emotional states based on this information. [Explanation of symbols]

[1070] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A conversation generation method utilizing artificial intelligence for automated interviews, A means for collecting input information using a sensor device that captures audio and video, A speech recognition means that converts collected audio data into text data, A method for analyzing emotions from text data and facial expression data, A method for evaluating the suitability of applicants and companies based on analyzed emotions and responses, A means of generating and outputting this data as a report, A system that includes this.

2. The system according to claim 1, which uses an algorithm to compare pre-set criteria with analyzed data in order to evaluate the degree of fit.

3. The system according to claim 1, wherein an audio and video sensor device collects data in real time and dynamically evaluates the emotional state based on this data.

Images