AI interview method and system based on multi-modal large model

Abstract

The invention discloses an AI interview method and system based on a multi-modal large model. The method comprises the steps that 1, an interview problem is automatically generated through a problem generation module; 2, in the process that the user answers the question, recording the voice and video of the user through an interview recording module, and sending the voice and video to an answer analysis module; 3, an answer analysis module analyzes the answer of the user to the interview question, and quantitatively scores the question answer condition of the user based on the scoring dimension; step 4, performing comprehensive analysis on the interview process by a scoring and questioning generation module, and generating questioning questions; and 5, performing comprehensive score calculation according to all questions answered by the user, generating an interview report, and sending the interview report to the HR. The invention relates to the technical field of artificial intelligence, and can solve the existing technical problems.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AI interview method and system based on a multimodal large model. Background Technology

[0002] In recent years, the rapid development of artificial intelligence (AI) technology has been profoundly changing the way various industries operate. Especially driven by cutting-edge technologies such as big data, cloud computing, and deep learning, AI can now efficiently process and analyze complex voice, image, and text information. These breakthroughs have not only improved production efficiency and the level of intelligent decision-making but have also brought new changes to enterprise human resource management. The rapid advancement of AI has led to the gradual formation of application scenarios such as intelligent recruitment and automated interviews. AI interviews, as an emerging interview method, rely on the integration of various AI technologies such as speech recognition, natural language processing, and sentiment analysis to automatically assess the abilities and qualities of interviewees, greatly improving the intelligence and automation of the recruitment process and saving companies significant human and time costs.

[0003] Currently, research on AI interview systems based on deep learning mainly focuses on the following areas:

[0004] A. An AI evaluation system based on intent recognition and semantic understanding of text or structured data.

[0005] B. A high-precision automatic interview question generation method based on self-distillation and industry knowledge base.

[0006] C. AI-powered interviewer video stream based on generative models.

[0007] D. Intelligent dialogue system based on multimodal large model.

[0008] In existing AI interview processes, questions are asked by constructing models based on preset discourse tags, industry knowledge bases, or self-distillation mechanisms. After the answers are completed, the recorded interviewee text, answer content, audio, and video information are analyzed by various evaluation element sub-models to achieve automated analysis and evaluation of the interview. While these methods can improve the intelligence level of the interview process to some extent and achieve functions such as personalized question generation, interview intent recognition, and interview scoring, they mainly rely on static tag libraries, knowledge bases, or example discourse sets. Moreover, they are reactive after the interview and cannot effectively cope with the complex and diverse real expressions in the interview setting. In addition, existing multimodal large-model intelligent dialogue systems construct user digital avatars and achieve high-fidelity dialogue interaction through multimodal fusion. Their technical framework revolves around "user avatar cloning" and cannot conduct objective job ability assessment. Their core technology relies on traditional GANs (non-large-model architectures), and the core capability of GANs is "generating samples" rather than "understanding dialogue logic," which does not match the core requirements of "intelligent dialogue" (semantic understanding and dynamic interaction).

[0009] Existing technologies have shortcomings in areas such as multimodal feature fusion, dynamic interactive feedback, and comprehensive assessment of job capabilities, making it difficult to fully reflect the comprehensive qualities and true abilities of interviewees in real-world communication environments. Therefore, while existing technologies can improve the automation level of AI interviews, they still have certain deficiencies in terms of adaptability, comprehensiveness of assessment, and fairness of judgment.

[0010] Existing AI interview-related technical solutions share some common technical drawbacks in practical applications, including:

[0011] 1) Limited adaptability to different scenarios: Existing intent recognition and answer generation methods usually rely on preset tags, example libraries or knowledge bases, which cannot provide personalized questions based on the candidate's personal learning and work experience. They are unable to cope with the diverse, open, innovative and complex expression scenarios in actual interviews, and their adaptability is insufficient.

[0012] 2) Insufficient feature fusion: Most solutions are based on text or structured data for analysis, or use independent sub-models to process data of different modalities separately, failing to effectively integrate multimodal information such as voice, facial expressions, and actions, resulting in insufficient accuracy and robustness in assessing the comprehensive abilities of interviewees.

[0013] 3) Weak data dependence and generalization ability: Models such as intent recognition and question generation are highly dependent on data quality and coverage. If the industry knowledge base or training samples are incomplete or outdated, it will easily affect the generalization and practicality of the system and make it difficult to continuously adapt to the ever-changing job requirements.

[0014] 4) Insufficient dynamic interaction and feedback: Some technical solutions focus on process automation and surface-level interactive experience, but lack intelligent analysis and real-time feedback on the dynamic behavioral details, in-depth abilities and on-the-spot reactions during the interview process, making it difficult to fully reflect the true level of the interviewee.

[0015] 5) Limited transparency in assessment and decision-making: The scores and assessment standards of AI assessment models are not transparent enough, which may lead to subjectivity and black box effects, affecting the fairness and explainability of recruitment decisions.

[0016] 6) System integration and security challenges: Existing solutions generally lack deep integration with subsequent recruitment processes, as well as security and privacy protection for sensitive information such as interview videos and audio, posing certain data risks and management difficulties.

[0017] The aforementioned technical shortcomings are common problems that current AI interview technology urgently needs to overcome in large-scale applications and actual recruitment processes.

[0018] The technical problems that existing AI interview technologies present in the recruitment of practical application talent include:

[0019] 1) Diverse backgrounds of interviewees: The educational backgrounds and work experiences of interviewees are extremely broad, including students from different majors as well as experienced professionals. A single question bank cannot cover all types, resulting in insufficient targeting and differentiation in the interview assessment.

[0020] 2) Frequent resume fraud issues: Resumes of interviewees are frequently found to be fraudulent. If questions are asked based solely on a professional question bank without considering the interviewee's resume, it is impossible to verify the authenticity of the resume, resulting in additional costs for the subsequent interview process.

[0021] 3) Poor timeliness of knowledge: Market conditions and professional skill requirements are constantly changing, and traditional knowledge bases and question banks are outdated and cannot reflect the latest industry trends and job requirements in a timely manner, affecting the effectiveness and foresight of interview results.

[0022] 4) Lack of dynamic follow-up questions: Existing AI interview solutions generally lack dynamic follow-up questioning mechanisms based on the interviewee's real-time performance, making it difficult to deeply explore the interviewee's comprehensive abilities and potential, resulting in limited depth and flexibility in interview assessment.

[0023] Therefore, there is a need to provide an AI interview method and system based on a multimodal large model that can solve the above-mentioned technical problems. Summary of the Invention

[0024] The purpose of this invention is to provide an AI interview method and system based on a multimodal large model, which can solve the above-mentioned technical problems.

[0025] This invention is implemented as follows:

[0026] An AI interview system based on a multimodal large model includes:

[0027] The user interaction module is used for playing interview questions, as well as for users to log in and register, upload resumes, and answer interview questions;

[0028] The question generation module is used to generate personalized interview questions;

[0029] The interview recording module is used to record the user's voice and video during the interview process;

[0030] The answer analysis module includes a voice analysis module and a video analysis module. The voice analysis module is used to analyze the user's voice, and the video analysis module is used to analyze the user's recorded video.

[0031] The scoring and follow-up question generation module is used to comprehensively score users' answers and generate targeted follow-up questions.

[0032] An AI interview method based on a multimodal large model AI interview system includes the following steps:

[0033] Step 1: Automatically generate interview questions using the question generation module;

[0034] Step 2: During the user's question-and-answer process, the interview recording module records the user's voice and video, and sends it to the answer analysis module;

[0035] Step 3: The answer analysis module analyzes the user's answers to the interview questions and quantifies the user's answer performance based on the scoring dimensions;

[0036] Step 4: The scoring and follow-up question generation module comprehensively analyzes the interview process and generates follow-up questions;

[0037] Step 5: Calculate a comprehensive score based on all the questions answered by the user, generate an interview report, and send it to HR.

[0038] Step 1 includes the following sub-steps:

[0039] Step 11: Input the preset question bank, job-specific knowledge base, and question dimensions in the question generation module;

[0040] Step 12: Users register and log in through the user interaction module and upload their personal information, including the interviewee's resume and job requirements;

[0041] Step 13: The question generation module automatically generates a personalized question bank based on the preset question bank, question dimensions, interviewee resume, and job requirements using question generation AI technology;

[0042] Step 14: The user starts the interview through the user interaction module. The user interaction module plays the questions, and the user answers the questions.

[0043] In step 11, the question dimensions, i.e. the question generation requirements, include things like job requirements, assessment dimensions, job competency dimensions, professional knowledge and skills, and work and learning experience.

[0044] The job-specific knowledge base is an enterprise knowledge base that is updated in real time based on the latest industry trends and professional skill requirements.

[0045] In step 12, the interviewee's resume includes educational background, project experience, and skill certificates. The NLP model extracts the above content from the interviewee's resume.

[0046] In step 2, the interview recording module sends the user's voice to the voice analysis module, and the user's video recording is sent to the video analysis module.

[0047] In step 3, the scoring dimensions include: accuracy of the answer, work and life attitude, fluency of the answer, tone of voice, mental state, and appearance.

[0048] Step 3 includes the following sub-steps:

[0049] Step 31: Based on the user's answer, the speech analysis module uses the ASR model to transcribe the audio file or audio stream into text in real time, and outputs the recognized text or text stream;

[0050] Step 32: Correct errors in the identified text by combining an NLP model with the question context;

[0051] Step 33: The voice analysis module scores the fluency, tone and emotion of the user's answer based on the voice, and analyzes the accuracy of the user's answer based on the corrected text, focusing on the quantitative scoring of key job competency indicators.

[0052] Step 34: The video analysis module collects user videos through visual information and quantifies and scores the user's mental state, appearance, demeanor, and facial expressions using multi-dimensional features.

[0053] Step 4 includes the following sub-steps:

[0054] Step 41: Take the corrected text, the interviewee's resume, the question context, the job-related professional knowledge base, and the score from Step 3 as input to the NLP model. The NLP model will analyze the content of the user's answer and output the interviewee's comprehensive evaluation and final quantitative score.

[0055] Step 42: Follow up with AI: Determine whether follow-up questions are needed. If yes, proceed to step 43; otherwise, proceed to step 44.

[0056] Step 43: Generate follow-up questions and play them through the user interaction module. The user answers the follow-up questions, then proceed to Step 2;

[0057] Step 44: Determine if there are any more interview questions. If yes, play the next interview question through the user interaction module, and the user answers the interview question, then proceed to Step 2. If no, end the interview.

[0058] In step 5, an interview report template is preset in the system. Based on the report template, the system summarizes multiple results of the comprehensive evaluation and quantitative scoring of the interview and automatically generates a standardized interview report.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] 1. This invention combines the interviewee's resume with a job-specific knowledge base, greatly improving the personalization and targeting of interview assessments, and can cover diverse educational backgrounds and work experiences.

[0061] 2. This invention ensures that the interview content is highly aligned with the latest industry developments and job requirements through a real-time updated professional knowledge base, a pre-set question bank, and a dynamic follow-up questioning mechanism.

[0062] 3. This invention effectively compensates for the limitations of single text or voice analysis through multimodal analysis of voice and video, and achieves accurate evaluation of the comprehensive qualities of interviewees.

[0063] 4. This invention automatically generates structured and standardized interview reports, reducing the workload of the human resources department and improving the efficiency of recruitment decisions.

[0064] 5. The system of this invention adopts a modular design, which facilitates practical deployment and flexible expansion, and can adapt to the diverse needs of different enterprises and positions. Attached Figure Description

[0065] Figure 1 This is a flowchart of the AI ​​interview method based on a multimodal large model according to the present invention. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0067] Please see the appendix Figure 1 An AI interview system based on a multimodal large model includes:

[0068] The user interaction module is used for playing interview questions, as well as for users (interviewees) to log in and register, upload resumes, and answer interview questions.

[0069] The question generation module is used to generate personalized interview questions.

[0070] The interview recording module is used to record the user's voice and video during the interview process.

[0071] The response analysis module includes a voice analysis module and a video analysis module. The voice analysis module is used to analyze the user's voice, and the video analysis module is used to analyze the user's recorded video.

[0072] The scoring and follow-up question generation module is used to comprehensively score users' answers and generate targeted follow-up questions.

[0073] Please see the appendix Figure 1 An AI interview method based on a multimodal large model includes the following steps:

[0074] Step 1: Automatically generate interview questions using the question generation module.

[0075] Step 1 includes the following sub-steps:

[0076] Step 11: Enter the preset question bank, job-specific knowledge base, and question dimensions in the question generation module.

[0077] The question dimensions mentioned above refer to the question generation requirements, which include prompts such as job requirements, assessment dimensions, job competency dimensions, professional knowledge and skills, and work and learning experience, to meet diverse, professional, and systematic interview questioning needs.

[0078] The aforementioned job-specific knowledge base can be an enterprise knowledge base, updated in real time based on the latest industry trends and professional skill requirements to ensure the system's generalizability and usability. Updating the knowledge base is a standard technical method in this field and will not be elaborated upon here.

[0079] Step 12: The user (interviewee) registers and logs in through the user interaction module and uploads personal information, including the interviewee's resume and job requirements.

[0080] The interviewee's resume includes educational background, project experience, skills certificates, etc. The NLP model can extract the above content from the interviewee's resume, so that the question generation module can generate more targeted personalized questions and effectively avoid resume fraud.

[0081] Step 13: The question generation module automatically generates a personalized question bank based on the preset question bank, question dimensions, interviewee resume and job requirements using question generation AI technology.

[0082] The question generation module uses an advanced NLP model to automatically generate interview questions. The NLP model expands and personalizes the question bank based on the input preset question bank and question dimensions, combined with RAG search technology. It associates the question bank with job competency dimensions and constructs a "resume entity-job competency" association graph. It outputs personalized interview questions that are highly relevant to the job and the candidate. This solves the technical problem in existing technologies of "unstructured resume information (such as free text description) → structured entity extraction → association with job graph" (such as solving the semantic alignment of "project management" in the resume with "progress control" in the job through entity linking technology).

[0083] The generation of the source tracing questions in this invention requires "precisely locating key information in the resume + designing contradiction verification points", and requires combining RAG search technology and contradiction detection algorithm. This is a technical path that cannot be covered by the existing "self-distillation + comparative learning" framework.

[0084] Step 14: The user starts the interview through the user interaction module. The user interaction module plays the questions, and the user answers the questions.

[0085] Step 2: During the user's question-answering process, the interview recording module records the user's voice and video, and sends it to the answer analysis module.

[0086] In step 2, the interview recording module sends the user's voice to the voice analysis module, and the user's video recording is sent to the video analysis module.

[0087] During the interview process, the system has audio and video capture capabilities, supporting complete recording or streaming of the entire interview to ensure that subsequent analysis has raw data support. All audio and video data are captured in real time, ensuring data continuity and integrity, and providing a foundation for multimodal analysis and evaluation.

[0088] Step 3: The answer analysis module analyzes the user's answers to the interview questions and quantifies the user's answer performance based on the scoring dimensions.

[0089] The scoring dimensions include: accuracy of answers, work and life attitude, fluency of answers, tone of voice, mental state, and appearance.

[0090] Step 3 includes the following sub-steps:

[0091] Step 31: Based on the user's answer, the speech analysis module uses an ASR model (such as OpenAI-Whisper or iFlytek) to transcribe the audio file or audio stream into text in real time, and outputs the recognized text or text stream.

[0092] The speech analysis module also analyzes the speech quality to ensure the continuity and integrity of real-time audio acquisition.

[0093] Step 32: By combining the NLP model with the question context, the identified text is corrected to improve the text accuracy.

[0094] Step 33: The voice analysis module scores the fluency of the user's response based on the voice (including fluctuations in speech rate, pause duration, etc.), tone and emotion of the response, and analyzes the accuracy of the user's response based on the corrected text (including the accuracy of professional terminology, such as whether the "concrete strength grade" for the construction position is correct), focusing on the quantitative scoring of key job competency indicators.

[0095] Step 34: The video analysis module collects user videos through visual information and quantifies and scores the user's mental state, appearance, facial expressions, and other multi-dimensional features.

[0096] The video analytics module outputs detailed analysis results and quantitative scores in the time dimension, effectively supplementing the shortcomings of traditional text and voice analysis, and comprehensively reflecting the non-verbal characteristics and on-site performance of interviewees.

[0097] The answer analysis module effectively integrates multimodal information such as voice, facial expressions, and actions, improving the accuracy and robustness of the system's subsequent assessment of the interviewee's comprehensive abilities and significantly enhancing the comprehensiveness and objectivity of the interview evaluation.

[0098] Step 4: The scoring and follow-up question generation module comprehensively analyzes the interview process and generates follow-up questions.

[0099] Step 4 includes the following sub-steps:

[0100] Step 41: Scoring AI: The corrected text, interviewee resume, question context, job-related knowledge base, and the score from Step 3 are used as input to the NLP model. The NLP model analyzes the content of the user's answer and outputs the interviewee's comprehensive evaluation and final quantitative score.

[0101] NLP models integrate multimodal information such as question context, interviewee resume, job-specific knowledge base, answer analysis and evaluation, fluency and tone of voice analysis, and demeanor and facial expression analysis to output structured interview conclusions.

[0102] When conducting comprehensive analysis and scoring, knowledge bases, prompts, RAG search, and agent configuration are used to enable customized configuration of quantitative weights for interview analysis, providing a basis for scoring, improving the accuracy of comprehensive scoring, meeting the personalized recruitment requirements of enterprises, and enhancing the fairness and interpretability of recruitment decisions.

[0103] Step 42: Follow up with AI: Determine whether follow-up questions are needed. If yes, proceed to step 43; otherwise, proceed to step 44.

[0104] The scoring and follow-up question generation module is equipped with a key follow-up question function. Specifically, based on the NLP model, it takes the context of the current question and the user's answer, the job knowledge base, the answer analysis and evaluation, and the follow-up question requirements as input. Based on the ability gap, it realizes intelligent guidance and dynamic follow-up questions, automatically generates targeted follow-up questions, further explores the interviewee's ability, and improves the depth of the interview through question and answer.

[0105] Step 43: Generate follow-up questions and play them through the user interaction module. The user answers the follow-up questions, and then proceed to Step 2.

[0106] Step 44: Determine if there are any more interview questions. If yes, play the next interview question through the user interaction module, and the user answers the interview question, then proceed to Step 2. If no, end the interview.

[0107] Step 5: Calculate a comprehensive score based on all the questions answered by the user, generate an interview report, and send it to HR (Human Resources).

[0108] In step 5, an interview report template is preset in the system. Based on the report template, the system summarizes multiple results such as comprehensive interview evaluation and quantitative scoring, and automatically generates a standardized interview report.

[0109] This standardized interview report has a clear structure and detailed content, making it easy for employers to quickly understand the comprehensive qualities of interviewees and providing a scientific basis for recruitment decisions.

[0110] This invention, through multi-module collaboration, covers aspects such as question generation, dynamic follow-up questioning, data collection, content and sentiment analysis, comprehensive evaluation, and report generation. It achieves full automation and intelligence in the AI ​​interview process, as well as comprehensive mining and multi-dimensional evaluation of interviewees' abilities. This effectively improves the scientific rigor and applicability of interviews and can effectively address many shortcomings encountered by companies when using AI to interview candidates. It also enhances the depth and accuracy of interviews and provides a fairer and more immersive interview experience.

[0111] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the invention. Therefore, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-modal large model-based AI interview system, characterized by: The system comprises: a user interaction module for playing interview questions, and user login registration, resume uploading and answering interview questions; a question generation module for generating personalized interview questions; an interview recording module for recording the user's voice and video during the interview; an answer analysis module, including a voice analysis module and a video analysis module, wherein the voice analysis module is used to analyze the user's voice, and the video analysis module is used to analyze the user's video; a scoring and follow-up question generation module for comprehensive scoring of the user's answers and generating targeted follow-up questions.

2. An AI interview method using the AI interview system based on the multi-modal large model according to claim 1, characterized in that: The system comprises the following steps: Step 1: automatically generate interview questions through the question generation module; Step 2: record the user's voice and video during the user's answering process through the interview recording module, and send them to the answer analysis module; Step 3: the answer analysis module analyzes the user's answers to the interview questions, and quantitatively scores the user's question answering based on the scoring dimensions; Step 4: the scoring and follow-up question generation module comprehensively analyzes the interview process and generates follow-up questions; Step 5: according to the comprehensive scoring calculation of all the questions answered by the user, generate an interview report and send it to the HR. 3.The AI interview method of claim 2, characterized in that: The step 1 comprises the following sub-steps: Step 11: input the preset question bank, post professional knowledge base and questioning dimension in the question generation module; Step 12: the user registers and logs in through the user interaction module, and uploads personal information, including the interviewer's resume and job requirements; Step 13: the question generation module automatically generates a personalized question bank based on the question generation AI technology according to the preset question bank, questioning dimension, interviewer's resume and job requirements; Step 14: the user starts the interview through the user interaction module, the user interaction module plays the questions, and the user answers the questions. 4.The AI interview method of claim 3, characterized in that: In step 11, the questioning dimension, i.e. the question generation requirement, includes, for example, job requirements, assessment dimensions, post ability dimensions, professional knowledge and skills, work and study experience; The post professional knowledge base is an enterprise knowledge base, which is updated in real time based on the latest industry trends and professional skill requirements.

5. The AI interview method of claim 3, wherein: In step 12, the interviewer's resume includes education experience, project experience and skill certificates, and the NLP model extracts the above-mentioned content from the interviewer's resume.

6. The AI interview method of claim 2, wherein: In step 2, the interview recording module sends the user's voice to the voice analysis module and the user's video to the video analysis module.

7. The AI interview method of claim 2, wherein: In step 3, the scoring dimensions include: answer accuracy, work and life attitude, answer fluency, answer tone, spirit, appearance and demeanor. 8.The AI interview method of claim 2, characterized in that: The step 3 comprises the following sub-steps: Step 31: for the user's answer content, the voice analysis module converts the audio file or audio stream into text in real time through the ASR model, and outputs the recognized text or text stream; Step 32: correct the recognized text through the NLP model combined with the question context; Step 33: the voice analysis module scores the user's answer fluency, answer tone and emotion based on the voice, and analyzes the user's answer accuracy based on the corrected text, focusing on the quantitative scoring of key post ability indicators; Step 34: The video analysis module collects the user's video through visual information, and quantitatively scores the user's spiritual appearance, image demeanor, and facial expression. 9.The AI interview method of claim 2, characterized in that: The step 4 includes the following sub-steps: Step 41: The corrected text, the interviewer's resume, the question context, the post professional knowledge base, and the score in step 3 are used as inputs of the NLP model, and the NLP model analyzes the answer content of the user's answer, and outputs the comprehensive evaluation and the final quantitative score of the interviewer; Step 42: Ask AI: Determine whether to ask, if yes, execute step 43, if no, execute step 44; Step 43: Generate a follow-up question, and play the follow-up question through the user interaction module, and the user answers the follow-up question, and go to step 2; Step 44: Determine whether there are still interview questions, if yes, play the next interview question through the user interaction module, the user answers the interview question, and go to step 2, if no, end the interview.

10. The AI interview method of claim 2, wherein: In step 5, the system presets an interview report template, the system collects the multiple results of the interview comprehensive evaluation and the quantitative score based on the report template, and automatically generates a standardized interview report.

Images