An artificial intelligence interview evaluation method and system based on LLM, and a storage medium

Abstract

The application discloses an artificial intelligence interview evaluation method and system based on an LLM and a storage medium, and relates to the technical field of artificial intelligence.The application constructs an evaluation standard knowledge base marked and divided, adopts an RAG retrieval strategy to obtain evaluation standards corresponding to the current investigation angle, uses a large language model to perform basic evaluation on user answers to obtain a basic score, and performs additional score evaluation based on the evaluation standards of other topics when other topics exist, and generates an evaluation result including a total score, a score summary, advantages and disadvantages, angle score reasons and additional score reasons by comprehensively combining the basic score and the additional score, and finally generates an interview evaluation report.The application also provides a corresponding system, an electronic device and a storage medium.The application solves the problems of inaccurate evaluation standard retrieval, single evaluation dimension and poor interpretability in the prior art, significantly improves the retrieval accuracy, realizes cross-topic comprehensive evaluation, and enhances the interpretability and comprehensiveness of the evaluation result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AI interview evaluation method, system, and storage medium based on LLM. Background Technology

[0002] Large Language Model (LLM) technology is in a period of rapid development and has gradually mastered the logical thinking ability to handle complex problems, bringing great convenience to people's work and life, and has great development potential. Large LLM models with complex logical thinking capabilities can effectively handle tasks requiring extensive logical reasoning, such as interviews and assessments.

[0003] In the past, interviews and surveys across various industries often required multiple people to participate in the entire process from interview to evaluation for each interviewee. This resulted in multiple people being able to evaluate the interview results of only one interviewee at a time, leading to low interview efficiency. Furthermore, since evaluating interview results requires a great deal of logical thinking, it demands a significant amount of mental labor from the evaluators, greatly increasing their workload.

[0004] In recent years, researchers have begun exploring the use of large language models (LLMs) to automate interviews and assessments. Existing technologies include frameworks that utilize LLMs as interviewers for multi-round interactive assessments (e.g., LLM-as-an-Interviewer) and methods that use LLMs for multi-dimensional text evaluation (e.g., the PatentScore framework). These technologies have improved the efficiency of interview assessments to some extent. However, existing technologies still have the following shortcomings and limitations:

[0005] (1) Inaccurate retrieval of evaluation criteria. When using Retrieval-Augmented Generation (RAG) technology to retrieve evaluation criteria, the semantic similarity of evaluation criteria from different topics and perspectives may lead to difficulties in similarity matching. As a result, the search results often contain criteria that are not relevant to the current perspective, which affects the accuracy of the evaluation.

[0006] (2) The assessment dimensions are too narrow and lack comprehensive consideration across topics. Existing assessment methods usually only score based on the pre-set standards of the current topic and fail to make full use of the strengths shown by the interviewee in other topics. For example, when assessing the topic of "communication skills", it is not possible to capture the cross-dimensional strengths shown by the interviewee in the topic of "professional skills", resulting in an incomplete assessment result.

[0007] (3) Insufficient interpretability of evaluation results. Many automated evaluation methods only output a score or grade, lacking detailed reasons for the score, analysis of advantages and disadvantages, and evaluation summary, making it difficult for users to understand and trust the evaluation results.

[0008] (4) Poor interview process flexibility. Most existing automated interview systems require users to answer questions in a strict order and do not support skipping questions or answering them again, which reduces user experience and interview adaptability.

[0009] (5) Strong model dependency. Some solutions require fine-tuning of the LLM for specific evaluation tasks, resulting in high training costs and difficulty in transferring to interview scenarios in different fields.

[0010] Therefore, there is an urgent need for an LLM-based AI interview evaluation method that can solve the above problems, improve the accuracy, comprehensiveness, interpretability and flexibility of interview evaluation, and reduce the dependence on model fine-tuning. Summary of the Invention

[0011] The technical problem to be solved by the present invention is: in order to overcome the above-mentioned technical problems, the present invention provides an AI interview evaluation method, system and storage medium based on LLM.

[0012] The technical solution adopted by this invention to solve its technical problem is: an AI interview evaluation method based on LLM, comprising the following steps:

[0013] Step S1: Construct an evaluation standard knowledge base. The evaluation standard knowledge base is generated by labeling and dividing the evaluation standards for multiple topics. Each topic contains multiple assessment angles, and each assessment angle corresponds to multiple rating levels (such as unqualified, qualified, good, and excellent). Each rating level is associated with the corresponding evaluation requirements. The labeling and division includes at least a three-layer structure: topic dimension, assessment angle dimension, and rating level dimension.

[0014] Step S2: Obtain the interview and Q&A records of users for the preset question list, and store them by topic.

[0015] Step S3: For the current topic, retrieve the evaluation criteria corresponding to the current perspective of this topic from the evaluation criteria knowledge base.

[0016] Step S4: Based on the retrieved evaluation criteria, use a large language model to perform a basic evaluation of the question and answer records for the current topic, and obtain the basic score and basic scoring reasons for the current topic.

[0017] Step S5: When there is at least one other topic besides the current topic, based on the evaluation criteria of the other topic, use the large language model to evaluate the question and answer records of the current topic with additional points, and obtain the additional points and the reasons for the additional points.

[0018] Step S6: Based on the base score, the reason for the base score, the bonus score, and the reason for the bonus score, generate the evaluation result for the current topic.

[0019] Step S7: Summarize the evaluation results of all topics and generate an interview evaluation report.

[0020] The retrieval in step S3 employs Retrieval Enhancement Generation (RAG) technology, specifically including:

[0021] The first retrieval stage: using the current perspective of the current topic as input, retrieve the evaluation criteria for all rating levels under that perspective, so that the large language model can determine the rating level to which the user's answer belongs;

[0022] The second retrieval stage involves using the assessment angle and rating level determined by the large language model as input to retrieve the specific evaluation requirements for that rating level, which are then used by the large language model for scoring.

[0023] The specific implementation method of the basic assessment in step S4 is as follows:

[0024] For each perspective of the current topic, the large language model is used to score the user's answer based on the evaluation criteria of that perspective, and the scores and reasons for each perspective are obtained. The scores of each perspective are then weighted and summed according to preset weights to obtain the base score for the current topic.

[0025] The specific implementation method of the bonus point assessment in step S5 is as follows:

[0026] S51. Calculate the difference between the highest score in the current topic's base score rating level and the base score, and use this as the total bonus score.

[0027] S52. Divide the total bonus points equally according to the number of other topics to obtain the bonus points amount for each other topic;

[0028] S53. For each other topic, the additional score is divided equally according to the number of perspectives for that topic, so as to obtain the additional score for each perspective.

[0029] S54. For each examination angle of each other topic, use the large language model to score the user's answer to the current topic based on the evaluation criteria of that examination angle. If the score reaches or exceeds the preset good threshold, calculate the actual bonus score for that examination angle: bonus score amount for that examination angle × (actual score / 100); otherwise, the bonus score for that examination angle is 0.

[0030] S55. Add up the actual bonus points for all other topics from all perspectives to get the bonus points for the current topic.

[0031] The current topic evaluation results generated in step S6 include: the total score of the current topic, the scoring summary, the advantages and disadvantages of the current topic, the basic score of the current topic, the basic scoring reasons for each evaluation angle of the current topic, the additional scores obtained from other topics and the scoring reasons for each additional score;

[0032] The advantages and disadvantages are based on the basic scoring reasons from various perspectives of the current topic, which are generated by a large language model. The scoring summary is based on the basic scoring reasons and the additional scoring reasons, which are also generated by a large language model.

[0033] The content and order of questions in the preset question list are fixed. During the interview, the large language model does not modify the content of any questions or generate new questions. The large language model is a general-purpose large language model and does not require fine-tuning for the interview evaluation task.

[0034] The present invention also provides an AI-based interview assessment system based on LLM, comprising: a knowledge base construction module, a data acquisition module, a retrieval module, a basic assessment module, an additional score assessment module, a topic result generation module, and a report generation module.

[0035] The knowledge base construction module is used to build an evaluation standard knowledge base. The evaluation standard knowledge base is generated by labeling and dividing the evaluation standards of multiple topics. Each topic contains multiple examination angles, each examination angle corresponds to multiple rating levels, and each rating level is associated with the corresponding evaluation requirements. The labeling and division includes at least a three-layer structure: topic dimension, examination angle dimension, and rating level dimension.

[0036] The data acquisition module is used to obtain the interview and question-and-answer records of users for a preset list of questions, and store them by topic.

[0037] The retrieval module is used to retrieve, for the current topic, evaluation criteria corresponding to the current perspective of the topic from the evaluation criteria knowledge base.

[0038] The basic assessment module is used to perform a basic assessment of the question and answer records of the current topic based on the retrieved assessment criteria and using a large language model, so as to obtain the basic score and basic scoring reasons for the current topic.

[0039] The bonus score evaluation module is used to evaluate the question and answer records of the current topic using a large language model based on the evaluation criteria of the other topics when there are at least one other topic besides the current topic, so as to obtain the bonus score and the reason for the bonus score.

[0040] The topic result generation module is used to generate the evaluation result of the current topic based on the base score, the reason for the base score, the bonus score, and the reason for the bonus score.

[0041] The report generation module is used to summarize the evaluation results of all topics and generate interview evaluation reports.

[0042] The retrieval module is further configured as follows:

[0043] The first retrieval unit is used to retrieve the evaluation criteria for all rating levels under the current perspective of the current topic as input, so that the large language model can determine the rating level of the user's answer.

[0044] The second retrieval unit is used to retrieve the specific evaluation requirements for the corresponding level based on the rating level determined by the large language model, using the evaluation perspective and rating level as input, for the large language model to score.

[0045] The bonus score assessment module is further configured as follows:

[0046] Calculate the difference between the highest score in the current topic's base score rating level and the base score, and use this difference as the total bonus score.

[0047] The total bonus points are divided equally among the number of other topics to obtain the bonus point amount for each other topic.

[0048] For each other topic, its bonus points are divided equally according to the number of perspectives on that topic, resulting in the bonus points for each perspective.

[0049] For each assessment angle of each other topic, the large language model is used to score the user's answer to the current topic based on the evaluation criteria of that assessment angle. If the score reaches or exceeds the preset good threshold, the actual bonus score for that assessment angle is calculated as: bonus score amount for that assessment angle × (actual score / 100); otherwise, the bonus score for that assessment angle is 0.

[0050] The bonus score for the current topic is obtained by summing up the actual bonus scores from all other topics and all perspectives.

[0051] The current topic evaluation results generated by the topic result generation module include: the total score of the current topic, the scoring summary, the advantages and disadvantages of the current topic, the base score of the current topic, the basic scoring reasons for each evaluation angle of the current topic, the additional scores obtained from other topics, and the scoring reasons for the additional scores; wherein, the advantages and disadvantages are generated by the large language model based on the basic scoring reasons based on the evaluation angle of the current topic, and the scoring summary is generated by the large language model based on the basic scoring reasons and the scoring reasons for the additional scores.

[0052] The data acquisition module is configured as follows:

[0053] Ask questions according to a pre-set fixed list and question order, without modifying the question content or generating new questions;

[0054] The basic assessment module and the bonus assessment module both use the general large language model, and this model has not been fine-tuned.

[0055] The present invention also provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program executable on the processor, and the electronic device implements the method described in the present invention when executing the program.

[0056] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the present invention.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. By dividing the evaluation criteria into three levels of annotation—topic, angle, and level—and adopting a two-step RAG retrieval strategy: first, all level criteria are retrieved to determine the level, and then the specific requirements of the corresponding level are retrieved. This increases the retrieval accuracy from approximately 49% to 99.88% for RAG retrieval methods that have never used the annotation division of this invention and only use pure vector similarity matching, effectively solving the problem of semantic similarity matching difficulties.

[0059] 2. It uniquely introduces a bonus point mechanism, scoring responses to the current topic from the perspective of other topics. This captures the interviewee's outstanding performance across multiple dimensions, making the evaluation results more comprehensive and objective. At the same time, the bonus points do not affect the base grade of the current topic, ensuring the rationality and fairness of the evaluation.

[0060] 3. Each rating item comes with detailed rating reasons generated by the large model, and automatically generates an analysis of advantages and disadvantages and a rating summary, so that users can clearly understand the evaluation basis and improve the credibility and practical value of the evaluation results.

[0061] 4. Professional evaluation can be completed by directly using a general-purpose large language model, combined with RAG technology and annotated evaluation standard library. There is no need to train or fine-tune the model for specific tasks. It can be quickly adapted to various scenarios such as personnel interviews, medical satisfaction surveys, and education and training feedback.

[0062] 5. Supports users to skip questions or answer them again, improving the interactive experience; at the same time, it strictly follows the content and order of the preset question list to ensure the consistency and comparability of the assessment and meet the needs of standardized interview scenarios.

[0063] 6. From interviews and dialogues, record storage, multi-module evaluation to report generation, the entire process is automated, greatly reducing manual intervention and lowering labor costs and mental burden. Attached Figure Description

[0064] Figure 1 This is an overall flowchart of the LLM-based artificial intelligence interview evaluation method in this embodiment of the invention.

[0065] Figure 2 This is a detailed flowchart of RAG retrieval in an embodiment of the present invention.

[0066] Figure 3 This is a flowchart illustrating the basic assessment and bonus score assessment process for a single topic in an embodiment of the present invention.

[0067] Figure 4 This is a module architecture diagram of the LLM-based artificial intelligence interview evaluation system in an embodiment of the present invention.

[0068] Figure 5 This is a schematic diagram of the structure of the evaluation standard knowledge base in an embodiment of the present invention.

[0069] Figure 6 This is a schematic diagram of the data structure for the evaluation results in an embodiment of the present invention.

[0070] Figure 7 This is a comparison chart of retrieval accuracy using labeled partitioning and not using labeled partitioning in an embodiment of the present invention; in the figure, (a) is labeled partitioning and (b) is unlabeled partitioning. Detailed Implementation

[0071] The invention will now be described in further detail with reference to the accompanying drawings. It should be emphasized that the following description is merely exemplary and not intended to limit the scope or application of the invention.

[0072] Example 1

[0073] This embodiment is based on the Dify platform to develop an intelligent interview evaluation process. The Dify platform provides visual workflow orchestration, RAG retrieval integration, and LLM invocation capabilities, enabling convenient implementation of the multi-module evaluation process of this invention. Of course, other platforms or frameworks with similar functions can also be used to implement this invention.

[0074] This embodiment provides an AI-based interview evaluation method based on LLM, such as... Figure 1 As shown, it includes the following steps.

[0075] Step S1: Construct an evaluation criteria knowledge base

[0076] First, an evaluation criteria knowledge base is constructed. This knowledge base is generated by labeling and dividing the evaluation criteria for multiple topics. Each topic contains multiple assessment angles, and each assessment angle corresponds to multiple rating levels, such as: unsatisfactory, satisfactory, good, and excellent. Each rating level is associated with the corresponding evaluation requirement text.

[0077] The labeling is structured in three layers: the first layer is the topic dimension, the second layer is the assessment angle dimension, and the third layer is the rating level dimension. For example, in an employee interview scenario, the topic "communication ability" includes assessment angles such as "clarity of expression" and "logical coherence"; each angle is set with four levels: "unsatisfactory", "satisfactory", "good" and "excellent", and each level has specific evaluation requirements, such as "excellent" requiring "fluent language, rigorous logic, and ability to effectively guide the conversation".

[0078] like Figure 5 As shown, the evaluation criteria knowledge base is stored in a tree structure for easy retrieval later.

[0079] Step S2: Obtain the interview Q&A record

[0080] The system retrieves user interview and Q&A records for a preset question list and stores them by topic. The content and order of the questions in the preset question list are fixed; the large language model does not modify any question content or generate new questions during the interview. Simultaneously, the system allows users to skip the current question or re-answer the previous question, and the large language model adjusts the questioning flow according to user instructions.

[0081] The Q&A records are stored in JSON format, with each topic corresponding to an array. Each element in the array contains the question text, the user's answer, and a timestamp. When a user requests to skip the current question, the answer record for that question is empty, the Q&A process is saved to the current topic's Q&A record, and the user continues to the next question. For skipped questions, the corresponding assessment angle does not participate in the calculation of the base score and bonus score; the weight of that angle is proportionally distributed to other angles under the same topic, or the angle's score is 0. When a user requests to answer the previous question again, the system allows the user to re-enter the answer and records the Q&A process to the current topic's Q&A record. Furthermore, the interview evaluation assesses the Q&A records for all topics. The interview evaluation is only triggered after the Q&A records for each topic have been completed. The interview evaluation will assess the Q&A records for each topic sequentially, and finally, the evaluation results for all topics are summarized to generate an evaluation report.

[0082] Step S3: Retrieve Evaluation Criteria

[0083] For the current topic, retrieve the evaluation criteria corresponding to the current perspective of this topic from the evaluation criteria knowledge base. For example... Figure 2 As shown, the retrieval employs search enhancement generation (RAG) technology, which specifically includes two stages:

[0084] The first retrieval stage: Taking the current evaluation angle of the current topic, such as "clarity of expression", as input, retrieve the evaluation criteria for all rating levels (unsatisfactory, satisfactory, good, excellent) under this evaluation angle, and return the results to the large language model.

[0085] The second retrieval stage: Based on all the rating criteria returned in the first retrieval stage, the large language model determines the rating level of the user's answer, such as "good". Then, using this evaluation perspective and the determined rating level as input, it retrieves the specific evaluation requirements for the corresponding level, such as the specific descriptive text for the "good" level, for the large language model to perform accurate scoring.

[0086] This two-step retrieval mechanism significantly improves the accuracy of evaluation criterion matching. Experiments show that after using the annotation partitioning and two-step RAG retrieval of this invention, the retrieval accuracy rate can reach 99.88%, while it is only 49.15% without annotation partitioning. See [link to relevant documentation]. Figure 7 .

[0087] Those skilled in the art can employ any known RAG implementation, such as using the Chroma vector database and OpenAI's embedding model, to calculate vector cosine similarity for retrieval. This invention does not limit the specific RAG implementation technology.

[0088] This invention employs a common large language model API call method. Each time it is called, the system concatenates the prompt words and user input into a message list, calls the chat interface, and sets `temperature=0.2` to ensure stable output. The model returns results in JSON format, which the system parses and stores in the database.

[0089] Step S4: Basic Assessment

[0090] Based on the retrieved evaluation criteria, a large language model is used to perform a basic evaluation of the question-and-answer records for the current topic. Specifically, for each perspective of the current topic, the large language model scores the user's answer based on the evaluation criteria for that perspective, obtaining a sub-score and a reason for scoring each perspective. Then, the sub-scores for each perspective are weighted and summed according to preset weights to obtain the base score for the current topic. The preset weights can be set by the user according to business needs. The basic scoring reasons for each perspective of the current topic will be recorded in the form of "[Perspective Score (out of 100)] Reason for Score", as shown in the table below:

[0091]

[0092] For example, the topic "communication skills" has three assessment angles: "clarity of expression" (40%), "logical coherence" (35%), and "interactive feedback" (25%). After scoring by the large language model, a weighted average is calculated to obtain the base score.

[0093] Step S5: Bonus Points Assessment

[0094] When at least one other topic exists besides the current topic, based on the evaluation criteria of the other topic, a large language model is used to evaluate the question-and-answer records for the current topic to obtain additional points and the reasons for the additional points. The specific calculation method for the additional points is as follows:

[0095] S51: Calculate the difference between the highest score in the current topic's base score rating level and the base score itself, and use this difference as the total bonus score. For example, if the base score is 83 and the highest score in the "Good" rating level is 89, then the total bonus score is 6.

[0096] S52: Divide the total bonus points equally among the number of other topics to obtain the bonus point amount for each other topic. Assuming there are 5 topics in total and 4 other topics, the bonus point amount for each other topic is 6 ÷ 4 = 1.5 points.

[0097] S53: For each other topic, divide its bonus points equally according to the number of perspectives for that topic, and obtain the bonus points for each perspective. If there are 4 perspectives for other topics, then the bonus points for each perspective are 1.5 ÷ 4 = 0.375 points.

[0098] S54: For each assessment angle of each other topic, the user's answer to the current topic is scored using a large language model based on the evaluation criteria of that assessment angle. In this embodiment, the score range for "good" is 80-89, and the score range for "excellent" is 90-100. The "good" threshold is set at 80 points, i.e., a score ≥ 80 points is considered "good" or "excellent". If the score reaches or exceeds the preset "good" threshold, the actual bonus score for that assessment angle is calculated as: bonus score for that assessment angle × (actual score / 100). Otherwise, the bonus score for that angle is 0.

[0099] S55: Add up the actual bonus points from all other topics and all assessment angles to obtain the bonus point for the current topic. The bonus point evaluation result for each other topic will be composed of: "Topic Bonus Point + Reasons for Evaluation of Each Angle". The reasons for evaluation of each angle will also be recorded in the form of "[Angle Score (out of 100)] Reasons for Score", as shown in the table below:

[0100]

[0101] Through the bonus points mechanism, the system can uncover the cross-dimensional advantages that users demonstrate in their answers to the current topic from the perspective of other topics, making the evaluation more comprehensive.

[0102] like Figure 3 As shown, the evaluation process for a single topic includes two stages: basic evaluation and bonus point evaluation. Basic evaluation stage: For each assessment angle of the current topic, the system uses that angle as input, retrieves the corresponding evaluation criteria through RAG retrieval, calls the large language model for scoring, obtains the sub-scores and scoring reasons for each angle, and then calculates the weighted basic score. Bonus point evaluation stage: The system iterates through all other topics except the current one. For each assessment angle of each other topic, it similarly retrieves the evaluation criteria through RAG retrieval, calls the large language model for scoring, and if the score reaches or exceeds the good threshold, bonus points are calculated according to a preset formula. Finally, the bonus points for the current topic are accumulated. Figure 3 The dashed box in the text indicates that the bonus assessment is an optional step and should be skipped if no other topics are available.

[0103] In a specific implementation of this invention, different prompt word templates were designed for basic evaluation, advantages and disadvantages analysis, and bonus score evaluation to guide the large language model to output structured results that meet the requirements. Specifically:

[0104] Basic assessment prompts: The large language model is required to take the current assessment angle and evaluation criteria as input and output the user's answer level, score, and reason for the score. The prompts explicitly state that the output format is key-value pairs, for example: {"angle": "clarity of expression", "rank": "good", "score": 85, "reason": "fluent language, clear logic, occasional repetition"}.

[0105] Advantages and disadvantages analysis prompts: The large language model is required to comprehensively extract the user's advantages and disadvantages in the current topic's answers based on the scoring reasons from various perspectives. The output format is: {"Advantages": "...", "Disadvantages": "..."}.

[0106] Bonus score assessment prompts: Similar to basic assessment prompts, but the assessment criteria for retrieval are derived from the perspective of other topics, and the output scores provide the basis for calculating bonus scores.

[0107] The three evaluation stages described above can utilize the same general-purpose large language model, or they can utilize different large language model instances as needed (e.g., using GPT-4 for basic evaluation and Llama 3 for strengths and weaknesses analysis). Using different model instances avoids bias from a single model and enhances the diversity and robustness of the evaluation. This invention does not limit the specific type of large language model; any general-purpose large language model with logical reasoning and text generation capabilities is applicable.

[0108] The following are examples of prompts for a single assessment angle in the basic evaluation phase:

[0109] 1. Role

[0110] You are a personal interview assistant responsible for comprehensively analyzing the overall rating of the "answers" to each question in the "User Q&A" section from the perspective of the current investigation angle, based on the RAG search results.

[0111] 2. Processing flow

[0112] 2.1 The first step is to confirm the "rating" (one of the four: "Excellent", "Good", "Pass" or "Fail") and the "rating requirements".

[0113] 2.2 The second step is to conduct specific scoring according to the "rating requirements" (excellent scoring range (90~100), good scoring range (80~89), qualified scoring range (60~79), unqualified scoring range (0~60)) to obtain the "score".

[0114] 3. Constraints

[0115] 3.1 "User Q&A": [Topic Q&A Records].

[0116] 3.2 The output format can only output key-value pairs, with the following format: {"angle":"

Current evaluation angle

[0117] The prompts for the strengths and weaknesses analysis and bonus point evaluation can be adapted to the template above and will not be repeated here.

[0118] Step S6: Generate the current topic evaluation result

[0119] Based on the base score, the reason for the base score, the bonus score, and the reason for the bonus score, an evaluation result for the current topic is generated. For example... Figure 6 As shown, the evaluation results include: the total score for the current topic, a scoring summary, the advantages and disadvantages of the current topic, the base score for the current topic, the basic scoring reasons for each evaluation angle of the current topic, and the additional points obtained from other topics and the reasons for those additional points. Among these:

[0120] Advantages and disadvantages analysis: Based on the basic scoring reasons from various assessment angles of the current topic, the large language model organizes and generates a summary of the advantages and disadvantages of the answers to the current topic.

[0121] Scoring Summary: The large language model generates a comprehensive evaluation summary based on the basic scoring reasons and the additional scoring reasons.

[0122] Step S7: Generate Interview Evaluation Report

[0123] The evaluation results for all topics are summarized to generate a complete interview evaluation report. The report can be exported in PDF, Word, and other formats for administrators or users to view.

[0124] In the above method, the large language model is a general-purpose large language model, such as GPT-4 or Llama, which does not require fine-tuning for the interview evaluation task, thus reducing deployment costs.

[0125] Example 2

[0126] like Figure 4 As shown, this embodiment provides an LLM-based artificial intelligence interview evaluation system, including the following modules:

[0127] Knowledge base construction module: used to build an evaluation standard knowledge base. The evaluation standard knowledge base is generated by labeling and dividing the evaluation standards of multiple topics. Each topic contains multiple examination angles, each examination angle corresponds to multiple rating levels, and each rating level is associated with the corresponding evaluation requirements. The labeling and division includes at least a three-layer structure of topic dimension, examination angle dimension, and rating level dimension.

[0128] Data Acquisition Module: This module acquires user interview and Q&A records for a preset list of questions and stores them by topic. It is configured to ask questions according to a fixed list and order, without modifying the question content or generating new questions; it also supports users skipping or re-answering.

[0129] The retrieval module is used to retrieve evaluation criteria corresponding to the current assessment perspective of the topic from the evaluation criteria knowledge base. The retrieval module includes a first retrieval unit and a second retrieval unit, each performing a two-step RAG retrieval.

[0130] Basic Assessment Module: Based on the retrieved assessment criteria, this module uses a large language model to perform a basic assessment of the question and answer records for the current topic, obtaining a basic score and the reasons for the basic score.

[0131] Bonus Points Assessment Module: When at least one other topic exists besides the current topic, this module assesses the question-and-answer records for the current topic using a large language model, based on the assessment criteria of the other topic, to obtain bonus points and the reasons for the bonus point assessment. This module calculates the bonus points according to steps S51-S55 in Example 1.

[0132] The topic result generation module is used to generate an evaluation result for the current topic based on the base score, basic scoring reasons, bonus points, and bonus point scoring reasons. The evaluation result includes the total score of the current topic, a scoring summary, the advantages and disadvantages of the current topic, the base score of the current topic, the basic scoring reasons for each assessment angle of the current topic, and the bonus points and bonus point scoring reasons obtained from other topics. The advantages and disadvantages are generated by a large language model based on the basic scoring reasons for the current topic's assessment angles, and the scoring summary is generated by a large language model based on the basic scoring reasons and bonus point scoring reasons.

[0133] Report generation module: Used to summarize the evaluation results of all topics and generate interview evaluation reports.

[0134] When generating the interview evaluation report, this embodiment also supports exporting the evaluation results and Q&A records as PDF format. Specifically, the system sends the evaluation results to the backend and calls the report generation component to generate a personal interview evaluation form PDF; at the same time, it sends the Q&A records to the backend to generate a personal interview record form PDF. Users can download or print the above PDF files for archiving or manual review.

[0135] In the above system, the basic assessment module and the bonus assessment module call the general large language model, and the model has not been fine-tuned.

[0136] The following describes the collaboration process between the modules of this invention in conjunction with the module interaction sequence.

[0137] After the user answers a question on the front end, the front end saves the question and answer record to the data storage module. The specific interaction sequence is as follows:

[0138] The system prompts the user to start the interview, and the user enters personal information, such as employee ID and name.

[0139] The system asks questions sequentially according to a pre-set question list. For each topic, the system asks the first question under that topic. After the user answers, the front end saves the current question and answer record to the data storage module and updates the question and answer record for the current topic.

[0140] The system continues to ask the next question on the topic, the user answers, and the record is updated; this process is repeated until all questions on the topic have been asked.

[0141] Once all questions in a topic have been asked, the system will automatically switch to the next topic and repeat the questioning and recording process until all questions in all topics have been asked.

[0142] After all questions are answered, the system will notify the user that the interview has ended and save all question and answer records.

[0143] During the interview, users can request to skip the current question or answer the previous question again at any time. The system will dynamically adjust the order of questions according to the user's instructions, but will not modify the content and order of the question list.

[0144] Once all question-and-answer records for all topics have been collected, the system triggers the interview evaluation process. During the evaluation, the system evaluates each topic sequentially, with the evaluation process being identical for each topic. The following example, using a single topic i, illustrates the interaction sequence between the evaluation modules:

[0145] Triggering Assessment: Once all question-and-answer records for all topics have been collected, the system sends the question-and-answer records for topic i (in sequence for each topic) and a list of assessment angles for all topics to the basic assessment module or the bonus assessment module. Basic Assessment: The basic assessment module sends the current assessment angle for the current topic to the retrieval module. The retrieval module retrieves the corresponding assessment criteria from the assessment criteria knowledge base through a two-step RAG process and returns it to the basic assessment module. Specifically:

[0146] For the first perspective i1 of topic i, the basic evaluation module sends a search request to the search module, inputting perspective i1.

[0147] The retrieval module retrieves all levels of the assessment criteria under angle i1 from the assessment criteria knowledge base and returns them to the basic assessment module.

[0148] The basic assessment module calls the large language model, judges the level of the user's answer in that angle based on the returned standard, and then scores it according to the specific standard of that level to obtain the sub-scores and scoring reasons for angle i1.

[0149] Repeat the above process to evaluate all perspectives of topic i, i1~in, in turn.

[0150] After the scores from each angle are completed, the basic evaluation module calculates the basic score for topic i by weighting it according to the preset weights.

[0151] Advantages and disadvantages analysis: The basic assessment module uses the scoring reasons from various perspectives to generate a summary of the advantages and disadvantages of the answer to topic i by calling the large language model.

[0152] Bonus Points Assessment: After the basic assessment is completed, the bonus points assessment module immediately begins the bonus points assessment. The system iterates through all other topics except topic i, and for each assessment angle of each other topic, repeats the above retrieval and invocation process:

[0153] The supplementary assessment module sends other topics for consideration to the retrieval module, and the retrieval module returns the assessment criteria for all levels under that perspective.

[0154] The bonus assessment module calls the large language model to determine the level of the user's answer to the current topic i from that perspective, and gives a score.

[0155] If the score reaches or exceeds the good threshold, the actual bonus score for that angle is calculated according to the preset formula: bonus score amount × (actual score / 100); otherwise, the bonus score is 0.

[0156] The actual bonus scores for all additional topics from all angles are summed to obtain the bonus score for topic i.

[0157] Total Score Calculation and Summary Generation: The bonus score evaluation module merges the bonus score with the basic score to generate the total score for topic i. Simultaneously, based on the reasons for the basic and bonus score evaluations, it uses a large language model to generate a score summary for topic i.

[0158] Results storage: The basic evaluation module / bonus score evaluation module saves the total score, rating summary, advantages and disadvantages, basic score, reasons for rating from each angle, bonus score, and reasons for each bonus score of topic i to the data storage module.

[0159] Once all topics have been evaluated, the report generation module reads the evaluation results of all topics from the data storage module and generates a complete interview evaluation report.

[0160] Example 3

[0161] To more clearly illustrate the specific content of the assessment criteria knowledge base, the following example uses company employee interviews as an example to provide an example of assessment criteria for some topics. As shown in Table 1, each topic includes multiple assessment angles, each with a weight, and corresponding assessment requirements texts for four levels: unsatisfactory, satisfactory, good, and excellent.

[0162] Table 1. Examples of Evaluation Criteria

[0163]

[0164] Table 1 lists assessment angles under the topic of "Planning and Execution Capability," including strategic planning and execution, goal management and monitoring, administrative efficiency and execution, and results-oriented and closed-loop management. Each angle provides specific behavioral descriptions at four levels. The assessment criteria for other topics (such as resource integration and communication coordination, and learning and growth) follow the same pattern. In practical applications, assessment criteria can be customized according to specific interview scenarios and stored in a knowledge base after being labeled and categorized using the method of this invention.

[0165] The following section uses an interview with a company employee as an example to explain in detail the implementation process of this invention.

[0166] 1. Topic Setting

[0167] The course covers five topics: planning and execution skills, complex problem-solving skills, resource integration and communication coordination, professional depth and industry research, and learning and growth.

[0168] 2. Example of evaluation criteria

[0169] Taking "Planning and Execution Ability" as an example, the assessment angles include: goal clarity, step rationality, time management, and risk prediction. Each angle corresponds to four levels (unsatisfactory, satisfactory, good, and excellent), and each level has specific evaluation requirements. For example, the "excellent" level requirement for the "goal clarity" angle is: "Able to clearly articulate long-term and short-term goals, and the goals are quantifiable and traceable."

[0170] 3. Interview Process

[0171] The system asks questions sequentially from a pre-set list, and the user answers. Users can skip questions they don't want to answer or re-answer the previous question. All question-and-answer records are stored by topic.

[0172] 4. Basic Assessment

[0173] For the topic of "planning and execution ability," the large language model scores four aspects, assuming the scores are: goal clarity 92 points, step rationality 85 points, time management 78 points, and risk prediction 80 points. Each aspect has a weight of 25%, so the base score = (92 + 85 + 78 + 80) × 25% = 83.75 points.

[0174] 5. Bonus points assessment

[0175] A base score of 83.75 falls within the "Good" range (80-89). The highest score is 89, so the total bonus points are 89 - 83.75 = 5.25. There are four other topics, each with a bonus point allocation of 5.25 ÷ 4 = 1.3125. Each other topic has four perspectives, with each perspective having a bonus point allocation of 1.3125 ÷ 4 = 0.328125.

[0176] Assuming that the "innovative thinking" aspect of the "complex problem-solving ability" assessment in other topics scores the answer to the current topic at 88 points (≥ good threshold), then the actual bonus points for this aspect = 0.328125 × (88 / 100) ≈ 0.28875 points. The total bonus points for all other topics and all aspects are added together, resulting in approximately 2.1 points.

[0177] 6. Evaluation Results

[0178] Total score = Base score + Bonus score = 83.75 + 2.1 = 85.85 points. A scoring summary, strengths and weaknesses, and reasons for each scoring aspect are also generated. The final report is as follows: Figure 6 As shown in the format.

[0179] 7. User Feedback

[0180] The human resources department found the personal rating reports generated by the system to be very comprehensive and well-founded. The satisfaction rate with the "skip / retake" function was over 98%, and the approval rate for the "rating reasons + advantages and disadvantages + summary" was over 96%.

[0181] After conducting an interview with Zhou Chen, an employee of a certain company, using the method described in this embodiment, the system automatically generated the following: Figure 6 Evaluation report of the structure shown. The main contents of the report are as follows:

[0182] Final score: 91.44 points

[0183] Basic scores for each topic: Planning and execution ability 93.5 points, complex problem-solving ability 85.0 points, resource integration and communication coordination 84.5 points, professional depth and industry insight 85.0 points, learning and growth 84.0 points.

[0184] Bonus points total: 1.45 + 1.36 + 1.38 + 1.41 ≈ 5.6 points were awarded for topics including complex problem-solving ability, resource integration and communication coordination, professional depth and industry insight, and learning and growth.

[0185] Evaluation summary: "Excellent planning and execution capabilities, innovative solutions to project pain points, and strong system analysis and resource integration abilities."

[0186] Strengths Analysis: Excellent performance in strategic decoding, goal management, action efficiency, and results orientation; proactive in coordinating resources, introducing new resources, and effectively promoting cooperation; accurate professional judgment, able to anticipate and solve problems.

[0187] Weaknesses: The flexibility of response strategies in certain extreme situations needs to be improved; the strategic influence on industry trends is slightly insufficient; and the systematic description of the learning methods is relatively brief.

[0188] The report shows that the present invention can comprehensively and meticulously evaluate users’ performance on multiple topics and capture cross-dimensional advantages through an additional scoring mechanism. The evaluation results have high interpretability and practicality.

[0189] Example 4

[0190] To verify the effectiveness of the labeled partitioning and RAG two-step retrieval in this invention, a comparative experiment was conducted. Under the same question set and evaluation criterion library, RAG retrieval with and without labeled partitioning was performed respectively. The test sample size was 1000 retrieval requests, and accuracy was defined as the proportion of returned evaluation criteria that perfectly matched the current perspective. The results are as follows: Figure 7 As shown, the retrieval accuracy rate is 99.88% with labeled partitioning, but only 49.15% without labeled partitioning. This indicates that the labeled partitioning of the present invention significantly improves the accuracy of matching evaluation criteria.

[0191] Furthermore, to illustrate the technical effectiveness of the bonus point mechanism, let's compare the same question-and-answer record from the same user. Assume the current topic is "communication skills," with a base score of 83 (good level). Without the bonus point mechanism, the system only outputs the base score and the reasons for the scores from each angle of the topic, failing to reflect the user's strengths in other topics (such as "professional skills"). With the bonus point mechanism, the system will review every assessment angle of other topics and score the current answer. For example, the user may unintentionally demonstrate "cross-domain knowledge integration ability" in the current answer, which falls under the "knowledge transformation and application" angle of the "professional skills" topic. The system will award 1.2 bonus points, explicitly stating in the bonus point scoring reason: "When describing communication cases, the user naturally incorporated innovative methods from their professional field, demonstrating good knowledge transfer ability." The final total score becomes 84.2 points. It is evident that the bonus point mechanism can capture cross-dimensional advantages, making the evaluation results more comprehensive and fair.

[0192] In summary, the method and system provided by this invention can achieve efficient, accurate, comprehensive, and interpretable automated interview assessment, and have significant practical value.

[0193] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An AI-based interview evaluation method based on LLM, characterized in that, Includes the following steps: Step S1: Construct an evaluation standard knowledge base. The evaluation standard knowledge base is generated by labeling and dividing the evaluation standards for multiple topics. Each topic contains multiple assessment angles, each assessment angle corresponds to multiple rating levels, and each rating level is associated with corresponding evaluation requirements. The labeling and division includes at least a three-layer structure: topic dimension, assessment angle dimension, and rating level dimension. Step S2: Obtain the interview and Q&A records of users for the preset question list, and store them by topic; Step S3: For the current topic, retrieve the evaluation criteria corresponding to the current perspective of this topic from the evaluation criteria knowledge base; Step S4: Based on the retrieved evaluation criteria, use a large language model to perform a basic evaluation of the question and answer records for the current topic, and obtain the basic score and basic scoring reasons for the current topic; Step S5: When there is at least one other topic besides the current topic, based on the evaluation criteria of the other topic, use the large language model to evaluate the question and answer records of the current topic with additional points, and obtain the additional points and the reasons for the additional points. Step S6: Based on the base score, the reason for the base score, the bonus score, and the reason for the bonus score, generate the evaluation result for the current topic; Step S7: Summarize the evaluation results of all topics and generate an interview evaluation report.

2. The LLM-based artificial intelligence interview evaluation method as described in claim 1, characterized in that: The retrieval in step S3 employs retrieval enhancement generation technology, specifically including: The first retrieval stage: using the current perspective of the current topic as input, retrieve the evaluation criteria for all rating levels under that perspective, so that the large language model can determine the rating level to which the user's answer belongs; The second retrieval stage involves using the assessment angle and rating level determined by the large language model as input to retrieve the specific evaluation requirements for that rating level, which are then used by the large language model for scoring.

3. The LLM-based artificial intelligence interview evaluation method as described in claim 1, characterized in that: The specific implementation method of the basic assessment in step S4 is as follows: For each perspective of the current topic, the large language model is used to score the user's answer based on the evaluation criteria of that perspective, and the sub-scores and scoring reasons for each perspective are obtained. The scores for each aspect of the evaluation are weighted and summed according to preset weights to obtain the base score for the current topic.

4. The LLM-based artificial intelligence interview evaluation method as described in claim 1, characterized in that: The specific implementation method of the bonus point assessment in step S5 is as follows: S51. Calculate the difference between the highest score in the current topic's base score rating level and the base score, and use this as the total bonus score. S52. Divide the total bonus points equally according to the number of other topics to obtain the bonus points amount for each other topic; S53. For each other topic, the additional score is divided equally according to the number of perspectives for that topic, so as to obtain the additional score for each perspective. S54. For each examination angle of each other topic, use the large language model to score the user's answer to the current topic based on the evaluation criteria of that examination angle. If the score reaches or exceeds the preset good threshold, calculate the actual bonus score for that examination angle: bonus score amount for that examination angle × (actual score / 100); otherwise, the bonus score for that examination angle is 0. S55. Add up the actual bonus points for all other topics from all perspectives to get the bonus points for the current topic.

5. The LLM-based artificial intelligence interview evaluation method as described in claim 1, characterized in that: The current topic evaluation results generated in step S6 include: the total score of the current topic, the scoring summary, the advantages and disadvantages of the current topic, the basic score of the current topic, the basic scoring reasons for each evaluation angle of the current topic, the additional scores obtained from other topics and the scoring reasons for each additional score; The advantages and disadvantages are based on the basic scoring reasons from various perspectives of the current topic, which are generated by a large language model. The scoring summary is based on the basic scoring reasons and the additional scoring reasons, which are also generated by a large language model.

6. The LLM-based artificial intelligence interview evaluation method as described in claim 1, characterized in that: The content and order of questions in the preset question list are fixed. During the interview, the large language model does not modify the content of any questions, nor does it generate new questions. The large language model described is a general-purpose large language model, which does not require fine-tuning for the interview assessment task.

7. An AI-based interview assessment system based on LLM, characterized in that, include: The module includes a knowledge base construction module, a data collection module, a retrieval module, a basic assessment module, an additional score assessment module, a topic results generation module, and a report generation module. The knowledge base construction module is used to build an evaluation standard knowledge base. The evaluation standard knowledge base is generated by labeling and dividing the evaluation standards of multiple topics. Each topic contains multiple examination angles, each examination angle corresponds to multiple rating levels, and each rating level is associated with the corresponding evaluation requirements. The labeling classification includes at least three layers of structure: topic dimension, assessment angle dimension, and rating level dimension. The data acquisition module is used to acquire interview and Q&A records of users on a preset list of questions and store them by topic. The retrieval module is used to retrieve, for the current topic, evaluation criteria corresponding to the current perspective of the topic from the evaluation criteria knowledge base; The basic assessment module is used to perform a basic assessment of the question and answer records of the current topic based on the retrieved assessment criteria and using a large language model to obtain the basic score and basic scoring reasons for the current topic. The bonus score evaluation module is used to evaluate the question and answer records of the current topic based on the evaluation criteria of the other topics and using a large language model to obtain bonus scores and the reasons for the bonus score evaluation when there are at least one other topic besides the current topic. The topic result generation module is used to generate the evaluation result of the current topic based on the base score, the reason for the base score, the bonus score, and the reason for the bonus score. The report generation module is used to summarize the evaluation results of all topics and generate interview evaluation reports.

8. The system according to claim 7, characterized in that, The retrieval module is further configured as follows: The first retrieval unit is used to retrieve the evaluation criteria for all rating levels under the current perspective of the current topic as input, so that the large language model can determine the rating level of the user's answer. The second retrieval unit is used to retrieve the specific evaluation requirements for the corresponding level based on the rating level determined by the large language model, using the evaluation perspective and rating level as input, for the large language model to score. The bonus score assessment module is further configured as follows: Calculate the difference between the highest score in the current topic's base score rating level and the base score, and use this difference as the total bonus score. The total bonus points are divided equally among the number of other topics to obtain the bonus point amount for each other topic. For each other topic, its bonus points are divided equally according to the number of perspectives on that topic, resulting in the bonus points for each perspective. For each assessment angle of each other topic, the large language model is used to score the user's answer to the current topic based on the evaluation criteria of that assessment angle. If the score reaches or exceeds the preset good threshold, the actual bonus score for that assessment angle is calculated as: bonus score amount for that assessment angle × (actual score / 100); otherwise, the bonus score for that assessment angle is 0. The bonus score for the current topic is obtained by summing up the actual bonus scores from all other topics and all perspectives. The current topic evaluation results generated by the topic result generation module include: the total score of the current topic, the scoring summary, the advantages and disadvantages of the current topic, the base score of the current topic, the basic scoring reasons for each evaluation angle of the current topic, the additional scores obtained from other topics, and the scoring reasons for the additional scores; wherein, the advantages and disadvantages are generated by the large language model based on the basic scoring reasons based on the evaluation angle of the current topic, and the scoring summary is generated by the large language model based on the basic scoring reasons and the scoring reasons for the additional scores; The data acquisition module is configured as follows: Ask questions according to a pre-set fixed list and question order, without modifying the question content or generating new questions; The basic assessment module and the bonus assessment module both use the general large language model, and this model has not been fine-tuned.

9. An electronic device, characterized in that, include: A memory and a processor, the memory storing a computer program executable on the processor, wherein the electronic device, when executing the program, implements the method of any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images