Interview evaluation method and device, electronic equipment and storage medium

Abstract

The invention relates to an interview evaluation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target interview video, wherein the target interview video at least comprises video images of interview candidates and voices of the interview candidates and interviewers; generating a voice transcription interview text of the target interview video through a voice transcription service of the first large model, wherein the voice transcription interview text comprises an interview candidate text and an interviewer text; respectively generating corresponding semantic vectors based on the interview candidate text and the interviewer text; generating a plurality of expression vectors corresponding to different specified expressions based on the interview video; and combining the semantic vectors of the two interviews and the plurality of expression vectors into an input vector of an interview evaluation model, inputting the input vector to the interview evaluation model, and outputting an interview score through the interview evaluation model. According to the embodiment of the invention, the comprehensiveness, accuracy and scientificity of interview evaluation can be effectively improved, and the cost is low.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and deep learning technology, and in particular to an interview evaluation method, device, electronic device and storage medium based on artificial intelligence and deep learning. Background Art

[0002] With the development of society and the continuous innovation of corporate recruitment models, interviews, as a key step in talent selection, have become an indispensable part of the recruitment process. Traditional interview evaluation methods primarily rely on interviewers making subjective judgments based on their interview experience through face-to-face communication with candidates. With the advancement of technology, interviews are becoming increasingly diverse. Interviews are no longer limited to face-to-face meetings, but can also be conducted via voice or video. Interview evaluation methods have also undergone significant changes due to these changes in interview methods and technological advancements. For example, based on the multimodal data generated by interviews, such as voice, video, and text, computer vision (expression and body movement recognition), speech analysis (intonation, speech rate, emotion), and natural language processing (NLP) are used to comprehensively evaluate candidate performance. This comprehensive evaluation typically involves scoring the corresponding evaluation dimensions based on the multimodal data, using different techniques depending on the dimension. For example, the dimensions provided in the solution provided in the Chinese invention patent with the publication number CN114186983B and the invention name “Multi-dimensional scoring method, system, computer equipment and storage medium for video interviews” include: statement quality, various competency levels, Mandarin proficiency, English speaking proficiency, external image and intention inquiry, wherein the XGBoost extreme gradient boosting algorithm is used to train multiple models to score the statement quality, various competency levels and other dimensions respectively. For another example, the solution provided in the Chinese invention patent with the publication number CN112001628B and the invention name “Intelligent interview video recommendation method” is based on facial expression features, voice emotion features and voice text features, and uses an ability evaluation model to generate a seven-dimensional video clip evaluation vector. Each dimension in the evaluation vector corresponds to an interview assessment indicator, and the ability evaluation model is a combination of a long-short memory model and a conditional random field. There are some other similar solutions, which will not be listed here one by one.

[0003] A comprehensive review of existing interview evaluation methods reveals that existing interview evaluation technology solutions typically utilize automated scoring to reduce manual intervention. This avoids the subjective influence and bias of interviewers in traditional interview evaluation methods, increases the objectivity of interview results, and speeds up processing, significantly improving interview efficiency. However, existing interview evaluation methods still have the following shortcomings:

[0004] First, existing interview evaluation technology solutions usually conduct multi-dimensional analysis and obtain evaluation results based on multi-dimensional analysis. Therefore, this method of evaluating candidates is limited to the dimensions set in the solution. When the set dimensions are different, the evaluation results will vary greatly, resulting in poor consistency in the evaluation results.

[0005] Secondly, in order to complete multi-dimensional analysis and final evaluation, weights are usually preset and a limited number of evaluation dimensions are weighted for calculation. The weights are usually preset based on experience, so it is difficult to adapt to all candidates and all scenarios.

[0006] Furthermore, the models used in existing solutions are usually complex and require high-performance computing resources (such as the Spark cluster in CN114186983B), and the cost of customized model development is high. Summary of the Invention

[0007] In response to the technical problems existing in the prior art, the present invention proposes an interview evaluation method, device, electronic device and storage medium, which realize interview evaluation of candidates based on interview videos, with high processing efficiency, accurate evaluation results and low development and use costs.

[0008] In order to solve the above technical problem, according to one aspect of the present invention, the present invention provides an interview evaluation method, comprising the following steps:

[0009] Obtaining a target interview video, wherein the target interview video includes at least a video image of the interview candidate and voices of the interview candidate and the interviewer;

[0010] Generate a speech transcription interview text of the target interview video through the speech transcription service of the first large model, wherein the speech transcription interview text includes the interview candidate text and the interviewer text;

[0011] Generate corresponding semantic vectors based on the interview candidate text and the interviewer text respectively;

[0012] Generate multiple expression vectors corresponding to different specified expressions based on the interview video; and

[0013] The semantic vectors and multiple expression vectors of both interviewees are combined into the input vector of the interview evaluation model, which is input into the interview evaluation model, and the interview evaluation model outputs the interview score, wherein the interview evaluation model is a neural network model trained with the interview video training set.

[0014] Optionally, when there are multiple interviewers, the voice transcription interview texts of the multiple interviewers are merged together to form the interviewer text.

[0015] Optionally, the step of generating a speech transcription interview text of the target interview video through the speech transcription service of the first large model includes:

[0016] Send the target interview video to the first model via the voice transcription service interface of the first model;

[0017] Receive and store the interview candidate text and interviewer text returned by the first model;

[0018] Count the length of the candidate's text and the interviewer's text respectively;

[0019] Check whether the text length of the candidate text and the interviewer text exceeds the preset length respectively;

[0020] In response to the text length of the interview candidate text and / or the interviewer text exceeding a preset length, inputting the interview candidate text and / or the interviewer text exceeding the preset length to the second large model via the large model key point generation service interface; and

[0021] Receive the key text returned by the second large model, and use the key text as the corresponding interview candidate text and / or interviewer text; wherein the first large model and the second large model are the same large model or different large models.

[0022] Optionally, the step of generating corresponding semantic vectors based on the interview candidate text and the interviewer text respectively includes:

[0023] Perform word segmentation processing on the interview candidate text and the interviewer text respectively to obtain their respective numerical text input word lists;

[0024] Inputting the digitized text input vocabulary of the interview candidate and the interviewer into the lightweight semantic model respectively; and

[0025] Obtain the output vectors of the lightweight semantic model after processing the digitized text input vocabulary of the interview candidate and the interviewer as the semantic vectors of each interviewee;

[0026] The lightweight semantic model is a semantic processing model trained with a training data set, and the training samples in the training data set are interview texts obtained by voice transcription of interview videos.

[0027] Optionally, the step of performing word segmentation processing on the interview candidate text and the interviewer text respectively to obtain respective numerical text input vocabulary includes:

[0028] Based on the preset BPE vocabulary, the interview candidate text and the interviewer text are segmented to obtain their respective segmentation vocabulary; and

[0029] Convert the words in each word list into word IDs to obtain a numerical text input word list;

[0030] Among them, the BPE vocabulary is a word segmentation table obtained by iterating the BPE algorithm based on the voice transcribed interview text obtained from the interview video.

[0031] Optionally, the step of generating expression vectors representing a plurality of expression vectors corresponding to different specified expressions based on the interview video includes:

[0032] Perform frame processing on the target interview video to obtain multiple video frame images;

[0033] Perform face detection on each video frame image to obtain the corresponding face image;

[0034] Each facial image is input to the expression recognition model, and the expression recognition model recognizes a specified expression corresponding to each facial image;

[0035] Count the proportion of each recognized specific expression in the total number of expressions; and

[0036] Convert the proportion of each specified expression into a multi-dimensional vector to obtain an expression vector corresponding to the specified expression;

[0037] The expression recognition model is a lightweight convolutional neural network model that has been trained using a training set, and each training sample in the training set has a label for a specified expression.

[0038] Optionally, there are three categories of designated expressions, and each designated expression is a basic expression representing the candidate's interview status determined based on historical interview data.

[0039] Optionally, the step of performing face detection on each video frame image to obtain a corresponding face image includes:

[0040] Using a single-stage object detection model to perform face detection on each video frame image to obtain the face boundary coordinates; and

[0041] Cropping is performed based on the face boundary coordinates to obtain the corresponding face image;

[0042] Among them, the single-stage target detection model is a lightweight convolutional neural network model trained with a training set.

[0043] Optionally, before performing face detection on each video frame image, the method further includes:

[0044] Verify the resolution of each video frame; and

[0045] In response to a resolution of the video frame image being less than a threshold, Gaussian noise and salt and pepper noise are added to the video frame image.

[0046] Optionally, the semantic vector and the expression vector are respectively multidimensional vectors, the difference between the number of dimensions of the semantic vector and the number of dimensions of the expression vector is less than a first threshold or the ratio of the number of dimensions of the semantic vector to the number of dimensions of the expression vector is less than a second threshold.

[0047] According to another aspect of the present invention, the present invention further provides an interview evaluation device, comprising:

[0048] A video acquisition module, configured to acquire a target interview video;

[0049] A text transcription module is configured to generate a speech transcription interview text of the target interview video through a speech transcription service of a large model, wherein the speech transcription interview text includes an interview candidate text and an interviewer text;

[0050] a semantic vector generation module configured to generate corresponding semantic vectors based on the interview candidate text and the interviewer text;

[0051] an expression vector generation module configured to generate expression vectors representing a plurality of expression vectors corresponding to different designated expressions based on the interview video; and

[0052] The evaluation module is configured to combine the semantic vectors and multiple expression vectors of the two interview parties into the input vector of the interview evaluation model and input it to the interview evaluation model, and output the interview score through the interview evaluation model, wherein the interview evaluation model is a neural network model trained with the interview video set as the training set.

[0053] According to another aspect of the present invention, the present invention also provides an electronic device, which includes a processor and a memory, wherein a computer program instruction set is stored on the memory, and the aforementioned interview evaluation method is implemented when the processor executes the computer program instruction set on the memory.

[0054] According to another aspect of the present invention, the present invention also provides a computer-readable storage medium, wherein a computer program instruction set is stored on the computer-readable storage medium, and the computer program instruction set implements the aforementioned interview evaluation method when executed by a processor.

[0055] According to another aspect of the present invention, the present invention further provides a computer program product, which includes a computer program instruction set, and when the computer program instruction set is executed by a processor, the above-mentioned interview evaluation method is implemented.

[0056] The embodiments of the present invention can effectively improve the comprehensiveness, accuracy and scientificity of interview evaluation, with high processing efficiency and low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Below, the preferred embodiments of the present invention will be further described in detail with reference to the accompanying drawings, in which:

[0058] Figure 1 is a flow chart of an interview evaluation method according to one embodiment of the present invention;

[0059] Figure 2 is a flow chart of a method for processing an interview text according to an embodiment of the present invention;

[0060] Figure 3 is a flow chart of a processing method for generating semantic vectors based on interview text according to one embodiment of the present invention;

[0061] Figure 4 is a flow chart of a method for constructing a BPE vocabulary according to an embodiment of the present invention;

[0062] Figure 5 is a flow chart of a method for generating expression vectors based on interview videos according to one embodiment of the present invention;

[0063] Figure 6 This is a principle block diagram of an interview evaluation device according to one embodiment of the present invention;

[0064] Figure 7 is a functional block diagram of a model training module according to one embodiment of the present invention;

[0065] Figure 8 is a flow chart of a data preparation and preprocessing method according to one embodiment of the present invention;

[0066] Figure 9 is a flow chart of a method for training an interview evaluation model according to one embodiment of the present invention;

[0067] Figure 10 is a flow chart of an interview evaluation method according to a specific embodiment of the present invention; and

[0068] Figure 11 FIG. 1 is a schematic diagram of the hardware structure principle of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0070] In the detailed description that follows, reference may be made to the various drawings that form part of this application and illustrate specific embodiments of the present application. In the drawings, similar reference numerals describe substantially similar components in different figures. Each specific embodiment of the present application is described below in sufficient detail to enable a person of ordinary skill in the art to implement the technical solutions of the present application. It should be understood that other embodiments may be utilized or that structural, logical, or electrical changes may be made to the embodiments of the present application.

[0071] In response to diversified and personalized recruitment forms, the present invention provides a standardized and process-based interview evaluation method. It no longer relies on the interviewer's subjective judgment and experience, nor is it limited by manually set evaluation dimensions and weights. Instead, it is based on the interview candidate's expression and the semantic content of the interview process. The evaluation results are objective and fair.

[0072] See also Figure 1 , Figure 1 This is a flow chart of an interview evaluation method according to one embodiment of the present invention, the interview evaluation method comprising the following steps:

[0073] Step S11: Obtain a target interview video, wherein the target interview video at least includes a video image of the interview candidate and the voices of the interview candidate and the interviewer.

[0074] Step S12: Generate a speech transcription interview text of the target interview video through the speech transcription service of the first large model. Specifically, the speech transcription interview text includes the interview candidate text and the interviewer text.

[0075] Step S13: Generate semantic vectors for both interviewees based on the voice-transcribed interview text. Specifically, corresponding semantic vectors are generated based on the candidate text and the interviewer text, respectively.

[0076] Step S14: generating a plurality of expression vectors corresponding to different specified expressions based on the interview video.

[0077] Step S15: combining the semantic vectors of both interviewees and multiple expression vectors into an input vector of the interview evaluation model.

[0078] Step S16: input the input vector to the interview evaluation model, and the interview evaluation model outputs the interview score.

[0079] With the widespread application of Internet technology and video technology, conducting interviews remotely and via video has become an efficient way of conducting recruitment interviews. During the interview process, the interview candidate usually faces the camera and display screen and communicates with the interviewer through a microphone. The interviewer usually records the interview process with the help of the video recording function of the software. In addition, in some on-site interview scenarios, the interview process is usually recorded in the form of a video. The interview video is stored in the computer device in the form of a file and has a corresponding name. For example, the interview video is named after the candidate's name, or named after the time or code name. When it is necessary to evaluate the interview of a specific interview candidate, in step S11, the interview video of the interview candidate is read from the specified location.

[0080] The large model is also called the Large Language Model (LLM), such as the Generative Pre-Trained Transformer (GPT) series models developed by Open AI, LaMDA developed by Google, PaLM, LLaMA developed by Meta, Wenxin Yiyan developed by Baidu, Tongyi Qianwen of Ali, Hunyuan of Tencent, the large language model named "AliceMind" developed by Alibaba Damo Academy, etc. The platforms that provide these large models usually include some other services, such as speech-to-text services, and provide an interface for users. In step S12, the present invention sends the target interview video to the first large model via the speech transcription service interface of the first large model, and the speech transcription function of the first large model transcribes the interview speech into text. The present invention utilizes the powerful semantic and speech recognition capabilities of the existing large models to accurately distinguish different speakers in the interview, accurately transcribe the speech content in the video into text, and can return the text of different speakers separately as required.

[0081] After obtaining the interview text, in order to increase the accuracy of semantic understanding and reduce the amount of data processing and the interference of irrelevant content in the subsequent semantic processing process, in one embodiment, the obtained interview text is also pre-processed, see Figure 2 . Figure 2 This is a flow chart of a method for processing an interview text according to an embodiment of the present invention. The interview text is a candidate text or an interviewer text, and the processing method includes the following steps:

[0082] Step S21: pre-process the current interview text to obtain a first interview text. The pre-processing includes removing colloquial affixes in the text (such as "this" and "that" that speakers often use when they cannot think of appropriate expressions during the presentation), eliminating repeated content (such as content repeated by speakers for the sake of coherence), and unifying the language of the text content (for example, when the speaker inserts English words into Chinese, the interview text is composed of both Chinese and English. During unification, the minority language content is translated into the majority language content based on the context).

[0083] Step S22: Count the length of the first interview text, for example, the number of characters in the text.

[0084] Step S23 checks whether the length of the first interview text exceeds a preset length. If the length of the interview text exceeds the preset length, step S24 is executed. If the length of the interview text does not exceed the preset length, step S26 uses the first interview text as the interview text for subsequent applications, and the processing flow ends.

[0085] In step S24, the main model generates the key points of the current interview text. For example, the interview text exceeding the preset length is input to the second main model via the main model key point generation service interface, and the main point text returned by the second main model is received.

[0086] Step S25: Use the key text as the interview text for subsequent applications, and end the processing flow.

[0087] The first large model and the second large model in this embodiment are the same large model or different large models.

[0088] In one embodiment, the DistilBERT model is used to generate semantic vectors in step S13. The DistilBERT model is a lightweight semantic model whose structure is a student network distilled from the BERT model. Through the self-attention mechanism in the network structure, it can learn the semantic content representing the entire text and output it as a multidimensional vector.

[0089] See also Figure 3 , Figure 3 This is a flow chart of a method for generating semantic vectors based on interview text according to an embodiment of the present invention. The method includes the following steps:

[0090] Step S131 : performing word segmentation processing on the interview text based on a preset BPE word list to obtain an interview text word segmentation word list.

[0091] Step S132: convert the segmented words in the interview text segmented word list into segmented word IDs to obtain a numerical text input word list.

[0092] Step S133: Input the digitized text input vocabulary into the DistilBERT model.

[0093] In step S134, the DistilBERT model processes the digitized text input vocabulary and outputs the corresponding semantic vector.

[0094] The BPE vocabulary is a word segmentation table obtained by iterating the BPE algorithm based on the voice transcription interview text obtained from the interview video. Figure 4 , Figure 4 FIG. 1 is a flow chart of a method for constructing a BPE vocabulary according to an embodiment of the present invention. The method for constructing a BPE vocabulary includes the following steps:

[0095] Step S31: Construct a etymology sample set. The etymology samples in the present invention are derived from the speech transcribed interview texts obtained from each interview video in the interview video training set. Each speech transcribed interview text is used as an etymology sample. The speech transcribed interview texts herein include the interview candidate text and the interviewer text.

[0096] Step S32: Split the text in all word source samples into the smallest single-word units to obtain an initial word list.

[0097] Step S33 determines a target adjacent unit pair in the current vocabulary. In the first iteration, the current vocabulary is the initial vocabulary. After each iteration, the current vocabulary is updated. In the first iteration, the first two adjacent single-word units in the initial vocabulary are used as the target adjacent unit pair. In subsequent iterations, the first word unit and its subsequent adjacent word unit are used as a target adjacent unit pair.

[0098] Step S34: Count the frequencies of target adjacent word unit pairs in the corpus.

[0099] Step S35: Determine whether the frequency of the target adjacent unit pair is greater than or equal to the threshold. If the frequency of the target adjacent unit pair is greater than or equal to the threshold, execute step S36. If the frequency of the target adjacent unit pair is less than the threshold, it is determined that they cannot be combined and used, and execute step S312.

[0100] Step S36: determine the target adjacent unit pair as a target word unit.

[0101] Step S37, determining whether the current vocabulary already contains the target word unit, if the current vocabulary does not contain the target word unit, executing step S38. If the current vocabulary contains the target word unit, executing step S311.

[0102] Step S38, determining whether the first unit in the target adjacent unit pair is a word unit with a segmentation ID, if the first unit in the target adjacent unit pair is a word unit with a segmentation ID, executing step S39. If the first unit in the target adjacent unit pair is a single-word unit, executing step S310.

[0103] In step S39, the target word unit replaces the target adjacent unit pair. The segmentation ID is not changed at this time, and step S312 is then executed.

[0104] In step S310, the target adjacent units are replaced by the target word units, and segmentation IDs are assigned in sequence. Then, step S312 is executed.

[0105] Step S311: discard the target word unit and execute step S312.

[0106] At this point, an iterative update of the vocabulary is completed.

[0107] Step S312, determine whether there are single-word units that have not been processed. If so, return to step S33. If not, the iteration is completed and the BPE vocabulary construction process ends.

[0108] In addition, in the above-mentioned process of constructing the vocabulary, when assigning a segmentation ID to a word unit, the segmentation ID is determined for the obtained segmentation in order. Therefore, in one embodiment, the number of segmentations in the vocabulary can be predetermined, for example, 10,000, so that each time a segmentation ID is obtained, it can further include judging whether the segmentation ID reaches 10,000 (the segmentation ID starts from 1) or 9,999 (the segmentation ID starts from 0). If it is reached, the construction process is stopped, thereby obtaining a BPE vocabulary of fixed size.

[0109] In this embodiment, each word in the interview text segmentation vocabulary obtained by segmenting the interview text based on the preset BPE vocabulary is a word in the BPE vocabulary. Therefore, in step S132, the words in the interview text segmentation vocabulary can be converted into segmentation IDs to obtain a digitized text input vocabulary.

[0110] The BPE vocabulary used in the present invention is derived from the interview text. Therefore, when the interview text is subsequently segmented to determine the semantic vector of the interview text, the accuracy of the segmentation can be effectively improved, thereby improving the accuracy of the semantic vector.

[0111] In one embodiment, the DistilBERT model is a trained model. During training, the number of word segments (or tokens) in the input vector of the DistilBERT model and the number of dimensions of the output vector are set, so that in step S134, the semantic vector output by the DistilBERT model with the preset number of dimensions can be obtained.

[0112] See also Figure 5 , Figure 5 This is a flow chart of a method for generating expression vectors based on interview videos according to one embodiment of the present invention. The method comprises the following steps:

[0113] Step S141, the target interview video is framed to obtain multiple video frame images. For example, one frame image is extracted every other duration. Or the total number of images to be extracted is first determined, and the interval duration for extracting one frame image is determined based on the total number of images and the duration of the target interview video. In another embodiment, before the target interview video is framed, the video image is aligned with the voice, and when the frame processing is performed, the extraction interval duration of the video frame image is determined based on the density of the voice data. For example, during a voice conversation, when valid voice is detected, the video frame image is extracted with a first duration (i.e., a short duration), and when no valid voice is detected, i.e., no voice conversation is conducted, the video frame image is extracted with a second duration (long duration) so as to capture the candidate's subtle expressions during the interview conversation. The second duration is greater than the first duration.

[0114] In step S142, face detection is performed on each video frame to obtain a corresponding face image. When performing face detection to obtain the corresponding face image, a single-stage object detection model (such as a YOLO model) is used to perform face detection on each video frame to obtain face boundary coordinates; the face image is then cropped based on the face boundary coordinates to obtain the corresponding face image. The single-stage object detection model is a lightweight convolutional neural network model trained using a training set.

[0115] In step S143, each facial image is input to the expression recognition model respectively, and the expression recognition model recognizes a specified expression corresponding to each facial image. The expression recognition model is a lightweight convolutional neural network model (for example, a MobileNetV3 model) that has been trained with a training set, and each training sample in the training set has a label with a specified expression. Each specified expression is a basic expression representing the candidate's interview status determined based on historical interview data. For example, three expression categories are designated as "nervous", "calm" and "confident". When training the expression recognition model, the label content of each training sample is one of the three expressions of "nervous", "calm" and "confident". Through a certain number of training samples, the expression recognition model can learn the specificity of these three expressions and can accurately classify the input facial images.

[0116] Step S144: Count the proportion of each recognized designated expression in the total number of expressions.

[0117] Step S145, converting the proportion of each specified expression into a multidimensional vector to obtain an expression vector corresponding to the specified expression. In a specific embodiment, a binning processing method is used to convert the proportion of the specified expression into a multidimensional vector. For example, the binning boundary value is determined based on the determined number of vector dimensions, and the range of expression proportion values is divided into multiple intervals based on the binning boundary value. Each interval is used as a dimension, and each dimension has an index value. The index value corresponding to the interval where the expression proportion value is located is determined to be 1, and the index value corresponding to the interval where the expression proportion value is not located is determined to be 0. All index values are combined together in the order of the intervals to form a multidimensional vector.

[0118] If the proportion of expression vectors in the input vector of the interview evaluation model is too small, the interview evaluation model will rely too much on semantic vectors during the evaluation process and will not be able to fully utilize the expression vectors, resulting in the prediction being too biased towards the influence of semantic content. Therefore, in one embodiment, the difference between the number of dimensions of the semantic vector and the number of dimensions of the expression vector is smaller than a first threshold, or the ratio of the number of dimensions of the semantic vector to the number of dimensions of the expression vector is smaller than a second threshold, that is, the proportion of expression vectors and semantic vectors in the input vector of the interview evaluation model is balanced, so that the interview evaluation model can fully utilize the two contents to make a fair, comprehensive and accurate evaluation of the interview candidates.

[0119] In addition, before executing step S142 to perform face detection on each video frame image, in order to ensure the accuracy of the detection results and subsequent expression recognition results, the resolution of each video frame image is also checked; when the resolution of the video frame image is less than a threshold, Gaussian noise and salt and pepper noise are added to the video frame image to improve the quality of the video frame image.

[0120] In order to avoid the model from over-relying on dominant features and ignoring secondary features during prediction due to the imbalance in the number of dimensions representing the two meanings in the model input vector, the difference between the number of dimensions of the semantic vector and the number of dimensions of the expression vector in the input vector of the interview evaluation model constituted by the present invention is less than a first threshold or the ratio of the number of dimensions of the semantic vector to the number of dimensions of the expression vector is less than a second threshold, so that the interview evaluation model can make full use of the two types of data to make fair and accurate scores during prediction. The first threshold here is, for example, greater than half of the total number of dimensions, and the second threshold is, for example, a value less than 2.

[0121] After the aforementioned expression recognition and semantic extraction, a highly semantically robust vector is generated. Therefore, the interview evaluation model in this invention can employ a relatively simple neural network model. Due to its relatively simple structure and low number of parameters, it offers excellent generalization and fit, resulting in fast training and effective inference, enabling accurate scoring.

[0122] The YOLOv10 model in this embodiment is an efficient target detection model, and its lightweight version can reduce the consumption of computing resources while ensuring the accuracy of face detection. MobileNetV3 further optimizes the operational efficiency of the expression recognition model. Since different facial features (such as the shape and texture of the eyes and mouth) are of different importance for recognizing different expressions, the attention module of SENet can dynamically adjust the weight of each feature channel, so that the feature channels that contribute more to expression recognition receive more attention, thereby improving the accuracy of recognition. At the same time, the special depth-separable convolution structure means that the model can complete high-precision sentiment analysis with lower computational overhead. Regarding the specific structure and application of the YOLOv10 model and the MobileNetV3 model, ordinary technicians in this field can refer to the relevant technical literature when the model was released, which will not be repeated here.

[0123] On the other hand, the present invention also provides an interview evaluation device, see Figure 6 , Figure 6It is a principle block diagram of an interview evaluation device according to an embodiment of the present invention, the interview evaluation device includes a video acquisition module 11, a text transcription module 12, a semantic vector generation module 13, an expression vector generation module 14 and an evaluation module 15, the video acquisition module 11 is used to acquire the target interview video; the text transcription module 12 is used to generate a speech transcription interview text of the target interview video through the speech transcription service of the large model, and the speech transcription interview text includes the interview candidate text and the interviewer text; the semantic vector generation module 13 is used to generate corresponding semantic vectors based on the interview candidate text and the interviewer text respectively; the expression vector generation module 14 is used to generate multiple expression vectors representing different specified expressions based on the interview video; the evaluation module 15 is used to combine the semantic vectors and multiple expression vectors of the interview parties into input vectors of the interview evaluation model, and input them to the interview evaluation model, and the interview evaluation model outputs the interview score, wherein the interview evaluation model is a neural network model trained with the interview video set as the training set.

[0124] In one embodiment, the text transcription module 12 sends the target interview video to the first large model via the voice transcription service interface of the first large model; receives and stores the interview candidate text and interviewer text returned by the first large model; respectively counts the text length of the interview candidate text and the interviewer text; respectively checks whether the text length of the interview candidate text and the interviewer text exceeds the preset length; in response to the text length of the interview candidate text and / or the interviewer text exceeding the preset length, inputs the interview candidate text and / or the interviewer text exceeding the preset length to the second large model via the large model key point generation service interface; receives the key point text returned by the second large model, and uses the key point text as the corresponding interview candidate text and / or interviewer text; wherein, the first large model and the second large model are the same large model or different large models. In addition, when there are multiple interviewers, the voice transcribed interview texts of the multiple interviewers are merged together to form the interviewer text.

[0125] When the semantic vector generation module 13 generates corresponding semantic vectors based on the interview candidate text and the interviewer text respectively, it performs word segmentation processing on the interview candidate text and the interviewer text respectively to obtain their respective numerical text input vocabulary lists; inputs the numerical text input vocabulary lists of the interview candidate and the interviewer respectively into the lightweight semantic model; and obtains the output vectors after the lightweight semantic model processes the numerical text input vocabulary lists of the interview candidate and the interviewer respectively as the semantic vectors of each of the interview parties.

[0126] In addition, when the interview candidate text and the interviewer text are respectively segmented to obtain their respective numerical text input vocabulary, the interview candidate text and the interviewer text are respectively segmented based on the preset BPE vocabulary to obtain their respective segmentation vocabulary; and the segmentations in their respective segmentation vocabulary are converted into segmentation IDs to obtain numerical text input vocabulary; wherein, the BPE vocabulary is a segmentation vocabulary obtained by iterating the BPE algorithm based on the voice transcribed interview text obtained from the interview video.

[0127] In one embodiment, the semantic vector generation module 13 uses the DistilBERT model to generate a semantic vector, and the semantic vector is a multi-dimensional vector, such as a 128-dimensional vector.

[0128] When generating expression vectors representing multiple, respectively, designated expressions based on an interview video, the expression vector generation module 14 first performs frame processing on the target interview video to obtain multiple video frame images; performs face detection on each video frame image to obtain a corresponding face image. In one embodiment, a single-stage target detection model is used to perform face detection on each video frame image to obtain face boundary coordinates; and then performs cropping based on the face boundary coordinates to obtain the corresponding face image. The single-stage target detection model is a lightweight convolutional neural network model trained using a training set, such as the YOLO model. Each face image is then input into an expression recognition model, which recognizes a designated expression corresponding to each face image; the expression recognition model is, for example, a MobileNetV3 model. The proportion of each designated expression recognized in the total number of expressions is then calculated; the proportion of each designated expression is converted into a multidimensional vector to obtain an expression vector corresponding to the designated expression.

[0129] The interview evaluation device of the present invention further includes a model training module 16, such as Figure 7 As shown, Figure 7 This is a block diagram of the model training module principle according to an embodiment of the present invention. The model training module 16 in this embodiment includes a YOLO model training unit, a MobileNetV3 model training unit, a BPE vocabulary building unit, a DistilBERT model training unit, and a neural network model training unit. Each of the above units needs to apply its own training set when training its own model. Figure 8 , Figure 8 1 is a flow chart of a data preparation and preprocessing method according to an embodiment of the present invention, which specifically includes:

[0130] Step S411: Collect video recordings of online or offline interviews. It should be noted that all videos are permitted by the parties for use in patent research and are legal and compliant.

[0131] Step S412: Determine the video label. Specifically, generate a supervisory label for the video based on the interview result corresponding to the video. For example, when the interview result is "passed", the label is "1", and when the interview result is "failed", the label is "0", thereby obtaining an interview video training set, in which each video marked with a supervisory label is regarded as a sample.

[0132] Step S413: divide the video into frames and save them as static images. Specifically, divide the video into frames and save them as static images through OpenCV, thereby obtaining an image library.

[0133] Step S414: transcribe all videos into text to build a text library. For example, the Tongyi Tingwu service interface provided by Alibaba Big Model can be used to transcribe all videos into text for building a BPE (Byte Pair Encoding) vocabulary.

[0134] For the YOLO model training unit and the MobileNetV3 model training unit, in order to train the YOLO model and the MobileNetV3 model, the training set is first determined. Specifically, a small number of video frame images are taken from different interview videos in the static image library to construct training samples, so as to increase the diversity of samples as much as possible. In order to improve the accuracy of expression recognition, the imbalance of training samples in terms of age, gender, etc. is improved by data enhancement. For example, when the proportion of female candidates in the sample is small, the number of female video images is translated and rotated by plus or minus 5 degrees, and the processed images are added as new samples to the current training sample set. In one embodiment, the total sample size in the training set is set to 100,000, and the size of each video frame image is modified to 640*640. Then, the face area annotation box is marked in the training sample, and the expression category label is marked. There are three expression categories in this implementation, namely "nervous", "calm" and "confident".

[0135] This example uses the YOLOv10-s network architecture, balancing speed and accuracy. Training is performed for 100 epochs with a batch size of 64 and Adam as the optimizer. For the MobileNetV3 model, this example uses the small version. Similarly, training is performed for 100 epochs with a batch size of 64 and Adam as the optimizer.

[0136] In addition, in order to improve the robustness and generalization of the model, Gaussian noise and salt and pepper noise can be randomly added to the samples when constructing them to cope with the situation of poor video resolution.

[0137] The BPE vocabulary building unit uses the text library to build the BPE vocabulary. For the specific process, see Figure 4 And the description will not be repeated here.

[0138] The DistilBERT model training unit uses interview texts that have been segmented using the BPE vocabulary as samples. The model can learn the semantic content vector representing the entire text through the self-attention mechanism (Self-Attention) module, and the output vector dimension is 128.

[0139] In one embodiment, the neural network model includes an input layer, four hidden layers with dimensions of 512, 256, 64, and 32, respectively, an output layer with one node, and a softmax function to normalize the output to [0, 1]. The loss function is cross entropy.

[0140] Figure 9 This is a flow chart of the interview evaluation model training method according to one embodiment of the present invention. First, the neural network model is iteratively trained based on the training set samples. A training process of iteratively training the neural network model is as follows: Figure 9 As shown, a video sample is first selected. Each image in the image set obtained by the corresponding frame processing is sequentially processed by the dashed line in the figure to obtain the corresponding facial expression category. The YOLO model is used to identify the facial image, and the MobileNetV3 model is used to identify the expression category. After processing all images, the proportions of the three expression categories are calculated, and then binning is performed to obtain a 32-dimensional vector for each expression category. While processing the image set, the interview candidate text and interviewer text in the video sample are each subjected to word segmentation processing using the BPE vocabulary to obtain the input vector of the DistilBERT model. After further processing by the DistilBERT model, a 128-dimensional semantic vector is output. The three 32-dimensional expression vectors and two 128-dimensional semantic vectors are combined to form the input vector of the neural network model and are input to the neural network model. The input data is passed layer by layer from the input layer of the neural network model to the output layer. In this embodiment, the output layer outputs one-dimensional data, which is a value ranging from 0 to 1, representing the probability of passing the interview. The difference between the model prediction and the true label is then measured by calculating the loss function. Backpropagation is used to calculate the gradient of the loss with respect to each parameter based on the chain rule to guide parameter updates. Adaptive optimizers, such as Adam and RMSprop, are then used to dynamically adjust the learning rate for parameter optimization.

[0141] After a certain number of iterations, the neural network model is verified using validation set samples, optimized based on the verification results, and finally evaluated using test set samples. Once the evaluation criteria are met, the neural network model is saved and deployed to the server environment to ensure stable operation. Of course, further comprehensive stress testing can be conducted to evaluate the performance of the overall interview assessment system under high concurrency conditions to ensure the reliability and stability of the service. In order to further improve the practicality and fit of the system, people who have served as interviewers are invited to participate in the testing. Through the professional experience of the interviewers, possible inconveniences in the system or inconsistencies with the actual interview process are discovered, feedback is collected in a timely manner, and the system is adjusted and optimized based on the feedback results to ensure the efficiency of the recruitment platform and its fit for actual applications.

[0142] Figure 10 1 is a flow chart of an interview evaluation method according to a specific embodiment of the present invention. The interview evaluation method in this embodiment includes the following steps:

[0143] Step S41: Data preparation and preprocessing. Figure 8 And the description will not be repeated here.

[0144] Step S42: Model building and training. Specifically, the YOLO model is used as a single-stage object detection model to perform face detection on each video frame, the MobileNetV3 model is used for expression recognition, the DistilBERT model is used to obtain semantic vectors, and a neural network model is used as the interview evaluation model.

[0145] In step S43, the target video is fed into the interview evaluation device for inference, and the resulting score is received. The score ranges from 0 to 1. If multiple candidates participate in an interview, the scores can be used as a ranking basis.

[0146] Then, based on user feedback, the aforementioned models are continuously optimized, and each model is fine-tuned through incremental data to improve the accuracy of the evaluation method of the present invention.

[0147] This method not only assesses a candidate's professional competence by considering the alignment between the interviewer's questions and the candidate's responses, but also assesses their psychological quality by determining whether they appear predominantly "nervous" or "confident" during the interview. It also integrates verbal expression and facial expressions to provide a more comprehensive evaluation framework. This data-driven approach, combined with historical recruitment data, uncovers patterns in successful hiring decisions, overcoming the limitations of manually set evaluation dimensions. This approach effectively improves the comprehensiveness, accuracy, and scientific nature of interview evaluations, making it applicable to a variety of candidates and scenarios.

[0148] The present invention cleverly utilizes the capabilities of large models, deep learning models, etc., simplifies the processing tasks of the model, and thus can use a model with a relatively simple structure and a small parameter scale to achieve accurate prediction, with low computational pressure, reduced consumption of computing resources, lowered requirements for hardware performance, and low cost. For example, a single-stage target detection model such as the YOLOv10 face detection model can accurately extract the face area in the video frame, ensure the accuracy of expression recognition, and provide a strong visual basis for evaluating the emotional response of the candidate. The MobileNetV3 used in the embodiment of the present invention is an efficient convolutional neural network that can perform in-depth analysis of the extracted face area and accurately capture the emotional changes of the candidate during the interview (such as nervousness, calmness or confidence) through expression recognition and classification, providing important data support for the emotional analysis in the interview evaluation. At the same time, after the video is transcribed into text in combination with the voice transcription service of the large model (such as Tongyi Tingwu), the speech content of different speakers in the interview can be accurately distinguished, enhancing the understanding of the interview process. Finally, the DistilBERT model's semantic vectorization of the transcribed text improves the depth and accuracy of semantic content analysis, so that the interview evaluation is not only limited to the surface language content, but also can fully understand the deep semantics behind the candidate's language. The present invention combines a large model with a semantic model, so that the present invention can undertake the two complex tasks of video transcription and text understanding with smaller model parameters. The DistilBERT model distilled by Bert simplifies the model structure, further reduces the memory and computing burden, and makes the overall process of the interview evaluation method of the present invention run smoothly in actual use. In short, the present invention enhances the data diversity and analysis depth in the interview evaluation process, and ensures efficient calculation and fast response by lightweighting each model, while avoiding large-scale hardware investment. It is practical, convenient, low-cost, and meets the needs of actual applications.

[0149] In another aspect, the present invention further provides an electronic device, see Figure 11 , Figure 11 This is a schematic diagram of the hardware structure principle of an electronic device according to an embodiment of the present invention. The electronic device can be implemented as a server or various other terminal devices, such as a desktop personal computer, a tablet computer, a laptop computer, a mobile phone, etc., which includes a processor 601 and a memory 602. The memory 602 stores a program instruction set, and the aforementioned interview evaluation method is implemented when the processor 601 executes the program instruction set on the memory 602.

[0150] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0151] The memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 602 may include removable or non-removable (or fixed) media. Where appropriate, the memory 602 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 602 is a non-volatile solid-state memory.

[0152] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it implements the aforementioned interview evaluation method.

[0153] In one example, the electronic device may further include a communication interface 603 and a bus 604. The processor 601, the memory 602, and the communication interface 603 are connected via the bus 604 and communicate with each other.

[0154] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiment of the present invention.

[0155] Bus 604 includes hardware, software or both, and the components of online data flow metering equipment are coupled to each other. For example, but not limitation, bus can include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnect (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 604 can include one or more buses. Although the embodiment of the present invention describes and shows a specific bus, the present invention considers any suitable bus or interconnection.

[0156] The present invention also provides a computer-readable storage medium having computer program instructions stored thereon, which can implement the interview evaluation method in the aforementioned embodiment when the computer program instructions can be executed by a processor. The computer-readable storage medium can be any tangible medium that contains or stores computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. The storage medium can be a transient computer-readable storage medium or a non-transient computer-readable storage medium. Non-transient computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include, for example, magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like.

[0157] The present invention also provides a computer program product comprising a set of computer program instructions that, when executed by a processor, implement the interview evaluation method of the aforementioned embodiment. The computer program product includes, but is not limited to, an application installation package published on a website or in an app store, an application plug-in, or a mini-program that can be run within certain applications.

[0158] The above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the scope of the present invention. Therefore, all equivalent technical solutions should also fall within the scope of the present invention.

Claims

1. An interview evaluation method comprising: Obtaining a target interview video, wherein the target interview video includes at least a video image of the interview candidate and voices of the interview candidate and the interviewer; Generate a speech transcription interview text of the target interview video through the speech transcription service of the first large model, wherein the speech transcription interview text includes the interview candidate text and the interviewer text; Generate corresponding semantic vectors based on the interview candidate text and the interviewer text respectively; Generate multiple expression vectors corresponding to different specified expressions based on the interview video; as well as The semantic vectors and multiple expression vectors of the two interviewees are combined into the input vector of the interview evaluation model and input into the interview evaluation model, and the interview evaluation model outputs the interview score, wherein the interview evaluation model is a neural network model trained with the interview video training set.

2. The interview evaluation method according to claim 1, wherein: When there are multiple interviewers, the interview texts of the multiple voice transcriptions are merged together to form the interviewer text.

3. The interview evaluation method according to claim 1 or 2, wherein: The steps for generating the speech transcription interview text of the target interview video through the speech transcription service of the first large model include: Send the target interview video to the first model via the voice transcription service interface of the first model; Receive and store the interview candidate text and interviewer text returned by the first model; Count the length of the candidate's text and the interviewer's text respectively; Check whether the text length of the candidate text and the interviewer text exceeds the preset length respectively; In response to the text length of the interview candidate text and / or the interviewer text exceeding a preset length, inputting the interview candidate text and / or the interviewer text exceeding the preset length to the second large model via the large model key point generation service interface; and Receive the key text returned by the second large model, and use the key text as the corresponding interview candidate text and / or interviewer text; wherein the first large model and the second large model are the same large model or different large models.

4. The interview evaluation method according to claim 1, wherein: The steps of generating corresponding semantic vectors based on the interview candidate text and the interviewer text include: Perform word segmentation processing on the interview candidate text and the interviewer text respectively to obtain their respective numerical text input word lists; Inputting the interview candidate digitized text input vocabulary and the interviewer digitized text input vocabulary into the lightweight semantic model respectively; and Obtain the output vector of the lightweight semantic model after processing the interview candidate's digitized text input vocabulary as the semantic vector of the interview candidate's text, and obtain the output vector of the lightweight semantic model after processing the interviewer's digitized text input vocabulary as the semantic vector of the interviewer's text; The lightweight semantic model is a semantic processing model trained with a training data set, and the training samples in the training data set are interview texts obtained by voice transcription of interview videos.

5. The interview evaluation method according to claim 4, wherein: The steps of performing word segmentation processing on the interview candidate text and the interviewer text to obtain respective numerical text input vocabulary include: Based on the preset BPE vocabulary, the interview candidate text and the interviewer text are segmented to obtain their respective segmentation vocabulary; and Convert the words in each word list into word IDs to obtain a numerical text input word list; Among them, the BPE vocabulary is a word segmentation table obtained by iterating the BPE algorithm based on the voice transcribed interview text obtained from the interview video.

6. The interview evaluation method according to claim 1, wherein: The steps of generating expression vectors representing multiple expressions corresponding to different specified expressions based on the interview video include: Perform frame processing on the target interview video to obtain multiple video frame images; Perform face detection on each video frame image to obtain the corresponding face image; Each facial image is input to the expression recognition model, and the expression recognition model recognizes a specified expression corresponding to each facial image; Count the proportion of each recognized specific expression in the total number of expressions; and Convert the proportion of each specified expression into a multi-dimensional vector to obtain an expression vector corresponding to the specified expression; The expression recognition model is a lightweight convolutional neural network model that has been trained using a training set, and each training sample in the training set has a label for a specified expression.

7. The interview evaluation method according to claim 6, wherein: There are three categories of designated expressions, and each designated expression is a basic expression representing the candidate's interview status determined based on historical interview data.

8. The interview evaluation method according to claim 6, wherein: The steps of performing face detection on each video frame image to obtain a corresponding face image include: Using a single-stage object detection model to perform face detection on each video frame image to obtain the face boundary coordinates; and Cropping is performed based on the face boundary coordinates to obtain the corresponding face image; Among them, the single-stage target detection model is a lightweight convolutional neural network model trained with a training set.

9. The interview evaluation method according to claim 6, wherein: Before performing face detection on each video frame image, it also includes: Verify the resolution of each video frame; and In response to a resolution of the video frame image being less than a threshold, Gaussian noise and salt and pepper noise are added to the video frame image.

10. The interview evaluation method according to claim 1, wherein: The semantic vector and the expression vector are respectively multidimensional vectors, the difference between the number of dimensions of the semantic vector and the number of dimensions of the expression vector is less than a first threshold, or the ratio of the number of dimensions of the semantic vector to the number of dimensions of the expression vector is less than a second threshold.

11. An interview evaluation device, comprising: A video acquisition module, configured to acquire a target interview video; A text transcription module is configured to generate a speech transcription interview text of the target interview video through a speech transcription service of a large model, wherein the speech transcription interview text includes an interview candidate text and an interviewer text; a semantic vector generation module configured to generate corresponding semantic vectors based on the interview candidate text and the interviewer text; an expression vector generation module configured to generate expression vectors representing a plurality of expression vectors corresponding to different specified expressions based on the interview video; as well as The evaluation module is configured to combine the semantic vectors and multiple expression vectors of the two interview parties into the input vector of the interview evaluation model and input it to the interview evaluation model, and output the interview score through the interview evaluation model, wherein the interview evaluation model is a neural network model trained with the interview video set as the training set.

12. An electronic device comprising a processor and a memory, wherein the memory stores a computer program instruction set, characterized in that: When the processor executes the computer program instruction set on the memory, the interview evaluation method described in any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program instruction set, wherein the computer program instruction set, when executed by a processor, implements the interview evaluation method described in any one of claims 1 to 10.

14. A computer program product comprising a set of computer program instructions, characterized in that When the computer program instruction set is executed by a processor, the interview evaluation method according to any one of claims 1 to 10 is implemented.

Images