Intelligent interview method and system based on human-computer interaction

Abstract

The invention discloses an intelligent interview method and system based on man-machine interaction, and relates to the technical field of man-machine interaction, man-machine interaction equipment synchronously acquires language content and voice emotion content, constructs multi-modal information based on matching of current expression, language content and voice emotion content in the same time dimension, and performs interview on the basis of the multi-modal information. And the interview event of the interviewee is determined according to the intelligent identification of the multi-modal information, so that the accuracy of the multi-modal information is improved, and the accuracy of the interview event of the interviewee is improved. Therefore, the interview matching degree is determined according to the interview cue word, the answer content of the interviewee and the corresponding post function; the psychological quality level of the interviewee is determined based on the traversal of the interview event of the interviewee, and the intelligent interview report of the interviewee is determined according to the psychological quality level of the interviewee, the interview matching degree and the previous work experience of the interviewee, so that the intelligent degree of intelligent interview of the interviewee is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to an intelligent interview method and system based on human-computer interaction. Background Art

[0002] With the development of science and technology, human-computer interaction devices are gradually applied to the interview field and act as intelligent interviewers. The interviewees conduct round-robin questions and answers with the human-computer interaction devices. In the existing technology, existing human-computer interaction devices rely more on fixed question banks or simple NLP matching methods to conduct interviews. They only conduct interviews in text-simulated interview scenarios and ignore the interviewees' expressions. They are not compatible with factors such as current expressions, language content, and voice emotional content, which affects the accuracy of the interviewees' interview events and thus affects the intelligence level of the interviewees' intelligent interviews. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide an intelligent interview method and system based on human-computer interaction.

[0004] The embodiment of the present invention provides an intelligent interview method based on human-computer interaction, comprising: Collecting facial images based on human-computer interaction equipment and determining key facial features based on facial image recognition; Determine the current expression based on the detection of key facial features, optimize the current expression based on the matching of the current expression with previous expression data, and mark the corresponding expression score; The human-computer interaction device synchronously collects language content and voice emotion content, constructs multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determines the interview event of the interviewee based on the intelligent recognition of this multimodal information; During the interview of the interviewee, the corresponding interview questions are collected, and interview prompts are determined based on the interviewee's information and the interview questions. The interview matching degree is determined based on the interview prompts, the interviewee's answers and the corresponding job functions; The psychological quality level of the interviewee is determined based on the traversal of the interviewee's interview events, and the intelligent interview report of the interviewee is determined according to the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience.

[0005] An embodiment of the present invention provides an intelligent interview system based on human-computer interaction. The intelligent interview system based on human-computer interaction is applied to the above-mentioned intelligent interview method based on human-computer interaction. The intelligent interview system based on human-computer interaction includes: A facial key feature module is used to collect facial images based on a human-computer interaction device and determine facial key features based on recognition of the facial images; The expression module is used to determine the current expression based on the detection of key facial features, optimize the current expression based on the matching of the current expression and previous expression data, and mark the corresponding expression score; The interview event module is used for the human-computer interaction device to synchronously collect language content and voice emotion content, construct multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determine the interview event of the interviewee based on the intelligent recognition of this multimodal information; The interview matching module is used to collect corresponding interview questions during the interview of the interviewee, determine the interview prompt words based on the interviewee's information and the interview questions, and determine the interview matching degree based on the interview prompt words, the interviewee's answers and the corresponding job functions; The intelligent interview report module is used to determine the interviewee's psychological quality level based on the traversal of the interviewee's interview events, and to determine the interviewee's intelligent interview report based on the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience.

[0006] Compared with the prior art, the present invention has the following beneficial effects: In an embodiment of the present invention, through the method in the embodiment of the present invention, a facial image is collected based on a human-computer interaction device, and key facial features are determined based on the recognition of the facial image; the current expression is determined based on the detection of key facial features, the current expression is optimized based on the matching of the current expression and previous expression data, and the corresponding expression score is marked; the human-computer interaction device synchronously collects language content and voice emotion content, constructs multimodal information based on the matching of the current expression, language content, and voice emotion content in the same time dimension, and determines the interview event of the interviewee based on the intelligent recognition of the multimodal information, introduces the current expression, and is compatible with the matching of the current expression, language content, and voice emotion content in the same time dimension, thereby improving the accuracy of the multimodal information and improving the accuracy of the interview event of the interviewee.

[0007] Therefore, in the interview event of the interviewee, the corresponding interview questions are collected, and the interview prompt words are determined according to the interviewee's information and the interview questions, and the interview matching degree is determined according to the interview prompt words, the interviewee's answer content and the corresponding job function; the interviewee's psychological quality level is determined based on the traversal of the interviewee's interview events, and the interviewee's intelligent interview report is determined according to the interviewee's psychological quality level, the interview matching degree and the interviewee's previous work experience, which realizes the overall consideration of the interviewee's psychological quality level, the interview matching degree and the interviewee's previous work experience, improves the accuracy of the interviewee's intelligent interview report, and improves the intelligence level of the interviewee's intelligent interview. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 1 is a flow chart of an intelligent interview method based on human-computer interaction in an embodiment of the present invention; Figure 2 1 is a flow chart of step S11 in the intelligent interview method based on human-computer interaction in an embodiment of the present invention; Figure 3 1 is a flow chart of step S12 in the intelligent interview method based on human-computer interaction in an embodiment of the present invention; Figure 4 1 is a flow chart of step S13 in the intelligent interview method based on human-computer interaction in an embodiment of the present invention; Figure 5 1 is a flow chart of step S14 in the intelligent interview method based on human-computer interaction in an embodiment of the present invention; Figure 6 1 is a flow chart of step S15 in the intelligent interview method based on human-computer interaction in an embodiment of the present invention; Figure 7 It is a schematic diagram of the structural composition of the intelligent interview system based on human-computer interaction in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0010] See also Figures 1 to 7 , an intelligent interview method based on human-computer interaction, applied to intelligent interview scenarios; the intelligent interview method based on human-computer interaction includes: Step S11: collecting a facial image based on a human-computer interaction device, and determining key facial features based on recognition of the facial image; Step S12: determining the current expression based on the detection of key facial features, optimizing the current expression based on the matching of the current expression with previous expression data, and marking the corresponding expression score; Step S13: The human-computer interaction device synchronously collects language content and voice emotion content, constructs multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determines the interview event of the interviewee based on intelligent recognition of the multimodal information; Step S14: In the interview event of the interviewee, corresponding interview questions are collected, and interview prompts are determined based on the interviewee's information and the interview questions. The interview matching degree is determined based on the interview prompts, the interviewee's answers and the corresponding job responsibilities; Step S15: determining the interviewee's psychological quality level based on the traversal of the interviewee's interview events, and determining an intelligent interview report for the interviewee based on the interviewee's psychological quality level, interview matching degree, and the interviewee's previous work experience; refer to Figure 2 In step S11, a facial image is collected based on a human-computer interaction device, and key facial features are determined based on recognition of the facial image; In the specific implementation process of the present invention, the specific steps are: S111: The interviewee is within the shooting range of the camera of the machine interaction device. The camera shoots the interviewee and collects a facial image of the interviewee. A plurality of facial regions are determined based on the facial image, and the regional shapes and regional positions of the plurality of facial regions are marked. S112: In each face area, a corresponding feature detection method is matched based on the regional morphology and regional position of the face area, and feature detection is performed on the face area based on the feature detection method to determine the corresponding facial features; based on the recognition of multiple facial features, the corresponding feature type is determined, and the key facial features are screened out.

[0011] In an embodiment of the present application, the person to be interviewed is within the shooting range of the camera of the machine interaction device. The camera shoots the person to be interviewed and collects the facial image of the person to be interviewed. Based on the division of the facial image, multiple facial areas are determined, and the regional morphology and regional position of the multiple facial areas are marked, which is compatible with the overall consideration of the division of the facial image and ensures the accuracy of the multiple facial areas.

[0012] At this time, the camera needs to have a sufficient field of view to ensure that the facial image of the interviewee can be fully captured; the interviewee needs to face the camera, and the face should occupy the center of the image to ensure that a high-quality facial image is captured; the camera's resolution and frame rate need to be high enough to ensure image clarity and smoothness; for example, use a camera with 1080P resolution and 30fps frame rate; the ambient light needs to be moderate to avoid excessive or weak light affecting the image quality.

[0013] The camera collects image data in real time through video streaming; each frame of the image is transmitted to the processing system for subsequent analysis and processing; during the image acquisition process, it is necessary to ensure that the image resolution and frame rate meet the requirements to ensure image quality and real-time performance; a face detection algorithm (such as OpenCV's Haar feature classifier, Dlib's face detector, MTCNN or YOLO based on deep learning, etc.) is used to process the collected face images; the face areas in the image are identified, and the location and size of each face area are returned; if there are multiple interviewees in the image, each face area will be detected separately.

[0014] Specifically, assume that the interviewee Xiao Li is sitting in a brightly lit room, facing a high-definition camera; the camera's field of view can fully capture Xiao Li's face, and the facial image is clear and unobstructed; the camera collects Xiao Li's facial image in real time at a frame rate of 30fps; the resolution of each frame is 1920x1080 pixels to ensure that the image is clear enough; the MTCNN algorithm is used to process Xiao Li's facial image; the MTCNN algorithm detects a clear face area in the image and returns the location and size of the area: upper left corner coordinates: (100,100); width: 300 pixels; height: 300 pixels.

[0015] For each facial region, record its shape (such as the width and height of the rectangular frame) and position (such as the coordinates of the upper left corner). This information will be used for subsequent feature detection and analysis. If there are multiple people to be interviewed, the shape and position of each facial region need to be marked separately.

[0016] Specifically, for Xiao Li's facial area, the following information is marked: area shape: width: 300 pixels; height: 300 pixels; area position: upper left corner coordinates: (100,100); the system can accurately capture the facial image of the interviewee, divide the facial area, and mark the shape and position of each facial area. This information will provide a basis for subsequent feature detection and expression analysis.

[0017] Furthermore, in each facial area, a corresponding feature detection method is matched based on the regional morphology and regional position of the facial area, and feature detection is performed on the facial area based on the feature detection method to determine the corresponding facial features; the corresponding feature types are determined based on the recognition of multiple facial features, and the key facial features are screened out, which is compatible with the overall consideration of the recognition of multiple facial features and ensures the accuracy of the corresponding feature types.

[0018] At this time, the appropriate feature detection algorithm is selected based on the shape (such as size and proportion) and position (such as position in the image) of the face area. Different feature detection algorithms are suitable for different face areas. For example, for high-resolution face areas, complex feature detection algorithms (such as deep learning-based models) can be used, while for low-resolution or partially occluded face areas, lightweight algorithms (such as Dlib) can be used. The selection of feature detection algorithms needs to consider the balance between computational efficiency and detection accuracy.

[0019] Specifically, assume that in step S111, we detect the facial area of ​​Xiao Li as follows: upper left corner coordinates: (100, 100); width: 300 pixels; height: 300 pixels; since the face area is large and clear, we can choose to use MediaPipe's FaceMesh model for feature detection because it can detect 468 high-precision facial key points.

[0020] Use the selected feature detection algorithm to process the face area and extract facial feature points, which include key points of eyebrows, eyes, nose, mouth, etc. The feature detection algorithm returns the coordinates of each key point (usually normalized coordinates or pixel coordinates).

[0021] Specifically, MediaPipe's FaceMesh model is used to perform feature detection on Xiao Li's facial area; the FaceMesh model returns the coordinates of 468 facial key points, for example: inner corner of the left eye: (150,150); inner corner of the right eye: (250,150); tip of the nose: (200,200); corners of the mouth: (180,250), (220,250).

[0022] Based on the detected feature points, different types of features are identified; for example, feature points can be divided into the following categories: eye features: including key points such as the corners of the eyes and eyelids; eyebrow features: including the starting point, end point and middle point of the eyebrows; nose features: including key points such as the tip of the nose and the wings of the nose; mouth features: including key points such as the corners of the mouth and the outline of the lips; each type of feature point has its own unique distribution and function, which is used for subsequent expression analysis; the key features that are most valuable for expression analysis and identity recognition are screened out from the detected feature points; the selection of key features needs to consider their sensitivity to changes in expression; for example, feature points such as the ups and downs of the eyebrows, the degree of opening and closing of the eyes, and the ups or downs of the corners of the mouth are very important for expression analysis; the screened key features will be used for subsequent expression recognition and psychological state analysis.

[0023] Specifically, for Xiao Li's facial feature points, we screened out the following key features: Eyebrow key features: left eyebrow midpoint: (150,130); right eyebrow midpoint: (250,130); Eye key features: left eye inner corner: (150,150); right eye inner corner: (250,150); Mouth corner key features: left mouth corner: (180,250); right mouth corner: (220,250); The system can extract key feature points from the detected facial area and screen out the most valuable features for expression analysis and identity recognition. These key features will provide important data support for subsequent expression recognition and psychological state analysis.

[0024] refer to Figure 3 In step S12, the current expression is determined based on the detection of key facial features, the current expression is optimized based on the matching of the current expression and previous expression data, and the corresponding expression score is marked; In the specific implementation process of the present invention, the specific steps are: S121: Collecting multiple key facial features, determining multiple facial feature combinations based on the feature positions and feature forms of the multiple key facial features, determining feature parameters corresponding to the key facial features based on the recognition of the multiple facial feature combinations, and determining a current expression based on the feature parameters corresponding to the key facial features and the feature positions and forms of the key facial features; S122: Collecting the user's personal information, determining previous expression data based on the user's personal information and matching with the information database, matching the current expression with the previous expression data, and outputting the corresponding expression area to be optimized; S123: Optimizing the current expression based on the synthesis of the expression area to be optimized and the current expression to output the optimized current expression, and marking the corresponding expression score based on the recognition of the optimized current expression.

[0025] In an embodiment of the present application, multiple key facial features are collected, multiple facial feature combinations are determined based on the feature positions and feature forms of the multiple key facial features, feature parameters corresponding to the key facial features are determined based on the identification of the multiple facial feature combinations, and the current expression is determined based on the feature parameters corresponding to the key facial features, the feature positions and forms of the key facial features, which is compatible with the overall consideration of the feature parameters corresponding to the key facial features, the feature positions and forms of the key facial features, and ensures the accuracy of the current expression.

[0026] At this point, in step S112, we have extracted the key feature points of the face through feature detection algorithms (such as MediaPipeFaceMesh). These key feature points include the specific positions (coordinates) of eyebrows, eyes, nose, mouth and other parts; the coordinates of key feature points are usually expressed in normalized form, ranging from 0 to 1, and can also be converted into pixel coordinates; according to the position and shape of the key feature points, they are combined into meaningful feature combinations; feature combinations can include: eyebrow combination: the starting point, end point and midpoint of the eyebrows; eye combination: key points such as corners of the eyes and eyelids; mouth corner combination: the position and angle of the corners of the mouth. These combinations will be used for subsequent expression analysis.

[0027] Specifically, eyebrow combination: midpoint of the left eyebrow: (0.3, 0.2); midpoint of the right eyebrow: (0.7, 0.2); eye combination: inner corner of the left eye: (0.3, 0.3); inner corner of the right eye: (0.7, 0.3); mouth corner combination: left corner of the mouth: (0.4, 0.7); right corner of the mouth: (0.6, 0.7).

[0028] Feature parameters refer to the specific numerical values ​​of each feature combination, which are used to quantify the changes in features; eyebrow parameters: the inclination angle of the eyebrows and the distance of up and down movement; eye parameters: the degree of eye openness or closure (calculated by the distance between the corners of the eyes and the eyelids); mouth corner parameters: the degree of uplift or drooping of the corners of the mouth (calculated by the distance between the corners of the mouth and the nose); based on the feature parameters, feature position and morphology, a pre-trained expression recognition model (such as a CNN-based model) is used to recognize the current expression; the expression recognition model will output the category of the current expression (such as happiness, sadness, anger, etc.); the input of the model can be a combination of feature parameters or a feature vector extracted directly from the image.

[0029] Specifically, the feature parameters are: eyebrow parameters: tilt angle 0 degrees, up and down movement distance 0; eye parameters: open and closed degree 100%; mouth corner parameters: upward degree 10%; use a pre-trained expression recognition model (such as a CNN-based model); input the feature parameters into the model, and the model outputs the current expression category as "smile".

[0030] Furthermore, the user's personal information is collected, and the previous expression data is determined based on the matching of the user's personal information and the information database. The current expression and the previous expression data are matched, and the corresponding expression area to be optimized is output, which is compatible with the overall consideration of the matching of the user's personal information and the information database, and ensures the accuracy of the previous expression data.

[0031] At this time, the user's personal information may include age, gender, occupation, cultural background, etc. This information can be obtained through the form filled out by the user when registering, or through other channels (such as social media, internal corporate databases); the collection of personal information is to better understand the user's facial expression characteristics, because people of different ages, genders and cultural backgrounds may have different facial expression habits; at the same time, based on the user's personal information, the user's historical facial expression data is searched in the information database; historical facial expression data may include the user's facial expression feature parameters and facial expression categories in different situations.

[0032] Specifically, assume that Xiao Li's personal information is as follows: Age: 25 years old; Gender: Male; Occupation: Software Engineer; Assume that Xiao Li's historical expression data is stored in the database as follows: Historical Expression 1: Situation: Interview; Expression Category: Smile; Feature Parameters: Eyebrow Tilt Angle: 0 degrees; Eyes Opening Degree: 100%; Mouth Corner Upturn Degree: 20%; Historical Expression 2: Situation: Interview; Expression Category: Neutral; Feature Parameters: Eyebrow Tilt Angle: 0 degrees; Eyes Opening Degree: 100%; Mouth Corner Upturn Degree: 0%.

[0033] Match the feature parameters of the current expression with historical expression data to find similar expression patterns; matching can be achieved by calculating the similarity between feature parameters (such as Euclidean distance); identify the differences between the current expression and historical expressions, and determine the expression area to be optimized; based on the matching results, determine the area in the current expression that needs to be optimized; for example, if the current degree of upward movement of the corners of the mouth is lower than the standard value in the historical data, the corner area of ​​the mouth needs to be optimized; determining the area to be optimized can help subsequent expression optimization algorithms adjust expressions more accurately.

[0034] Specifically, assume that the characteristic parameters of the current expression are as follows: eyebrow tilt angle: 0 degrees; eye openness: 100%; mouth corner upturned degree: 10%; matching with historical expression 1 (smile): eyebrow tilt angle: 0 degrees (match); eye openness: 100% (match); mouth corner upturned degree: 10% (lower than 20% of historical data); the current expression's mouth corner upturned degree is 10%, which is 20% lower than the standard value of historical smiling expressions; expression area to be optimized: mouth corner area: the mouth corner upturned degree needs to be increased.

[0035] Therefore, the current expression is optimized based on the synthesis of the expression area to be optimized and the current expression to output the optimized current expression, and the corresponding expression score is marked based on the recognition of the optimized current expression. The recognition of the optimized current expression is introduced to mark the corresponding expression score.

[0036] At this point, in step S122, we have determined the areas in the current expression that need to be optimized, such as the corners of the mouth; in step S122, we have determined the areas in the current expression that need to be optimized, such as the corners of the mouth; Optimization method: linear interpolation: if the current feature parameters are lower than the standard value, they can be gradually adjusted to the target value through linear interpolation; the optimized expression feature parameters will be closer to the standard value in the historical expression data; the optimized feature parameters can be used for subsequent expression recognition and analysis; the optimized current expression is re-identified using a pre-trained expression recognition model; the expression recognition model will output the category and confidence (or score) of the current expression; the expression score can be a numerical value indicating the similarity between the current expression and the standard expression.

[0037] Specifically, the area to be optimized: the corner of the mouth area; the current feature parameters: the degree of upward movement of the corners of the mouth: 10%; the historical standard value: the degree of upward movement of the corners of the mouth: 20%; the optimization method: adjust the degree of upward movement of the corners of the mouth from 10% to 20%; the optimized feature parameters: the tilt angle of the eyebrow: 0 degrees; the degree of eye opening and closing: 100%; the degree of upward movement of the corners of the mouth: 20%; the optimized current expression: smile; the expression recognition model: the pre-trained CNN model; the input: the optimized feature parameters; the output: the expression category: smile; the expression score: 90 points (indicating that it is very similar to the standard smile expression); the system can adjust the feature parameters of the current expression according to the expression area to be optimized, and re-recognize the optimized expression through the expression recognition model, and finally mark the corresponding expression score. This process not only improves the accuracy of expression recognition, but also provides more reliable data support for subsequent expression analysis and psychological state assessment.

[0038] refer to Figure 4 In step S13, the human-computer interaction device synchronously collects language content and voice emotion content, constructs multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determines the interview event of the interviewee based on the intelligent recognition of the multimodal information; In the specific implementation process of the present invention, the specific steps are: S131: The human-computer interaction device monitors the interviewee and collects interaction data of the interviewee, determines language content and voice emotion content based on the screening of the interaction data of the interviewee, and marks the collection time of the current expression, the collection time of the language content, and the collection time of the voice emotion content; S132: In the time dimension, time matching is performed on the current expression, language content, and voice emotion content, and multimodal information is constructed while ensuring time synchronization; S133: Determine a plurality of sub-interview events based on the recognition of the multimodal information, and determine an interview event for the interviewee based on a combination of the plurality of sub-interview events.

[0039] In an embodiment of the present application, the human-computer interaction device monitors the interviewee and collects the interaction data of the interviewee, determines the language content and voice emotion content based on the screening of the interaction data of the interviewee, and marks the collection time of the current expression, the collection time of the language content, and the collection time of the voice emotion content, which is compatible with the overall consideration of the screening of the interaction data of the interviewee and ensures the accuracy of the language content and voice emotion content.

[0040] At this time, human-computer interaction equipment (such as cameras, microphones, computers, etc.) is used to monitor the interviewees in real time; the camera is used to capture the interviewees' facial expressions and gestures; the microphone is used to capture the interviewees' voice content; the computer is used to process and analyze the collected data; the equipment needs to have high resolution and high frame rate to ensure the accuracy and real-time nature of the data.

[0041] Interaction data includes video data, audio data, and text data; video data is used to capture facial expressions and gestures; audio data is used to capture voice content; text data is transcribed into text using speech recognition technology; data collection needs to ensure real-time and accuracy; audio data is transcribed into text content using speech recognition technology; speech emotion recognition technology is used to analyze the emotional characteristics of speech, such as intonation, speaking speed, pauses, etc.; screening method: Use a speech recognition API (such as the Alibaba Cloud speech recognition API) to transcribe audio data into text; use a speech emotion recognition model (such as extracting audio features based on OpenSMILE and combining it with a CNN+LSTM model) to analyze the emotional content of speech.

[0042] Specifically, let's assume that Xiao Li is participating in an interview, and the camera and microphone are collecting his video and audio data in real time. The camera has a resolution of 1080P and a frame rate of 30fps. The microphone has a sampling rate of 16kHz, which can clearly capture the voice content. Video data: The camera captures Xiao Li's facial expressions and gestures. Audio data: The microphone captures Xiao Li's voice content. Text data: The audio is transcribed into text using speech recognition technology. Audio data: Xiao Li's response is captured by the microphone. Speech recognition: The speech recognition API transcribes the audio into text content: "I have three years of software development experience." Speech sentiment analysis: The speech sentiment analysis results show that Xiao Li's emotion is "confident," with a high emotion score. Mark the acquisition timestamp for each modal data to ensure subsequent time alignment; the timestamp includes: the acquisition time of the current expression: the timestamp of the video frame; the acquisition time of the language content: the time interval of the speech transcription; the acquisition time of the speech emotion content: the timestamp of the audio frame; the timestamp marking needs to ensure high precision, usually using millisecond timestamps.

[0043] Specifically, the collection time of the current expression is: 00:01:22.500; the collection time of language content is: 00:01:22.000 to 00:01:25.000; the collection time of voice emotion content is: 00:01:22.000 to 00:01:25.000; the system can collect the interviewee's multimodal data (expression, voice, text) in real time, and mark the collection time of each data. These data will provide the basis for subsequent time alignment and multimodal information fusion.

[0044] Furthermore, in the time dimension, the current expression, language content, and voice emotion content are time-matched, and multimodal information is constructed while ensuring time synchronization, which introduces the construction of multimodal information while ensuring time synchronization.

[0045] At this point, align the timestamps of the current facial expressions, language content, and voice emotion content to ensure that they are in the same time dimension; select a suitable time window (such as 0.2 seconds or 0.5 seconds) and align the data within this window; alignment method: sliding window: slide at a fixed time interval (such as 0.2 seconds) to extract the data within each window; or, timestamp matching: match the timestamps of the facial expressions, language, and emotion data within each window to ensure that their time difference is within the allowable range (such as ±0.1 seconds).

[0046] Specifically, within the time window from 00:01:22.000 to 00:01:22.200: Expression: smile (timestamp 00:01:22.500); Language content: I have three years of software development experience (timestamp 00:01:22.000); Voice emotion: confident (timestamp 00:01:22.000); The aligned expression, language content and voice emotion data are combined into a unified multimodal information structure. JSON or other structured formats can be used to store multimodal information; A unified timestamp is selected as the time marker of the multimodal information. The timestamp of the expression data is usually selected because it is usually the most accurate; The system can time-align the collected multimodal data (expression, language, emotion) and construct a unified multimodal information structure. These multimodal information will provide the basis for subsequent interview event recognition and analysis.

[0047] Therefore, multiple sub-interview events are determined based on the recognition of multimodal information, and the interview event of the interviewee is determined based on the combination of multiple sub-interview events. This is compatible with the overall consideration of the combination of multiple sub-interview events, ensuring the accuracy of the interview event of the interviewee. At the same time, the current expression is introduced, which is compatible with the matching of the current expression, language content, and voice emotion content in the same time dimension, thereby improving the accuracy of multimodal information and the accuracy of the interview event of the interviewee.

[0048] At this time, the multimodal information includes data such as facial expressions, language content, and voice emotions, which have been time-aligned (step S132); a sub-interview event refers to the specific behavior or reaction of the interviewee at a specific time point or time period during the interview process; for example, "showing confidence when answering questions" or "appearing nervous when asking questions"; at the same time, a pre-trained machine learning model (such as a Transformer-based model) is used to analyze the multimodal information; the model can identify specific sub-interview events based on the combination of facial expressions, language content, and voice emotions; for example, if the model detects that the interviewee has a confident expression, fluent language, and stable voice emotions when answering questions, it can be identified as "showing confidence when answering questions."

[0049] An interview event refers to the comprehensive performance of the interviewee within a certain period of time during the interview process; it is composed of multiple sub-interview events; at the same time, multiple sub-interview events are combined according to the time sequence to form a complete interview event, and different interview events can be divided into time windows (such as 1 minute); for example, if within 1 minute, the interviewee is confident when answering questions, seems a little nervous when asking questions, but regains confidence when summarizing, then these sub-events can be combined into one interview event: "shows confidence and professionalism when answering technical questions, but seems a little nervous when asking questions"; the system can analyze multimodal information (expressions, language, emotions), identify multiple sub-interview events, and combine these sub-events into a complete interview event. This process can not only capture the specific behaviors and reactions of the interviewee during the interview process, but also provide the interviewer with a more comprehensive evaluation basis.

[0050] refer to Figure 5 In step S14, in the interview event of the interviewee, the corresponding interview questions are collected, and interview prompts are determined based on the interviewee's information and the interview questions. The interview matching degree is determined based on the interview prompts, the interviewee's answers and the corresponding job functions; In the specific implementation process of the present invention, the specific steps are: S141: Real-time monitoring of interview events of the interviewee, collecting corresponding interview questions based on the analysis of the interview events of the interviewee, collecting information of the interviewee, determining the interview position based on the analysis of the interviewee's information, and determining interview prompts based on the job responsibilities of the interview position and the interview questions; S142: Collect the answers of the interviewee, determine a first matching coefficient based on the answers of the interviewee and the interview prompts, determine a second matching coefficient based on the answers of the interviewee and the corresponding job functions, and determine the interview matching degree based on the first matching coefficient and the second matching coefficient.

[0051] In an embodiment of the present application, the interview events of the interviewees are monitored in real time, and corresponding interview questions are collected based on the analysis of the interview events of the interviewees. At the same time, the information of the interviewees is collected, the interview position is determined based on the analysis of the information of the interviewees, and the interview prompt words are determined based on the functions of the interview position and the interview questions, which is compatible with the overall consideration of the functions of the interview position and the interview questions, and ensures the accuracy of the interview prompt words.

[0052] At this point, the system needs to capture multimodal data (such as facial expressions, voice, language content, etc.) during the interview process in real time and parse out specific interview events; an interview event refers to the behavior or reaction of the interviewee within a certain period of time during the interview, such as "showing confidence when answering questions"; the collected data is analyzed through a pre-trained multimodal analysis model (such as a Transformer-based model) to identify specific interview events; at the same time, the corresponding interview questions are collected. Interview questions refer to questions raised by human-computer interaction devices during the interview; the system needs to record each interview question and its corresponding timestamp for subsequent analysis; and through timestamp matching, the interview question corresponding to each interview event is determined.

[0053] Collect the information of the interviewees, which includes their basic information (such as name, age, gender, work experience, educational background, etc.). This information can be pre-entered into the interview system or dynamically collected during the interview process. By parsing this information, key information can be extracted for subsequent analysis. At the same time, the interview position refers to the position applied for by the interviewee. By parsing the interviewee's information (such as work experience, educational background, etc.), the interview position can be determined. Keyword matching (such as "software engineer", "senior software engineer") can be used to determine the position.

[0054] At the same time, interview prompts refer to prompts generated based on the functions of the interview position and the interview questions, which are used to guide the language model to analyze the interviewee's answers; prompts can be fixed templates or dynamically adjusted according to specific circumstances; for example: "Please evaluate the interviewee's technical ability and logical thinking based on the following answers: '{interviewee's answer}'"; "Please evaluate the interviewee's job suitability based on the following answers: '{interviewee's answer}'"; the system can monitor interview events in real time, collect corresponding interview questions and interviewee information, determine the interview position, and generate interview prompts, which will be used for subsequent language model analysis to evaluate the match between the interviewee's answers and the position.

[0055] Therefore, the answers of the interviewees are collected, and the first matching coefficient is determined based on the answers of the interviewees and the interview prompts. The second matching coefficient is determined based on the answers of the interviewees and the corresponding job functions. The interview matching degree is determined based on the first matching coefficient and the second matching coefficient, which is compatible with the overall consideration of the first matching coefficient and the second matching coefficient to ensure the accuracy of the interview matching degree.

[0056] At this time, the answer content refers to the interviewee's answer to the interview questions during the interview; the interviewee's voice answer is transcribed into text content through voice recognition technology; redundant characters and stop words are removed to ensure the accuracy and readability of the text content; for the first matching coefficient, the interview prompt word refers to the prompt word generated according to the functions of the interview position and the interview question, which is used to guide the language model to analyze the interviewee's answer; the first matching coefficient refers to the matching degree between the interviewee's answer content and the interview prompt word; use a pre-trained language model (such as LLM) to analyze the interviewee's answer and generate a scoring label; the scoring criteria can include multiple dimensions such as technical ability, logical thinking, and language expression.

[0057] For the second matching coefficient, job function refers to the specific duties and requirements of the interview position; the second matching coefficient refers to the matching degree between the interviewee's answers and the job function; the degree of fit between the interviewee's answers and the job function is evaluated through keyword matching, semantic analysis, etc.; the scoring criteria may include multiple dimensions such as technical ability, experience matching, and job adaptability.

[0058] Regarding the interview matching degree, the interview matching degree refers to the final score obtained by combining the first matching coefficient and the second matching coefficient. The final matching degree can be calculated by weighted average or other methods. The weights of the first matching coefficient and the second matching coefficient can be adjusted according to actual needs. Optionally, assuming that the weight of the first matching coefficient is 0.5 and the weight of the second matching coefficient is 0.5, the interview matching degree is: Interview matching degree = 0.5*first matching coefficient + 0.5*second matching coefficient.

[0059] In one embodiment of the present application, the interview prompt is: "Please evaluate the interviewee's technical ability and logical thinking based on the following answer: 'I have three years of software development experience, mainly using Java and Python for project development'"; a first score matching table is collected, and the first score matching table is shown in Table 1: Table 1 First score matching table

[0060] The first matching coefficient: Assuming the weight distribution is: technical ability 0.5, logical thinking 0.3, language expression 0.2; the first matching coefficient = 0.5*1.0+0.3*0.7+0.2*0.9=0.88.

[0061] Use keyword matching and semantic analysis to assess the fit between the interviewee's answers and the job function. Use a matching table to quantify the fit of each dimension. For the job function requirement: "Proficient in Java and Python, with more than three years of software development experience," collect the second score matching table. The second score matching table is as follows: Table 2 Second score matching table

[0062] Assume that the weight distribution is: technical ability match 0.5, experience match 0.3, job adaptability 0.2; the second matching coefficient = 0.5*1.0+0.3*1.0+0.2*0.9=0.98; assume that the weight of the first matching coefficient is 0.5, and the weight of the second matching coefficient is 0.5; interview matching degree = 0.5*first matching coefficient + 0.5*second matching coefficient; interview matching degree = 0.5*0.88+0.5*0.98=0.93.

[0063] refer to Figure 6 In step S15, the psychological quality level of the interviewee is determined based on the traversal of the interview events of the interviewee, and the intelligent interview report of the interviewee is determined according to the psychological quality level of the interviewee, the interview matching degree and the interviewee's previous work experience; In the specific implementation process of the present invention, the specific steps are: S151: Collecting interview events of the interviewee and traversing the interview events of the interviewee to collect the interviewee's psychological performance behavior, and determining the interviewee's psychological quality level based on the identification of the psychological performance behavior; S152: Determine a first interview report based on the interviewee's psychological quality level and interview matching degree, determine a second interview report based on the interviewee's psychological quality level and the interviewee's previous work experience, and determine an intelligent interview report of the interviewee based on the synthesis of the first interview report and the second interview report.

[0064] In an embodiment of the present application, interview events of the interviewee are collected and traversed to collect the interviewee's psychological performance behaviors, and the interviewee's psychological quality level is determined based on the identification of the psychological performance behaviors, which is compatible with the overall consideration of the identification of the psychological performance behaviors and ensures the accuracy of the interviewee's psychological quality level.

[0065] At this time, the interview events of the interviewee are collected, and video and audio data are collected using a camera and microphone; the audio data is transcribed into text content through voice recognition technology; expression recognition and emotion analysis technology are used to extract the psychological performance behavior of the interviewee; the collected interview events are stored in a structured data format, such as JSON; each interview event is checked one by one, and the psychological performance behavior of the interviewee is extracted; psychological performance behaviors include confidence, tension, anxiety, hesitation, etc., which can be obtained through expression recognition, voice emotion analysis and other technologies; pre-trained models (such as expression recognition models and voice emotion recognition models based on deep learning) are used to identify psychological performance behaviors.

[0066] Based on the identification results of psychological performance behaviors, the psychological quality level of the interviewee is determined; level classification: high: confident behaviors account for more than 70%; medium: confident behaviors account for between 30%-70%; low: confident behaviors account for less than 30%; count the frequency of confident behaviors and calculate their proportion in the total behaviors, optional, number of confident behaviors: 2; total number of behaviors: 3; proportion of confident behaviors = (2 / 3)*100%≈66.67%; psychological quality level: medium.

[0067] Furthermore, the first interview report is determined based on the interviewee's psychological quality level and interview matching degree, the second interview report is determined based on the interviewee's psychological quality level and the interviewee's previous work experience, and the intelligent interview report of the interviewee is determined based on the synthesis of the first interview report and the second interview report, which is compatible with the overall consideration of the synthesis of the first interview report and the second interview report, ensuring the accuracy of the intelligent interview report of the interviewee. At the same time, it realizes the overall consideration of the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience, improves the accuracy of the interviewee's intelligent interview report, and improves the intelligence level of the interviewee's intelligent interview.

[0068] At this time, for the first interview report, the psychological quality level: has been determined in step S151, indicating the psychological state of the interviewee during the interview; the interview matching degree: has been calculated in step S142, indicating the degree of matching between the interviewee's answer and the job requirements; the psychological quality level and the interview matching degree are combined to evaluate the overall performance of the interviewee in the interview to output the first interview report; report content: psychological quality level: high, medium, low; interview matching degree: a value between 0 and 1; advantages: highlights of the interviewee's performance in the interview; improvement suggestions: suggestions for the interviewee's shortcomings.

[0069] For the second interview report, psychological quality level: has been determined in step S151; previous work experience: the work experience and background information of the interviewee; comprehensively consider the psychological quality level and previous work experience, and evaluate the suitability of the interviewee's background and position to output the second interview report; report content: psychological quality level: high, medium, low; previous work experience: specific work experience and background; position suitability: high, medium, low; strengths: highlights of the interviewee's work experience; improvement suggestions: suggestions for the interviewee's shortcomings.

[0070] The final interview report is generated by combining the contents of the first interview report and the second interview report; report content: Name of the interviewee: the name of the interviewee; Psychological quality level: high, medium, low; Interview matching: a value between 0 and 1; Previous work experience: specific work experience and background; Job adaptability: high, medium, low; Strengths: highlights of the interviewee's performance; Improvement suggestions: suggestions for the interviewee's shortcomings; The system can generate a detailed intelligent interview report based on the psychological quality level, interview matching and previous work experience. This report not only evaluates the interviewee's performance in the interview, but also provides targeted improvement suggestions to help the interviewer understand the interviewee more comprehensively.

[0071] See also Figure 7 , Figure 7 : is a schematic diagram of the structure of an intelligent interview system based on human-computer interaction in an embodiment of the present invention; the intelligent interview system based on human-computer interaction includes: The facial key feature module 21 is used to collect facial images based on the human-computer interaction device and determine the facial key features based on the recognition of the facial images; Expression module 22, for determining the current expression based on the detection of key facial features, optimizing the current expression based on the matching of the current expression with previous expression data, and marking the corresponding expression score; Interview event module 23, which is used for the human-computer interaction device to synchronously collect language content and voice emotion content, construct multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determine the interview event of the interviewee based on the intelligent recognition of the multimodal information; The interview matching module 24 is used to collect corresponding interview questions during the interview of the interviewee, determine interview prompts based on the interviewee's information and the interview questions, and determine the interview matching degree based on the interview prompts, the interviewee's answers and the corresponding job responsibilities; The intelligent interview report module 25 is used to determine the interviewee's psychological quality level based on the interviewee's interview events, and determine the interviewee's intelligent interview report according to the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience.

[0072] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An intelligent interview method based on human-computer interaction, characterized in that: include: Collecting facial images based on human-computer interaction equipment and determining key facial features based on facial image recognition; Determine the current expression based on the detection of key facial features, optimize the current expression based on the matching of the current expression with previous expression data, and mark the corresponding expression score; The human-computer interaction device synchronously collects language content and voice emotion content, constructs multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determines the interview event of the interviewee based on the intelligent recognition of this multimodal information; During the interview of the interviewee, the corresponding interview questions are collected, and interview prompts are determined based on the interviewee's information and the interview questions. The interview matching degree is determined based on the interview prompts, the interviewee's answers and the corresponding job functions; The psychological quality level of the interviewee is determined based on the traversal of the interviewee's interview events, and the intelligent interview report of the interviewee is determined according to the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience.

2. The intelligent interview method based on human-computer interaction according to claim 1, characterized in that: The method of collecting a facial image based on a human-computer interaction device and determining key facial features based on recognition of the facial image includes: The interviewee is within the shooting range of the camera of the machine interaction device, and the camera shoots the interviewee and collects a facial image of the interviewee, determines multiple facial regions based on the facial image, and marks the regional shapes and regional positions of the multiple facial regions; In each facial area, the corresponding feature detection method is matched based on the regional morphology and regional position of the facial area, and feature detection is performed on the facial area based on the feature detection method to determine the corresponding facial features; the corresponding feature type is determined based on the recognition of multiple facial features, and the key facial features are screened out.

3. The intelligent interview method based on human-computer interaction according to claim 1, characterized in that: The method of determining the current expression based on the detection of key facial features, optimizing the current expression based on the matching of the current expression with previous expression data, and marking the corresponding expression score includes: Collecting multiple key facial features, determining multiple facial feature combinations based on the feature positions and feature forms of the multiple key facial features, determining feature parameters corresponding to the key facial features based on the recognition of the multiple facial feature combinations, and determining the current expression based on the feature parameters corresponding to the key facial features and the feature positions and forms of the key facial features; Collect the user's personal information, determine the previous expression data based on the user's personal information and the information database, match the current expression with the previous expression data, and output the corresponding expression area to be optimized; The current expression is optimized based on the synthesis of the expression area to be optimized and the current expression to output the optimized current expression, and the corresponding expression score is marked based on the recognition of the optimized current expression.

4. The intelligent interview method based on human-computer interaction according to claim 1, characterized in that: The human-computer interaction device synchronously collects language content and voice emotion content, constructs multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determines the interview event of the interviewee based on intelligent recognition of the multimodal information, including: The human-computer interaction device monitors the interviewee and collects the interviewee's interaction data, determines the language content and voice emotion content based on the screening of the interviewee's interaction data, and marks the collection time of the current expression, the collection time of the language content, and the collection time of the voice emotion content.

5. The intelligent interview method based on human-computer interaction according to claim 4 is characterized in that: The human-computer interaction device synchronously collects language content and voice emotion content, constructs multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determines the interview event of the interviewee based on intelligent recognition of the multimodal information, further comprising: In the time dimension, the current expression, language content, and voice emotion content are time-matched, and multimodal information is constructed while ensuring time synchronization; Based on the recognition of multimodal information, a plurality of sub-interview events are determined, and an interview event of the interviewee is determined according to a combination of the plurality of sub-interview events.

6. The intelligent interview method based on human-computer interaction according to claim 1, characterized in that: In the interview event of the interviewee, the corresponding interview questions are collected, and interview prompts are determined based on the interviewee's information and the interview questions. The interview matching degree is determined based on the interview prompts, the interviewee's answers and the corresponding job functions, including: Monitor the interview events of the interviewees in real time, collect corresponding interview questions based on the analysis of the interview events of the interviewees, and at the same time, collect the information of the interviewees, determine the interview position based on the analysis of the interviewees' information, and determine the interview prompts based on the functions of the interview position and the interview questions.

7. The intelligent interview method based on human-computer interaction according to claim 6, characterized in that: In the interview event of the interviewee, corresponding interview questions are collected, and interview prompts are determined based on the interviewee's information and the interview questions. The interview matching degree is determined based on the interview prompts, the interviewee's answers and the corresponding job functions, and further includes: Collect the answers of the interviewees, determine the first matching coefficient based on the answers of the interviewees and the interview prompts, determine the second matching coefficient based on the answers of the interviewees and the corresponding job functions, and determine the interview matching degree based on the first matching coefficient and the second matching coefficient.

8. The intelligent interview method based on human-computer interaction according to claim 1, characterized in that: The method of determining the interviewee's psychological quality level based on the interviewee's interview events and determining the interviewee's intelligent interview report based on the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience includes: Collect interview events of the interviewees, and traverse the interview events of the interviewees to collect the psychological performance behaviors of the interviewees, and determine the psychological quality level of the interviewees based on the identification of the psychological performance behaviors.

9. The intelligent interview method based on human-computer interaction according to claim 8, characterized in that: The method of determining the interviewee's psychological quality level based on the interviewee's interview events and determining the interviewee's intelligent interview report based on the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience also includes: The first interview report is determined based on the interviewee's psychological quality level and interview matching degree, the second interview report is determined based on the interviewee's psychological quality level and the interviewee's previous work experience, and the intelligent interview report of the interviewee is determined based on the synthesis of the first interview report and the second interview report.

10. An intelligent interview system based on human-computer interaction, characterized in that: The intelligent interview system based on human-computer interaction is applied to the intelligent interview method based on human-computer interaction according to any one of claims 1 to 9, and the intelligent interview system based on human-computer interaction includes: A facial key feature module is used to collect facial images based on a human-computer interaction device and determine facial key features based on recognition of the facial images; The expression module is used to determine the current expression based on the detection of key facial features, optimize the current expression based on the matching of the current expression and previous expression data, and mark the corresponding expression score; The interview event module is used for the human-computer interaction device to synchronously collect language content and voice emotion content, construct multimodal information based on the matching of current expression, language content, and voice emotion content in the same time dimension, and determine the interview event of the interviewee based on the intelligent recognition of this multimodal information; The interview matching module is used to collect corresponding interview questions during the interview of the interviewee, determine the interview prompt words based on the interviewee's information and the interview questions, and determine the interview matching degree based on the interview prompt words, the interviewee's answers and the corresponding job functions; The intelligent interview report module is used to determine the interviewee's psychological quality level based on the traversal of the interviewee's interview events, and to determine the interviewee's intelligent interview report based on the interviewee's psychological quality level, interview matching degree and the interviewee's previous work experience.

Images