A method and system for remote interactive identity verification and anomalous behavior identification

Abstract

The application relates to a remote interactive identity authentication and abnormal behavior identification method and system, and relates to the technical field of remote online identity recognition. A reference face feature template and a reference voiceprint feature vector template are generated according to an identity card image and initial audio data uploaded by a candidate; a face feature vector is extracted from video stream data and compared with the reference face feature template to determine a switching frequency; an audio feature vector extracted from audio stream data is compared with the reference voiceprint feature vector template to determine an audio abnormality duration; face key points are extracted from the video stream data to determine a line-of-sight abnormality duration; an interviewee's answer speech is determined based on an interviewer's question speech to determine an interaction abnormality duration; the switching frequency, the audio abnormality duration, the line-of-sight abnormality duration and the interaction abnormality duration are calculated, an evaluation result is generated according to a comprehensive risk value, and the evaluation result is sent to an interviewer terminal. The provided technical scheme improves the authenticity of an interview result.

Description

Technical Field

[0001] This application relates to the field of remote online identity recognition technology, specifically to a method and system for remote interactive identity verification and abnormal behavior recognition. Background Technology

[0002] With the widespread adoption of remote work and global recruitment, remote video interviews have become a crucial part of talent screening for companies. How to efficiently and accurately verify candidate identities and ensure the fairness and authenticity of the interview process in a contactless online environment has become a key challenge for the field of human resource technology.

[0003] To address integrity issues in remote settings, existing technologies primarily employ online examination proctoring systems. These systems utilize a range of robust control techniques, such as forcing full-screen mode, locking browsers, monitoring screen sharing, and using cameras for environmental scanning, to strictly restrict user devices and behaviors to prevent cheating.

[0004] However, the highly intrusive monitoring methods in the aforementioned technologies inherently conflict with the natural and open communication atmosphere required in an interview setting. The core purpose of an interview is not only to assess a candidate's professional knowledge, but also to evaluate their communication skills, quick thinking, and on-the-spot responsiveness. Forced full-screen monitoring, environmental scanning, and other high-pressure monitoring measures can place immense psychological pressure on candidates, putting them in a tense and unnatural state, thus severely impacting the demonstration of their true abilities. Therefore, existing technologies cannot provide candidates with a non-intrusive interview experience while ensuring high integrity, leading to distorted evaluation results. Summary of the Invention

[0005] This application provides a method and system for remote interactive authentication and abnormal behavior recognition. This method solves the problem in the prior art that high integrity monitoring and good user experience cannot be achieved at the same time. While ensuring the fairness of remote interviews, it provides candidates with a non-intrusive interview environment and improves the authenticity of the evaluation results.

[0006] In a first aspect, this application provides a method for remote interactive identity verification and abnormal behavior identification. The method includes: receiving an ID card image uploaded by a candidate and real-time acquisition of the candidate's facial video information and initial audio data; generating a baseline facial feature template based on the ID card image and facial video information, and processing the initial audio data to obtain a baseline voiceprint feature vector template; real-time acquisition of video stream data and audio stream data during remote interaction, extracting facial feature vectors from the video stream data, continuously comparing the facial feature vectors with the baseline facial feature template to identify candidate identity switching events, and determining the number of switching events based on the number of times the candidate identity switching events occur; and comparing the audio feature vectors extracted from the audio stream data with the baseline voiceprint feature vector template. A comparison is performed to identify non-reference voiceprints that do not match the reference voiceprint feature vector template, and the duration of non-reference voiceprints is accumulated to determine the duration of audio anomalies. Facial key points are extracted from video stream data to estimate the candidate's gaze direction, and the time when the gaze direction deviates from the preset normal gaze range is accumulated to determine the duration of gaze anomalies. Based on the interviewer's questions and the candidate's answers, the response delays in multiple question-and-answer cycles are determined, and the duration when the response delay is greater than the response delay threshold is accumulated to determine the duration of interaction anomalies. The number of switching, duration of audio anomalies, duration of gaze anomalies, and duration of interaction anomalies are calculated to obtain a comprehensive risk value. An evaluation result is generated based on the comprehensive risk value and sent to the interviewer.

[0007] By adopting the above technical solution, the system receives the ID card image uploaded by the candidate and collects facial video information and initial audio data in real time. A baseline facial feature template and a baseline voiceprint feature vector template are established before the interview begins. Throughout the remote interaction process, video and audio stream data are collected in real time. Facial feature vectors extracted from the video stream data are continuously compared with the baseline facial feature template to identify candidate identity switching events, effectively preventing proxy testing. Audio feature vectors extracted from the audio stream data are compared with the baseline voiceprint feature vector template, and the duration of non-baseline voiceprints is accumulated to determine the duration of audio anomalies, effectively identifying whether third-party assistance in answering questions exists. Facial key features are extracted from the video stream data. By estimating the candidate's gaze direction and accumulating the time the gaze deviates from the preset normal gaze range, it can determine whether the candidate frequently checks off-screen prompts. Based on the interviewer's questions and the candidate's answers, it determines the response delay in each of multiple question-and-answer cycles, and accumulates the duration of response delays greater than a response delay threshold to determine the duration of interaction anomalies, identifying whether the candidate answered after receiving off-site guidance. It calculates a comprehensive risk value based on the number of switching, the duration of audio anomalies, the duration of gaze anomalies, and the duration of interaction anomalies, and generates an evaluation result that is sent to the interviewer. This allows the interviewer to obtain a complete evaluation of the interview performance while also grasping the candidate's integrity data, thereby making a more accurate judgment.

[0008] Optionally, a baseline facial feature template is generated based on the ID card image and facial video information. The initial audio data is then processed to obtain a baseline voiceprint feature vector template. Specifically, this includes: extracting the ID card image from the ID card image; performing liveness detection on the facial video information to determine if the candidate is a live object; if the liveness detection passes, extracting the ID card image's facial feature vector and the real-time facial feature vector from the facial video information; calculating the similarity between the ID card image's facial feature vector and the real-time facial feature vector to obtain an initial similarity score; when the initial similarity score is greater than a preset similarity score, confirming the candidate's identity verification, selecting multiple facial image frames from the facial video information, and fusing the basic facial feature vectors of the multiple facial image frames to obtain a baseline facial feature template; collecting initial audio data of the candidate reading a preset text; performing speech activity detection on the initial audio data to obtain valid speech segments; extracting acoustic features from the valid speech segments to obtain voiceprint feature vectors, and generating a baseline voiceprint feature vector template based on the voiceprint feature vectors.

[0009] By employing the above technical solutions, facial images are extracted from ID card images, and liveness detection is performed on facial video information to confirm that the candidate is a living person, effectively preventing the possibility of fraud using static photos. After the liveness detection is passed, the facial feature vectors from the ID card image and the real-time facial feature vectors from the facial video information are extracted and similarity calculations are performed to obtain an initial similarity score, ensuring that the person participating in the interview is indeed the same person as the ID card holder, avoiding impersonation. When the initial similarity score is greater than a preset similarity score, the candidate's identity verification is confirmed. Multiple facial image frames are selected from the facial video information, and the basic facial feature vectors of these facial image frames are fused to obtain a baseline facial feature template, reducing the misjudgment rate caused by changes in environmental factors. Initial audio data of the candidate reading a preset text is collected, and voice activity detection is performed to obtain valid speech segments. Acoustic feature extraction is performed on the valid speech segments to obtain voiceprint feature vectors, and a baseline voiceprint feature vector template is generated, which can accurately identify voices that are not the candidate's own, effectively preventing third-party proxy answers.

[0010] Optionally, facial feature vectors are extracted from the video stream data, and the facial feature vectors are continuously compared with a baseline facial feature template to identify candidate identity switching events. The number of switching events is determined based on the number of times the candidate identity switching events occur. Specifically, this includes: selecting video stream data based on a preset frequency to obtain the current face image frame to be detected; extracting facial feature vectors from the current face image frame to be detected and calculating the similarity between the facial feature vectors and the baseline facial feature template to obtain a real-time similarity score; when the real-time similarity score is greater than or equal to a preset matching threshold, it is determined as a single match; when the real-time similarity score is less than the preset matching threshold, it is determined as a single mismatch; counting the consecutive number of single mismatches, and when the consecutive number reaches a preset number, it is confirmed as a candidate identity switching event; counting the number of candidate identity switching events during remote interaction and using the number of occurrences as the number of switching events.

[0011] By adopting the above technical solution, the current face image frame to be detected is selected from the video stream data, avoiding the resource consumption caused by feature extraction and comparison for each video frame. The face feature vector is extracted from the current face image frame to be detected and its similarity is calculated with the baseline face feature template to obtain a real-time similarity score, in order to deal with cheating methods such as mid-process personnel replacement. When the real-time similarity score is less than the preset matching threshold, it is judged as a single mismatch. The number of consecutive single mismatches is counted. Only when the number of consecutive mismatches reaches a preset number is it confirmed as a candidate identity switching event. The continuous judgment process effectively avoids misjudgment caused by occasional factors of the candidate and greatly improves the accuracy of identity switching recognition. The number of times the candidate identity switching event occurs during remote interaction is counted and the number of occurrences is used as the switching count. The switching count, as an objective numerical indicator, can more fairly reflect whether the candidate has engaged in proxy testing and the degree of cheating during the interview process.

[0012] Optionally, the audio feature vector extracted from the audio stream data is compared with a reference voiceprint feature vector template to identify non-reference voiceprints that do not match the reference voiceprint feature vector template, and the duration of non-reference voiceprints is accumulated to determine the duration of audio anomalies. Specifically, this includes: performing speech activity detection on the audio stream data to obtain multiple real-time speech segments; extracting acoustic features from each real-time speech segment to obtain real-time voiceprint feature vectors; calculating the distance between the real-time voiceprint feature vector and the reference voiceprint feature vector template to obtain a real-time voiceprint distance value; determining whether the real-time voiceprint distance value is greater than a preset voiceprint distance threshold; when the real-time voiceprint distance value is less than or equal to the preset voiceprint distance threshold, marking the real-time speech segment corresponding to the real-time voiceprint distance value as a reference voiceprint; when the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, marking the real-time speech segment corresponding to the real-time voiceprint distance value as a non-reference voiceprint; and statistically summing the durations of all non-reference voiceprint segments identified in remote interaction to obtain the duration of audio anomalies.

[0013] By employing the above technical solution, multiple real-time speech segments are obtained through voice activity detection of audio stream data, achieving accurate extraction of valid speech signals and avoiding the inclusion of environmental noise in the voiceprint analysis scope. This ensures the accuracy of subsequent voiceprint comparison. Acoustic features are extracted from each real-time speech segment to obtain real-time voiceprint feature vectors, and the distance between the real-time voiceprint feature vector and the baseline voiceprint feature vector template is calculated to obtain the real-time voiceprint distance value. When the real-time voiceprint distance value is less than or equal to a preset voiceprint distance threshold, it is marked as a baseline voiceprint; when the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, it is marked as a non-baseline voiceprint. This effectively identifies whether there is third-party assistance in answering questions or off-site guidance during the interview process. The duration of all non-baseline voiceprint segments identified in remote interaction is statistically summed to obtain the duration of audio anomalies, so as to distinguish between occasional interference and systematic cheating behavior in the future.

[0014] Optionally, facial key points are extracted from the video stream data to estimate the candidate's gaze direction, and the time during which the gaze direction deviates from a preset normal gaze range is accumulated to determine the duration of gaze abnormality. Specifically, this includes: extracting the candidate's facial key points from the video stream data in real time; calculating the candidate's head pose angle based on the facial key points and a preset 3D face model; determining whether the head pose angle continuously exceeds the preset normal gaze range; when the time the head pose angle continuously exceeds the preset normal gaze range reaches a preset single duration threshold, confirming it as a gaze abnormality event and recording the duration of the gaze abnormality event; and statistically summing the durations of all gaze abnormality events identified during remote interaction to obtain the duration of gaze abnormality.

[0015] By adopting the above technical solution, facial key points of the candidate are extracted from the video stream data in real time. Based on the facial key points and the preset 3D face model, the candidate's head posture angle is calculated. It is determined whether the head posture angle continuously exceeds the preset normal line of sight range, allowing the candidate to perform normal behavior within a reasonable range. This avoids misjudging natural gaze movement as abnormal behavior. When the head posture angle continuously exceeds the preset normal line of sight range for a period of time that reaches a preset single duration threshold, it is confirmed as a gaze abnormality event and the duration is recorded. Based on the duration, the candidate's normal behavior is effectively filtered out. The duration of all gaze abnormality events identified during remote interaction is statistically summed to obtain the gaze abnormality duration, which reflects whether the candidate has gaze deviation behavior.

[0016] Optionally, based on the interviewer's questions and the candidate's answers, the response latency in each of the multiple question-and-answer cycles is determined, and the duration of the accumulated response latency exceeding a response latency threshold is used to determine the duration of the interaction anomaly. Specifically, this includes: performing voice activity detection on the interviewer's questions and the candidate's answers to determine the end time of the question and the start time of the answer in each question-and-answer cycle, and using the difference between the end time of the question and the start time of the answer as the response latency of the current question-and-answer cycle; collecting the candidate's initial speech in the initial stage, performing speech recognition on the initial speech to obtain the baseline text, and calculating the language complexity of the baseline text to obtain the baseline complexity score; performing speech recognition on the candidate's answers in the current question-and-answer cycle to obtain the answer text, and calculating the language complexity of the answer text to obtain the real-time complexity score; when the response latency exceeds the response latency threshold, and the ratio of the real-time complexity score to the baseline complexity score exceeds a preset complexity gain threshold, the current answer cycle is marked as an interaction anomaly event, and the response latency duration corresponding to the interaction anomaly event is recorded; and the response latency durations of all interaction anomaly events identified during the remote interaction are statistically summed to obtain the duration of the interaction anomaly.

[0017] By adopting the above technical solution, the response delay in the time dimension is combined with the linguistic complexity in the content dimension, which breaks through the limitations of traditional monitoring methods that only focus on surface behavioral characteristics. By deeply analyzing the linguistic features of candidates' answers, the solution effectively identifies covert cheating behaviors, ensuring a high detection rate while effectively controlling the false alarm rate. This provides interviewers with accurate integrity assessment criteria and can accurately identify candidates who attempt to gain an unfair advantage through off-site guidance without interfering with normal interview communication.

[0018] Optionally, a comprehensive risk value is calculated based on the number of switching attempts, the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies. Specifically, this includes: assigning risk weights to the number of switching attempts, the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies, respectively, where the risk weight for the number of switching attempts is greater than the risk weights for the duration of audio anomalies, visual anomalies, and interaction anomalies; multiplying the number of switching attempts by their respective risk weights to obtain a switching risk score; processing the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies using a non-linear mapping function to obtain audio severity coefficients, visual severity coefficients, and interaction severity coefficients; multiplying these coefficients by their respective risk weights to obtain audio risk scores, visual risk scores, and interaction risk scores; and summing the switching risk scores, audio risk scores, visual risk scores, and interaction risk scores to obtain the comprehensive risk value.

[0019] By employing the aforementioned technical solution, risk weights are assigned to the number of switching attempts, duration of audio anomalies, duration of visual anomalies, and duration of interaction anomalies, respectively. The switching attempt count is multiplied by its corresponding risk weight to obtain a switching risk score. Based on a nonlinear mapping function, the duration of audio anomalies, duration of visual anomalies, and duration of interaction anomalies are processed to obtain audio severity coefficients, visual severity coefficients, and interaction severity coefficients, effectively distinguishing between occasional anomalies and systemic cheating behavior. The audio severity coefficient, visual severity coefficient, and interaction severity coefficient are multiplied by their respective risk weights to obtain audio risk scores, visual risk scores, and interaction risk scores, respectively. The switching risk scores, audio risk scores, visual risk scores, and interaction risk scores are then summed to obtain a comprehensive risk value. This achieves a comprehensive quantitative assessment of the candidate's overall integrity. Compared to methods relying solely on a single-dimensional indicator, this approach more comprehensively and accurately reflects the candidate's true integrity level during the interview process, avoiding misjudgments caused by the occasional failure or interference of a particular monitoring dimension.

[0020] A second aspect of this application provides a system for remote interactive identity verification and abnormal behavior recognition. The system includes a receiving unit, a first processing unit, a second processing unit, and an evaluation unit. The receiving unit receives an ID card image uploaded by a candidate and collects the candidate's facial video information and initial audio data in real time. The first processing unit generates a baseline facial feature template based on the ID card image and facial video information, and processes the initial audio data to obtain a baseline voiceprint feature vector template. The second processing unit collects video stream data and audio stream data in real time during remote interaction, extracts facial feature vectors from the video stream data, continuously compares the facial feature vectors with the baseline facial feature template to identify candidate identity switching events, and determines the number of switching events based on the number of times the candidate identity switching events occur. The system also extracts facial feature vectors from the audio stream data. The extracted audio feature vector is compared with a baseline voiceprint feature vector template to identify non-baseline voiceprints that do not match the baseline voiceprint feature vector template, and the duration of non-baseline voiceprints is accumulated to determine the duration of audio anomalies. Facial key points are extracted from video stream data to estimate the candidate's gaze direction, and the time when the gaze direction deviates from the preset normal gaze range is accumulated to determine the duration of gaze anomalies. Based on the interviewer's questions and the candidate's answers, the response delays in multiple question-and-answer cycles are determined, and the duration when the response delay is greater than the response delay threshold is accumulated to determine the duration of interaction anomalies. An evaluation unit calculates the number of switching, the duration of audio anomalies, the duration of gaze anomalies, and the duration of interaction anomalies to obtain a comprehensive risk value. An evaluation result is generated based on the comprehensive risk value and sent to the interviewer.

[0021] In a third aspect, this application provides an electronic device including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory, causing the electronic device to perform any of the methods described above in this application.

[0022] In a fourth aspect, this application provides a computer-readable storage medium storing instructions that, when executed, perform any of the methods described above in this application.

[0023] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. Receive the ID card image uploaded by the candidate and collect facial video information and initial audio data in real time. Establish a baseline facial feature template and a baseline voiceprint feature vector template before the interview begins. Throughout the remote interaction process, collect video and audio stream data in real time. Continuously compare the facial feature vector extracted from the video stream data with the baseline facial feature template to identify candidate identity switching events, effectively preventing proxy testing. Compare the audio feature vector extracted from the audio stream data with the baseline voiceprint feature vector template and accumulate the duration of non-baseline voiceprints to determine the duration of audio anomalies, effectively identifying whether third-party assistance is being used to answer questions. Extract facial key points from the video stream data to estimate... By tracking the candidate's gaze direction and accumulating the time their gaze deviates from a preset normal gaze range, it can be determined whether the candidate frequently checks off-screen prompts. Based on the interviewer's questions and the candidate's answers, the response delays in multiple question-and-answer cycles are determined, and the duration of response delays exceeding a threshold is accumulated to determine the duration of interaction anomalies, identifying whether the candidate answered only after receiving off-site guidance. A comprehensive risk value is calculated based on the number of switching, the duration of audio anomalies, the duration of gaze anomalies, and the duration of interaction anomalies, and an evaluation result is generated and sent to the interviewer. This allows the interviewer to obtain a complete evaluation of the interview performance while also grasping the candidate's integrity data, thus making a more accurate judgment. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a method for remote interactive authentication and abnormal behavior identification provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a remote interactive authentication and abnormal behavior recognition system provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0025] Explanation of reference numerals in the attached drawings: 201, receiving unit; 202, first processing unit; 203, second processing unit; 204, evaluation unit; 300, electronic device; 301, processor; 302, memory; 303, user interface; 304, network interface; 305, communication bus. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0027] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0028] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0029] Therefore, how to address the current dilemma of balancing high-level integrity monitoring with a good user experience in existing technologies is a pressing issue. This application provides a method for remote interactive identity verification and abnormal behavior identification, applied in a server. The server in this application is a platform providing online interview identity assessment services. Figure 1 This is a flowchart illustrating a method for remote interactive authentication and abnormal behavior identification provided in an embodiment of this application. (Refer to...) Figure 1 The method includes the following steps S101-S107.

[0030] S101: Receives the ID card image uploaded by the candidate and collects the candidate's facial video information and initial audio data in real time.

[0031] In S101 above, since interviewers in remote interview scenarios cannot directly verify candidates' physical documents and observe their real appearance as in traditional offline interviews, it is necessary to establish a credible anchor point for the candidate's identity before the interview begins. This credible anchor point can be constructed based on document information, real-time facial features, and voiceprint features. When a candidate enters the interview system, the client first guides the candidate to submit an image of their personal ID card via file upload or camera capture. This ID card image should contain clear and identifiable facial information. The uploaded image undergoes format verification and quality checks to ensure sufficient resolution to support subsequent OCR text recognition and facial feature extraction. If the image has quality issues such as blurriness, glare, occlusion, or low resolution, the candidate is promptly prompted to retake the photo or upload it to avoid affecting the accuracy of subsequent verification due to insufficient raw data quality. The collection of the ID card image provides a baseline of the candidate's legal identity information, including name, ID number, and facial features in the ID photo. This information will be parsed and extracted using OCR technology in subsequent steps and used as a comparison standard to verify whether the real-time face matches the ID holder.

[0032] After acquiring the ID card image, the client's camera is activated to begin real-time acquisition of the candidate's facial video information. This video information uses RGB color video stream format with a resolution of no less than 720p to ensure the accuracy of facial key point detection. A frame rate of at least 15fps is recommended to support subsequent liveness detection and motion response analysis. Real-time acquisition of facial video information is not simply static photography, but continuous video stream capture. This allows observation of the candidate's dynamic facial features over time, providing necessary video sequence data for issuing random action commands and verifying the candidate's real-time responses. Through real-time video stream acquisition, complete action process frame sequences can be captured when the candidate performs actions such as blinking, opening their mouth, or shaking their head. This allows for accurate calculation of the rate of change in eyelid distance or the change in head Euler angles, effectively distinguishing between a real live face and non-liveness attacks such as photos, videos, or masks.

[0033] Initial audio data of the candidate is synchronously acquired via the client's microphone. This audio data is in raw PCM format, with a sampling rate of 16kHz or higher recommended to ensure the frequency resolution required for voiceprint feature extraction. The acquisition of initial audio data is typically accompanied by an initial interaction between the system and the candidate, such as guiding the candidate to give a brief self-introduction or answer pre-set open-ended questions. This process not only completes the audio data acquisition but also reduces the candidate's nervousness through natural conversation, avoiding the discomfort caused by simply issuing recording instructions.

[0034] S102: Generate a baseline facial feature template based on the ID card image and facial video information, and process the initial audio data to obtain a baseline voiceprint feature vector template.

[0035] In step S102 above, a baseline facial feature template is generated based on the ID card image and facial video information. The initial audio data is then processed to obtain a baseline voiceprint feature vector template. Specifically, this includes: extracting the ID card image from the ID card image; performing liveness detection on the facial video information to determine if the candidate is a live object; if the liveness detection passes, extracting the ID card image's facial feature vector and the real-time facial feature vector from the facial video information; calculating the similarity between the ID card image's facial feature vector and the real-time facial feature vector to obtain an initial similarity score; when the initial similarity score is greater than a preset similarity score, confirming the candidate's identity verification, selecting multiple facial image frames from the facial video information, and fusing the basic facial feature vectors of the multiple facial image frames to obtain a baseline facial feature template; collecting initial audio data of the candidate reading a preset text; performing speech activity detection on the initial audio data to obtain valid speech segments; extracting acoustic features from the valid speech segments to obtain a voiceprint feature vector, and generating a baseline voiceprint feature vector template based on the voiceprint feature vector.

[0036] Specifically, a baseline facial feature template is generated based on the ID card image and facial video information. Since the ID card image undergoes further quality checks after uploading, the verification process is prevented from failing due to input data quality issues. For qualified ID card images, grayscale processing is first performed, converting the color RGB image to a single-channel grayscale image to reduce computational complexity. Then, adaptive threshold binarization technology is applied to enhance the contrast of text and image boundaries, making the text and facial photo areas more clearly distinguishable in the image. Next, an OCR text detection model based on a convolutional recurrent neural network is used to perform layout analysis on the binarized image. This model, through multi-scale feature extraction and the sequence modeling capabilities of recurrent neural networks, can accurately locate various functional areas in the ID card image, including the name text area, the ID number text area, and the rectangular bounding boxes of the ID photo area. The detected ID photo area is precisely cropped, extracting a rectangular region of interest (ROI) containing the complete face. This ROI is typically located at a fixed position in the upper left or upper right corner of the ID card. The cropped ID photo area is saved as a separate image file for subsequent processing.

[0037] After obtaining the facial image from the identification document, liveness detection is performed on the facial video information to confirm that the candidate is a living person, preventing candidates from using non-living materials such as printed photos, screen-displayed videos, or highly realistic masks to impersonate real people and pass identity verification. Liveness detection is implemented based on a dynamic interactive verification method. A series of action commands are randomly issued to the candidate. These commands include, but are not limited to, simple facial movements such as blinking, opening the mouth, shaking the head, nodding, and smiling. The randomness of the commands ensures that attackers cannot prepare corresponding video materials in advance. The current timestamp is precisely recorded as the command timestamp when each command is issued. After the commands are issued, frame-by-frame analysis of the real-time video stream begins. A facial landmark detection algorithm is used to process each frame of the image. This algorithm can detect 68 or more key feature points on the face, including the two-dimensional coordinates of locations such as the corners of the eyes, eyelids, tip of the nose, corners of the mouth, and jawline. For blink detection, the eyelid aspect ratio (EAR) is calculated in real time. This value is obtained by dividing the vertical distance between key points on the upper and lower eyelids by the horizontal distance between key points at both ends of the eye. When the eyes are fully open, the EAR value is usually greater than 0.3, and when the eyes are closed, the EAR value drops to below 0.2. By monitoring the complete change process of the EAR value rapidly decreasing from greater than 0.3 to less than 0.2 and then rapidly recovering to greater than 0.3 in a short period of time, it is determined that the candidate has completed a real blink. The dynamic analysis method based on geometric features can effectively distinguish real blinks from subtle movement changes that cannot be reflected in static photos or low-quality videos.

[0038] For head movement commands such as head shaking and nodding, a head pose estimation technique based on the PnP algorithm is employed. This algorithm establishes a correspondence between 2D facial key points and a 3D universal face model, and uses the geometric principles of perspective projection to deduce the rotation angle of the head in three-dimensional space. This includes the yaw angle (representing the angle of left and right head rotation), the pitch angle (representing the angle of up and down head nodding), and the roll angle (representing the angle of left and right head tilting). While the candidate performs a head shaking motion, the yaw angle is continuously calculated for each frame. When a complete yaw process is detected—from near 0 degrees to over 20 degrees and then decreasing back to near 0 degrees, or from 0 degrees to less than -20 degrees and then increasing back to 0 degrees—it is determined that the candidate has completed a head shaking motion. This continuous tracking mechanism based on three-dimensional pose angles accurately captures the true trajectory of the head movement. In contrast, pre-recorded planar videos or photos, lacking realistic three-dimensional spatial motion information, cannot exhibit the same dynamic changes as real people in pose angle calculations. All detected action sequences are rigorously matched with the issued instruction sequences for temporal verification. This requires not only that the action types be completely consistent, but also that the response time of each action falls within a reasonable time window. Typically, the human reaction time to simple action instructions is between 0.5 and 2 seconds. If the candidate's response time is too short, it may be due to the use of an automated script. If the response time is too long, it may be due to waiting for off-site assistance. By comprehensively analyzing the accuracy, coherence, and timing of the actions, it is determined whether the candidate is a real living being and can respond to instructions independently.

[0039] If the liveness detection passes, the facial feature vectors from the ID card image and the real-time facial feature vectors from the facial video information are extracted separately. This involves preprocessing the ID card face image cropped from the ID card image, using a face detection algorithm to accurately locate the bounding box of the face region, and then using a facial landmark detection algorithm within the bounding box to extract the coordinates of key feature points such as the eyes, nose, and mouth. Based on the positional relationships of these key points, an affine transformation is performed to achieve face alignment, rotating the face to a standard frontal pose and aligning the eyes horizontally to eliminate the influence of shooting angle and head posture on feature extraction. A deep convolutional neural network model trained on large-scale face data is used to encode the features of the preprocessed ID card face image to obtain the facial identity features. Real-time facial feature vectors are extracted from the video frames in the facial video information that have passed liveness detection verification. The processing flow is basically the same as that for ID card face feature extraction, including preprocessing steps such as face detection, landmark localization, affine alignment, and size normalization, and then feature encoding is performed using the same deep convolutional neural network model. Since the video stream is a continuous dynamic sequence, several high-quality frontal face images are selected from the video frames after liveness detection. The evaluation criteria include facial clarity, illumination uniformity, and pose orientation. Real-time facial feature vectors are extracted for each selected frame. Due to slight changes in expression, illumination fluctuations, or pose deviations between different frames, the extracted feature vectors will have subtle differences, but overall they should cluster in similar regions in the feature space, representing the identity features of the same person.

[0040] An initial similarity score is obtained by calculating the similarity between the facial feature vectors on identification documents and real-time facial feature vectors. The most commonly used similarity calculation method is cosine similarity, which measures the degree of similarity by calculating the cosine of the angle between two vectors. The specific formula is the dot product of the two vectors divided by the product of their respective magnitudes. The value of cosine similarity ranges from -1 to +1. When the two vectors are in the same direction, a cosine similarity of 1 indicates perfect similarity; when the two vectors are orthogonal, a cosine similarity of 0 indicates no correlation; and when the two vectors are in opposite directions, a cosine similarity of -1 indicates complete dissimilarity. In facial recognition applications, the cosine similarity of feature vectors extracted from facial photos of the same person taken under different conditions is usually greater than 0.8 or even close to 0.9, while the cosine similarity of facial feature vectors from different people is usually less than 0.7. This significant distinguishability in the feature space makes cosine similarity an effective indicator for determining whether facial identities match. When the initial similarity score is greater than the preset similarity score, the candidate's identity verification is confirmed. The preset similarity score is a threshold parameter obtained through statistical optimization of a large amount of experimental data, usually set to 0.85. The preset similarity score can also be adjusted according to security level requirements. Experimental data shows that when the verification threshold is set to 0.85, the verification pass rate for the real person can reach over 98%, while the rejection rate for impersonation can reach over 99%, ensuring that only the real document holder can pass the verification and enter the subsequent interview stage. If the calculated initial similarity score is lower than the preset similarity score, it means that the face in the current video is not the same person as the face in the document photo. The verification process is immediately terminated, and the candidate is informed of the identity verification failure, indicating that someone else's document may have been uploaded or the person currently participating in the verification is not the document holder. The candidate is required to check and re-verify, effectively preventing identity theft.

[0041] After confirming the candidate's identity verification, multiple face image frames are selected from the face video information, and the basic face feature vectors of these frames are fused to obtain a baseline face feature template. Since a single video frame may be affected by factors such as instantaneous lighting changes, slight facial expression fluctuations, and image noise, the feature vector extracted from a single frame may contain some random errors. By fusing feature vectors from multiple frames, these random errors can be eliminated through statistical averaging. After successful liveness detection and initial similarity verification, a period of continuous video frames (e.g., 5 to 10 seconds) is continuously acquired from the video stream. During this time, the candidate can remain relaxed. From this period, 10 to 20 frames of the highest quality are selected as the input source for the fusion process, ensuring that each frame used for fusion is a high-quality face image.

[0042] Each selected face image frame undergoes the same preprocessing and feature encoding process as the ID card face feature extraction, including face alignment, size normalization, and deep network feature extraction, to obtain a basic face feature vector for each frame. A feature vector fusion algorithm is then used to integrate these basic face feature vectors, summing the values ​​of all vectors in each dimension and dividing by the number of vectors to obtain an average feature vector with the same dimensions as the basic feature vector. Besides simple averaging, a weighted average fusion method can also be used, assigning different weights to each frame based on its quality score. Higher-quality frames have a larger weight in the fusion, while lower-quality frames have a relatively smaller weight. This weighted fusion strategy further improves the quality of the fused feature vector and reduces interference from low-quality images. The average feature vector or weighted average feature vector obtained from the fusion process serves as the baseline face feature template, which is stored in a temporary memory database for this interview and used as a standard reference for determining whether identity swapping has occurred during subsequent real-time monitoring.

[0043] Furthermore, the initial audio data is processed to obtain a baseline voiceprint feature vector template. During the identity verification stage, a pre-set text is displayed to the candidate. This pre-set text is typically a phonetically optimized standard sentence containing rich phoneme combinations and prosodic variations, fully reflecting the speaker's vocal tract characteristics, pronunciation habits, and speech style. The candidate's self-introduction can also be used as the pre-set text. Audio signals are acquired in real-time via the client's microphone, and the audio data is recorded in raw PCM format. High-quality audio acquisition parameters are set to ensure that the detailed information required for voiceprint feature extraction is not lost during the acquisition stage.

[0044] Speech activity detection (VAD) is performed on the initial audio data to obtain valid speech segments. To accurately separate the valid segments containing candidate speech from the original audio stream, which includes background noise, silence segments, and speech segments, pure noise and silence are filtered out, ensuring that subsequent voiceprint feature extraction focuses on the actual speech signal rather than environmental interference. The VAD technology is based on the analysis of the audio signal's energy and spectral characteristics. It calculates the energy value of each short frame, such as a 20-millisecond or 30-millisecond audio segment. The energy value is calculated by summing the squares of the amplitudes of all sampling points within the frame, reflecting the intensity of the sound signal during that time period. The spectral characteristics of the audio are also analyzed to improve the accuracy of VAD, as speech and noise signals have different distribution characteristics in the frequency domain. Speech signals typically have strong energy concentration in the mid-to-low frequency range, such as 300Hz to 3400Hz, while background noise has a relatively flat spectral distribution or is concentrated in specific frequency bands. A Fast Fourier Transform (FFT) is performed on each frame of audio to obtain its frequency domain representation. The energy distribution of each frequency band is calculated, and the spectral shape and energy concentration characteristics are analyzed to help determine whether the frame contains speech. After speech activity detection and processing, multiple valid speech segments are obtained. Acoustic feature extraction is performed on these valid speech segments to obtain speaker signature vectors, converting the time-domain audio waveform signal into a feature vector representation that characterizes the speaker's personalized voice characteristics. Each valid speech segment is first processed to obtain a processed audio signal. This processed audio signal is then segmented into short-time frames, and a Hamming window function is calculated for each frame signal. The calculated frame signal is then subjected to a Fourier transform to obtain a complex spectrum. The power spectrum is obtained by taking the square of the modulus of the complex spectrum; the power spectrum at the linear frequency scale is converted to the Mel frequency scale. The Mel frequency scale is filtered and combined to obtain a Mel frequency energy vector. The logarithm of the Mel frequency energy vector is taken to obtain the logarithmic Mel spectrum, and a discrete cosine transform is performed on the Mel spectrum to obtain Mel frequency cepstral coefficients. Mel frequency cepstral coefficient feature vectors are extracted from all frames of the entire valid speech segment, resulting in a Mel frequency cepstral coefficient feature matrix. The number of rows in the matrix equals the number of frames, and the number of columns equals the number of Mel frequency cepstral coefficients. The Mel frequency cepstral coefficient feature matrix is ​​encoded using a deep neural network voiceprint recognition model to obtain the voiceprint feature vector.

[0045] A baseline voiceprint feature vector template is generated based on the voiceprint feature vector. Since the initial audio data contains the complete process of the candidate reading the preset text, multiple valid speech segments may be obtained after speech activity detection. Voiceprint feature vectors are extracted from each segment, and statistical modeling is performed to obtain a baseline template that can represent the overall voice characteristics of the candidate. The arithmetic mean of the voiceprint feature vectors of all segments can be calculated. By averaging multiple segments, the random errors that may exist in a single segment can be eliminated, resulting in a more stable voiceprint representation.

[0046] S103: During remote interaction, video stream data and audio stream data are collected in real time. Facial feature vectors are extracted from the video stream data. The facial feature vectors are continuously compared with the baseline facial feature template to identify candidate identity switching events. The number of switching events is determined based on the number of times the candidate identity switching events occur.

[0047] In step S103 above, after the candidate provides the aforementioned identity verification and establishes the baseline template, the formal interview process automatically begins. The client utilizes the device's camera and microphone to collect video and audio stream data in real time. Facial feature vectors are extracted from the video stream data and continuously compared with the baseline facial feature template to identify candidate identity switching events. The number of switching events is determined based on the frequency of these events. Specifically, this includes: selecting video stream data at a preset frequency to obtain the current face image frame to be detected; extracting facial feature vectors from the current face image frame and calculating the similarity between the facial feature vectors and the baseline facial feature template to obtain a real-time similarity score; determining a single match when the real-time similarity score is greater than or equal to a preset matching threshold; determining a single mismatch when the real-time similarity score is less than the preset matching threshold; counting the consecutive occurrences of single mismatches; and confirming a candidate identity switching event when the consecutive occurrences reach a preset number; and counting the number of candidate identity switching events during the remote interaction process and using this count as the switching count.

[0048] Specifically, since interviews typically last a considerable amount of time, performing complete face detection, feature extraction, and comparison calculations on every frame of the video stream would result in a large number of images requiring processing. Furthermore, even if the candidate's view is quickly replaced by another person during the interview, this physical replacement process would still take at least a few seconds. Therefore, continuous detection at a millisecond-level frequency is completely unnecessary. A preset frequency of 0.2Hz can be set, meaning one frame is selected from the video stream every 5 seconds for subsequent processing. Within each 5-second sampling period, all video frames within that period are collected. Typically, a 5-second window contains 75 to 150 frames, depending on the video frame rate setting. These candidate frames are then quickly assessed for image quality, and a sharpness score is calculated for each frame. The sharpness score is calculated based on the Laplacian variance operator. After all candidate frames within each 5-second period have completed the sharpness score calculation, they are sorted in descending order, and the frame with the highest score is selected as the current face image frame to be detected in that period. This quality priority is to avoid selecting frames that are momentarily blurred due to a candidate quickly turning their head or blinking at a certain moment. The selected current face image frame to be detected is also marked with a precise timestamp as the basis for subsequent evaluation results.

[0049] The process involves extracting facial feature vectors from the current face image frame to be detected and calculating the similarity score with the baseline face feature template to obtain a real-time similarity score. This allows for the localization of the face region within the selected face image frame. This can be achieved by running a lightweight face detection model, which employs either the MTCNN architecture based on cascaded convolutional networks or the YOLO series object detection architecture optimized with model compression. Taking MTCNN as an example, this model comprises three cascaded convolutional neural network stages. The first stage is the Proposal Network (P-Net), which rapidly scans the input image at different scales, generating a large number of candidate windows that may contain faces and their confidence scores. The P-Net network structure is very shallow, with only three convolutional layers, resulting in extremely fast inference speed, processing the entire image within milliseconds. The second stage is the Refine Network (R-Net), which refines the candidate windows output by P-Net. A deeper convolutional network extracts and classifies features from image patches within each candidate box, filtering out most non-face regions. Simultaneously, boundary regression is performed on the remaining candidate boxes to obtain more accurate face localization. The third stage is the Output Network, or O-Net, which is the deepest and most accurate network. It further refines the position of the face bounding boxes and outputs the coordinates of five key points of the face, including the centers of the two eyes, the tip of the nose, and the two corners of the mouth. After the three-stage cascaded processing of MTCNN, all detected face bounding boxes in the current image frame and their corresponding five or more key point coordinates are obtained.

[0050] Once a valid single face is detected, face alignment is performed using the detected facial landmark coordinates. Alignment standardizes face images that may have tilt, rotation, or scaling differences to a uniform pose and scale, eliminating feature differences caused by these geometric transformations. This ensures that the subsequently extracted feature vectors accurately reflect the face's intrinsic identity information without being affected by external pose. The aligned face image is then input into a deep convolutional neural network model for processing, yielding the current frame's face feature vector. The cosine similarity between the current frame's face feature vector and a baseline face feature template is calculated to obtain a real-time similarity score. This real-time similarity score is compared to a preset matching threshold for matching. A match is considered a single occurrence when the real-time similarity score is greater than or equal to the preset matching threshold, and a mismatch is considered a single occurrence when the real-time similarity score is less than the preset matching threshold. The preset matching threshold is the critical standard for determining whether the person is the same person or different people. The preset matching threshold can be set to 0.80 and was obtained by analyzing video data from hundreds of normal remote interviews.

[0051] For example, in a 30-minute interview, 360 similarity tests were performed at 5-second intervals. In the first 20 minutes (240 tests), the real-time similarity score fluctuated between 0.82 and 0.91, and all tests were judged as single matches. This indicates that the candidate was present and in a stable state throughout the 20 minutes. In the 252nd test at the 21st minute, the candidate turned to get a water glass, resulting in a larger angle of their profile, causing the similarity score to drop to 0.76, below the threshold and judged as a single mismatch. However, in the 253rd test, the candidate had returned to a facing posture, and the similarity score rebounded to 0.85, again being judged as a single match. This brief mismatch is a normal change in posture and should not trigger an alarm.

[0052] The system counts the number of consecutive mismatches. When this number reaches a preset threshold, it is confirmed as a candidate identity switching event. To prevent misjudging normal, occasional fluctuations in characteristics as identity switching, a continuity check is implemented. This is achieved by maintaining a mismatch continuity counter, which is initialized to 0 at the start of remote interaction. Each time a match is detected as False (single mismatch), the counter increments by 1; each time a match is detected as True (single match), the counter is reset to zero and the counting restarts. This counting logic ensures that only consecutive mismatches accumulate; any match interrupts the counting process. Only consecutive mismatches trigger the final event confirmation. The preset threshold is the number of consecutive mismatches required to trigger an identity switching event. This threshold can be set to 3, meaning that a candidate identity switching event is confirmed only when the similarity score of three consecutive detections is below the 0.80 threshold.

[0053] After confirming an identity switching event, the mismatch continuity counter is reset to zero, and the next round of continuity monitoring begins. This allows for the detection of multiple identity switching events; for example, a candidate might attempt to switch identities multiple times at different stages of the interview, and each switch is detected and recorded independently. It's important to note that the interview is not immediately interrupted or the candidate is warned after an event is confirmed. Instead, the event is silently recorded and monitoring continues. This non-intrusive approach avoids interference with the normal interview process, and even if a misjudgment occurs, it will not negatively impact the candidate's experience. All anomaly determinations are finalized after the interview through comprehensive analysis and possible manual review. The number of candidate identity switching events during remote interaction is counted and used as the switching count. When the interview ends normally or is prematurely terminated, the final value of the switching count counter represents the total number of identity switching detected in that interview. In an ideal honest interview scenario, the candidate participates in person from beginning to end without any identity substitution. In this case, the switching count should remain at 0. In this situation, the candidate's performance in the identity continuity verification dimension is perfect, and there is no suspicion of proxy testing.

[0054] S104: The audio feature vector extracted from the audio stream data is compared with the reference voiceprint feature vector template to identify non-reference voiceprints that do not match the reference voiceprint feature vector template, and the duration of the non-reference voiceprints is accumulated to determine the duration of audio anomalies.

[0055] In S104 above, the audio feature vector extracted from the audio stream data is compared with the reference voiceprint feature vector template to identify non-reference voiceprints that do not match the reference voiceprint feature vector template, and the duration of non-reference voiceprints is accumulated to determine the duration of audio anomalies. Specifically, this includes: performing speech activity detection on the audio stream data to obtain multiple real-time speech segments; extracting acoustic features from each real-time speech segment to obtain real-time voiceprint feature vectors; calculating the distance between the real-time voiceprint feature vector and the reference voiceprint feature vector template to obtain a real-time voiceprint distance value; determining whether the real-time voiceprint distance value is greater than a preset voiceprint distance threshold; when the real-time voiceprint distance value is less than or equal to the preset voiceprint distance threshold, marking the real-time speech segment corresponding to the real-time voiceprint distance value as a reference voiceprint; when the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, marking the real-time speech segment corresponding to the real-time voiceprint distance value as a non-reference voiceprint; and statistically summing the durations of all non-reference voiceprint segments identified in remote interaction to obtain the duration of audio anomalies.

[0056] Specifically, speech activity detection is performed on the audio stream data to obtain multiple real-time speech segments. The purpose of this speech detection step is to accurately separate real human speech segments from the continuous audio stream and eliminate interference from silence, background noise and non-speech sounds. The audio stream data is then processed into frames to obtain multiple audio frames. Frame segmentation is the process of dividing a long time series into short time windows of fixed length.

[0057] To avoid boundary effects caused by complete information fragmentation between adjacent audio frames, a frame shift technique is used to divide the audio stream into sliding window-like frames. Frame shift is defined as the time interval between the start positions of two adjacent frames, typically set to 10 milliseconds, or 160 sampling points. This means that each frame has an overlap region of 15 milliseconds, or 240 sampling points, with an overlap rate of 60%. The target audio frame is obtained from multiple audio frames for subsequent energy and zero-crossing rate calculations. Here, the target audio frame refers to the frame currently being analyzed to determine whether it contains speech. The sum of squares of the amplitudes of all sampling points within the target audio frame is calculated to obtain the target energy corresponding to the target audio frame. Then, the number of times the audio signal in the target audio frame crosses the zero level is calculated to obtain the target zero-crossing rate. The zero-crossing rate is another important characteristic parameter in speech signal processing; it counts the number of times the sign of two adjacent sampling points in a sampling point sequence changes, i.e., the case where the previous sampling point is positive and the next is negative, or vice versa.

[0058] The system determines whether the target energy is greater than an energy threshold and whether the target zero-crossing rate is less than a zero-crossing rate threshold. This combination of conditions is used to distinguish between speech frames and non-speech frames. The energy threshold is set dynamically based on the background noise level of the current environment. When the target energy is greater than the energy threshold and the target zero-crossing rate is less than the zero-crossing rate threshold, the target audio frame is likely to contain voiced speech components. This is because such a frame has sufficient energy above the background noise and a low zero-crossing rate characteristic consistent with quasi-periodic voiced speech, and is therefore marked as a speech frame. When the target energy is less than or equal to the energy threshold, or the target zero-crossing rate is greater than or equal to the zero-crossing rate threshold, the target audio frame does not conform to the characteristics of voiced speech and may be silence, background noise, or unvoiced segments, and is therefore marked as a non-speech frame. The audio frames that are continuously labeled as speech frames are merged to obtain real-time speech segments. The purpose of the merging operation is to aggregate the discrete single-frame judgment results into continuous speech segments. This is because in real speech, a word or phrase usually lasts for hundreds of milliseconds or even seconds, corresponding to tens to hundreds of audio frames. These frames are normally continuously labeled as speech frames. Merging them into a speech segment facilitates subsequent overall feature extraction and analysis.

[0059] Acoustic feature extraction is performed on each real-time speech segment to obtain a real-time voiceprint feature vector. Voiceprint feature extraction is the process of converting the time-domain audio waveform signal into a numerical vector that can characterize the speaker's personality traits. Mel-frequency cepstral coefficients (MFCCs) are used as the representation of voiceprint features. The first step of MFCC feature extraction is to pre-emphasize the real-time speech segment. Pre-emphasis is a high-pass filtering operation. The pre-emphasized speech segment is then segmented into frames, resulting in multiple short frames within the speech segment. The final MFCC coefficients are calculated for each short frame, and the final MFCC coefficients of each frame are extracted to obtain the feature matrix. To obtain a unified voiceprint representation for the entire speech segment, the features from multiple frames need to be aggregated into a single vector to obtain the real-time voiceprint feature vector.

[0060] After obtaining the real-time voiceprint feature vector, it needs to be compared with the candidate's baseline voiceprint feature vector template to determine the source of the speech. However, in remote interview scenarios, the audio stream contains the voices of both the interviewer and the candidate. If all detected speech segments are directly compared with the candidate's baseline voiceprint template, the interviewer's normal questioning speech may be incorrectly marked as a non-baseline voiceprint because it does not match the candidate's voiceprint, thus significantly increasing the duration of audio anomalies and generating a large number of false alarms. To avoid this mislabeling problem, speaker separation is required before voiceprint comparison. This correctly assigns speech segments in the audio stream to either the interviewer or the candidate. Only speech segments belonging to the candidate need to be matched and verified against the candidate's baseline voiceprint template, while speech segments belonging to the interviewer should be excluded from the anomaly detection range.

[0061] During the initial interaction phase of the interview, when the interviewer makes opening remarks such as self-introduction, reads the interview rules, or asks the first question, a segment of the interviewer's voice data, ranging from 10 to 30 seconds in length, is collected. This voice data undergoes the same processing flow as the candidate's baseline voiceprint, including voice activity detection to filter out valid voice segments, MFCC feature extraction, and multi-frame feature statistical aggregation, ultimately generating an interviewer voiceprint feature vector template. The interviewer's voiceprint template, along with the candidate's baseline voiceprint template, is stored in the context of the current interview session as a reference standard for subsequent speaker recognition. Throughout the entire interaction process after the formal start of the interview, for each real-time voice segment obtained from voice activity detection, a real-time voiceprint feature vector is first extracted. Then, the distances between the real-time voiceprint feature vector and the candidate's baseline voiceprint template and the interviewer's voiceprint template are calculated, yielding the candidate's voiceprint distance value and the interviewer's voiceprint distance value.

[0062] By comparing these two distance values, if the candidate's voiceprint distance value is less than the interviewer's voiceprint distance value, it indicates that the voiceprint features of the current speech segment are closer to the candidate's, and the speech segment is classified as the candidate's speech and marked as such. If the interviewer's voiceprint distance value is less than or equal to the candidate's voiceprint distance value, it indicates that the voiceprint features of the current speech segment are closer to the interviewer's or are at a similar distance to both, and the speech segment is classified as the interviewer's speech and marked as such. This speaker separation method based on the minimum distance criterion can effectively distinguish speech segments in the audio stream into candidate speech and interviewer speech. It should be noted that in practical applications, a confidence threshold can be set. Only when the difference between the two distance values ​​is greater than this threshold is a definitive classification made. If the difference is too small, it indicates that the two are too close to be reliably distinguishable, and the segment can be marked as uncertain and temporarily excluded from the analysis. This conservative strategy can further reduce the risk of misclassification.

[0063] After speaker separation, only the real-time speech segments marked as candidate speech segments will enter the subsequent voiceprint anomaly detection process. The distance between the real-time voiceprint feature vector and the baseline voiceprint feature vector template is calculated to obtain the real-time voiceprint distance value. The baseline voiceprint feature vector template is established during the pre-interview identity verification stage and represents the voiceprint feature baseline of the candidate who passed the initial verification. The purpose of distance calculation is to quantify the similarity between the voiceprint features of the current speech segment and the baseline voiceprint; the smaller the distance, the more similar they are, and the larger the distance, the greater the difference, possibly indicating different speakers. In the field of voiceprint recognition, commonly used distance measurement methods include Euclidean distance, Mahalanobis distance, and cosine distance. Considering computational efficiency and real-time requirements, this application adopts Euclidean distance as the main distance measurement method to obtain the real-time voiceprint distance value. It is then determined whether the real-time voiceprint distance value is greater than a preset voiceprint distance threshold. The preset voiceprint distance threshold is the critical value for distinguishing between the same speaker and different speakers. The preset voiceprint distance threshold can be set to a value between 15 and 18, and this range is obtained by referencing statistical analysis of a large amount of real speaker data.

[0064] When the real-time voiceprint distance value is less than or equal to the preset voiceprint distance threshold, the real-time speech segment corresponding to the real-time voiceprint distance value is marked as the reference voiceprint. This means that the acoustic features of the real-time speech segment are highly consistent with the reference voiceprint template, and it is determined that the real-time segment is the voice of the original candidate who passed the initial identity verification, and the interview continues normally. When the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, the real-time speech segment corresponding to the real-time voiceprint distance value is marked as a non-reference voiceprint. This indicates that the acoustic features of the real-time speech segment are significantly different from the reference voiceprint, and may come from different speakers. This situation corresponds to several possible scenarios. One is that someone else assists, that is, a second person provides answer hints to the candidate in a low voice outside the camera's field of view. Although the second person does not directly face the camera to answer the question, the sound of their voice will be captured by the microphone. After speaker separation, since the voiceprint is neither close to the candidate nor close to the interviewer, it will be classified as an abnormal sample of the candidate's speech segment, forming a non-reference voiceprint speech segment. Another scenario involves candidates using voice-changing software or playing pre-recorded audio, attempting to answer questions with a voice other than their own or an AI-synthesized voice. These voices have significantly different voiceprint characteristics from the original candidate and will be detected as non-reference voiceprints. The duration of all non-reference voiceprint segments identified during remote interactions is statistically analyzed and accumulated to obtain the duration of audio anomalies. This cumulative statistical analysis aims to quantify the severity and persistence of non-reference voiceprint phenomena. Since the interviewer's speech segments have been excluded through speaker separation before the statistical analysis, the accumulated duration accurately reflects the extent of third-party assistance or audio cheating during the candidate's answers.

[0065] To further improve detection accuracy, secondary filtering using audio energy analysis is required when marking non-reference voiceprints. Although speaker separation has excluded the main segments of the interviewer's speech, some boundary cases may still exist. These include physiological sounds made by the candidate during the answer, such as coughing, clearing their throat, or drinking water, as well as background noise such as doorbells or keyboard clicks. These sounds, lacking clear voiceprint features, may be incorrectly attributed to the candidate's speech during speaker separation. However, their voiceprint feature vectors differ significantly from the reference voiceprint, causing the real-time voiceprint distance value to exceed a threshold and thus being marked as a non-reference voiceprint. To avoid including these non-cheating sounds in the duration of audio anomalies, after determining that the real-time voiceprint distance value is greater than a preset voiceprint distance threshold, the average energy of the real-time speech segment is further calculated. This involves averaging the energy values ​​of all audio frames within the segment to obtain the segment's average energy. The average energy of the segment was compared with a preset energy threshold, which was set between 50% and 70% of the average energy of normal human voice conversation. The selection of this threshold was based on experimental data. The energy of normal human voice conversation is usually concentrated in a stable range, while physiological sounds such as coughing and drinking water may have high instantaneous energy, but the average energy is usually low, and the energy of background noise is even lower.

[0066] When the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, and the average energy of the segment is greater than the preset energy threshold, both conditions are met before the real-time speech segment is finally confirmed and marked as a non-reference voiceprint. This indicates that the segment does not match the candidate's reference voiceprint features, but has sufficient energy to indicate that it is a meaningful human voice segment rather than occasional noise. Such segments are most likely to come from prompts from third-party assistants or pre-recorded audio played by the candidate. When the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, but the average energy of the segment is less than or equal to the preset energy threshold, although the voiceprint does not match, the energy is low, and it is judged as a low-energy abnormal segment. This segment is not marked as a non-reference voiceprint, but is marked as suspected environmental interference. Such segments are not included in the cumulative duration of audio anomalies, but will be recorded in the log for reference during manual review. Through this filtering mechanism that combines voiceprint distance and energy as dual criteria, it is possible to effectively distinguish between real cheating audio and the candidate's normal physiological sounds or environmental noise, significantly reducing the false alarm rate and improving the accuracy of audio anomaly detection.

[0067] For example, in a 40-minute remote interview, voice activity detection extracts about 200 real-time voice segments from the audio stream. First, these 200 segments are separated into speakers. By calculating and comparing the distance between the voiceprint feature vector of each segment and the candidate's baseline voiceprint template and the interviewer's voiceprint template, 110 segments are assigned to the candidate's voice segments and 90 segments are assigned to the interviewer's voice segments. Subsequently, voiceprint anomaly detection is performed only on these 110 candidate voice segments. The Euclidean distance between the voiceprint feature vector and the candidate baseline voiceprint template was calculated for each of the 110 segments. Among them, the distance values ​​of 20 segments were between 18 and 35, exceeding the threshold of 15 to 18, and were initially judged as voiceprint mismatch. Further energy analysis of these 20 segments revealed that the average energy of 15 segments was low, ranging from 30% to 60% of the preset energy threshold. After listening to the audio content of these segments, it was confirmed that they mainly consisted of the sounds of the candidate coughing, clearing their throat, drinking water, and environmental noise such as brief doorbell sounds in the background. Although these 15 segments did not match the voiceprint, they were marked as suspected environmental interference rather than non-baseline voiceprints because their energy was below the threshold, and were not included in the duration of audio abnormality. The average energy of the other five segments were 85%, 92%, 78%, 88%, and 95% of the preset energy threshold, respectively, all significantly higher than the threshold. Their voiceprint distance values ​​were 22, 28, 19, 25, and 31, respectively, which also greatly exceeded the threshold. These five segments met both the conditions of voiceprint mismatch and sufficient energy, and were finally identified as non-reference voiceprints. Their durations were 8 seconds, 12 seconds, 15 seconds, 10 seconds, and 20 seconds, respectively, totaling 65 seconds. The durations of these five non-reference voiceprint segments were added together to obtain an audio abnormality duration of 65 seconds. In the subsequent manual review, the reviewers played the audio clips of these five segments and confirmed that three of them were spoken in a deep male voice, containing keywords for the answers to the questions. This was clearly inconsistent with the candidate's female identity and was therefore determined to be assistance from someone else outside the room. The other two segments, although spoken in female voices, had timbre, speaking speed, and rhythm that differed significantly from the candidate's baseline voiceprint. Furthermore, the speech was unusually fluent and professional, far exceeding the candidate's initial expression level. This was determined to be either pre-recorded audio or AI-generated speech. Based on the combined results, it was confirmed that the abnormal duration of these 65 seconds of audio corresponded to genuine cheating behavior.

[0068] S105: Extract facial key points from video stream data to estimate the candidate's gaze direction, and accumulate the time when the gaze direction deviates from the preset normal gaze range to determine the duration of gaze abnormality.

[0069] In step S105 above, facial key points are extracted from the video stream data to estimate the candidate's gaze direction, and the time during which the gaze direction deviates from the preset normal gaze range is accumulated to determine the duration of the gaze abnormality. Specifically, this includes: extracting the candidate's facial key points from the video stream data in real time; calculating the candidate's head pose angle based on the facial key points and a preset 3D face model; determining whether the head pose angle continuously exceeds the preset normal gaze range; when the time when the head pose angle continuously exceeds the preset normal gaze range reaches a preset single duration threshold, confirming it as a gaze abnormality event and recording the duration of the gaze abnormality event; and statistically summing the durations of all gaze abnormalities identified during the remote interaction process to obtain the duration of the gaze abnormality.

[0070] Specifically, face detection is performed on the current frame image obtained from the video stream to locate face regions. Although faces usually occupy the main position of the image and their position is relatively stable in interview scenarios, it is still necessary to determine the accurate face bounding box through detection algorithms, because keypoint detection algorithms usually require aligned face images as input to obtain the best results. The preprocessed face image is fed into the PFLD keypoint detection model for inference. This model is an end-to-end convolutional neural network. The input is a normalized RGB image tensor, and the output is a vector containing 68 multiplied by 2, or 136 values, which are the accurate locations of 68 facial keypoints of the candidate in the current video frame. These keypoints accurately annotate the spatial positions of facial features such as eyes, nose, and mouth.

[0071] The candidate's head pose angles are calculated based on facial landmarks and a pre-defined 3D face model. Head pose describes the orientation of the head in three-dimensional space, typically represented by three Euler angles: yaw, pitch, and roll. These three angles define the rotational transformation required to transition from a neutral frontal pose to the current pose. Estimating 3D head pose from facial landmarks in a 2D image is a perspective geometry problem. The pre-defined 3D face model used is a 3D geometric model of an average face, containing the coordinates of 68 3D vertices corresponding to 68 2D landmarks. This model is usually built based on a large-scale face scan database such as BU-3DFE, obtained by registering and averaging 3D face meshes from thousands of different individuals.

[0072] In remote interview scenarios, candidates use various camera models that are often impossible to pre-calibrate. Therefore, an approximate intrinsic parameter based on standard assumptions is used to convert the rotation vector into a more intuitive Euler angle representation. After obtaining the rotation matrix R, Euler angles are extracted from the matrix elements. The calculated three head posture angles, expressed in degrees, represent the candidate's current head orientation. These angle values ​​change in real-time with the candidate's head movements. In normal interview interactions, when listening to the interviewer's questions or stating their ideas, the candidate's head posture should remain relatively stable and facing the camera. The yaw and pitch angles should fluctuate slightly around zero degrees. A preset normal gaze range is used to define what constitutes a normal head posture. This preset normal gaze range includes the normal yaw angle range and the normal pitch angle range. These two ranges define an allowable posture space; only when the head posture angle falls within this space is it considered a normal gaze at the camera. Through observation and statistical analysis of real interview scenarios, the normal yaw angle range is set to -20 degrees to +20 degrees, and the normal pitch angle range is set to -15 degrees to +15 degrees. It should be noted that these thresholds are not absolute hard standards and can be adjusted according to specific scenario requirements in practical applications. For example, in highly technical interviews, candidates may need to demonstrate algorithms or draw diagrams on a whiteboard. In this case, it is a reasonable work requirement for them to look away from the camera. The thresholds can be appropriately relaxed or the detection logic can be improved by adding recognition and exemptions for such scenarios.

[0073] To determine whether the head posture angle continuously exceeds the preset normal line of sight, for each frame in the video stream, the current head posture angle is calculated and compared with the preset normal line of sight range. First, it checks whether the yaw angle is within the normal yaw angle range, i.e., whether the yaw is greater than or equal to -20 degrees and less than or equal to +20 degrees. Then, it checks whether the pitch angle is within the normal pitch angle range, i.e., whether the pitch is greater than or equal to -15 degrees and less than or equal to +15 degrees. Only when both conditions are met simultaneously is the head posture of the current frame considered to be within the normal line of sight range; otherwise, it is considered to be outside the range.

[0074] Determining whether a deviation from the normal range is sustained requires considering the time dimension. Instantaneous posture deviations might simply be normal movements of the candidate, such as turning the head or moving the neck habitually. Only deviations lasting for a certain period constitute a visual abnormality. Therefore, it's necessary to track the posture state of consecutive frames and measure the duration of the deviation. When a head posture angle exceeding the preset normal visual range is detected in a frame, the timestamp of the deviation's start is recorded. Subsequent video frames are then processed. If the posture continues to exceed the range in subsequent frames, the timer accumulates. Only when the duration of the deviation reaches a preset single-event duration threshold is it confirmed as a visual abnormality event. Based on the analysis of real cheating behavior patterns, when candidates look at their phones or other supplementary materials to obtain answers or hints, they typically need to read text or content on the screen. Therefore, the preset single-event duration threshold can be set to 5 seconds. Experimental data shows that in normal interviews, instances of candidates' head posture deviating from the normal range for more than 5 seconds are extremely rare, occurring in less than 1% of cases. However, in simulated cheating tests, this proportion exceeds 80%, indicating that the 5-second threshold has good discriminatory power.

[0075] When a head posture angle continuously exceeds the preset normal line of sight for 5 seconds, it is confirmed as a visual abnormality event, and the start time of the visual abnormality event is recorded. This time corresponds to the timestamp of the frame in which the posture was first detected to be outside the range. The event end timestamp field is initially updated to the current frame timestamp, as the deviation may still be ongoing. The duration is calculated as the current time minus the start time. If the head posture angle returns to the preset normal line of sight in subsequent frames, the current visual abnormality event is considered to have ended, and the event end timestamp is recorded as the timestamp of the frame in which the posture returned to normal. The duration of a single visual abnormality event is calculated as the recorded event end timestamp minus the start timestamp. The durations of all visual abnormality events identified during remote interaction are statistically analyzed and accumulated to obtain the duration of visual abnormality.

[0076] For example, in a 40-minute interview, eight abnormal gaze events were detected, with durations of 7 seconds, 12 seconds, 5 seconds, 18 seconds, 6 seconds, 9 seconds, 25 seconds, and 8 seconds, respectively. By adding up these durations, the final duration of the abnormal gaze event was 90 seconds, or 1 minute and 30 seconds.

[0077] S106: Based on the interviewer's questions and the candidate's answers, determine the response delay in each of the multiple question-and-answer cycles, and determine the duration of the cumulative response delay that exceeds the response delay threshold to determine the duration of the interaction anomaly.

[0078] In S106 above, the response delay in each of multiple question-and-answer cycles is determined based on the interviewer's questions and the candidate's answers. The duration of the accumulated response delay exceeding the response delay threshold is used to determine the duration of the interaction anomaly. Specifically, this includes: performing voice activity detection on the interviewer's questions and the candidate's answers to determine the end time of the question and the start time of the answer in each question-and-answer cycle, and using the difference between the end time of the question and the start time of the answer as the response delay of the current question-and-answer cycle; collecting the candidate's initial speech in the initial stage, performing speech recognition on the initial speech to obtain the baseline text, and calculating the language complexity of the baseline text to obtain the baseline complexity score; performing speech recognition on the candidate's answer in the current question-and-answer cycle to obtain the answer text, and calculating the language complexity of the answer text to obtain the real-time complexity score; when the response delay exceeds the response delay threshold, and the ratio of the real-time complexity score to the baseline complexity score exceeds a preset complexity gain threshold, the current answer cycle is marked as an interaction anomaly event, and the response delay duration corresponding to the interaction anomaly event is recorded; and the response delay durations of all interaction anomaly events identified during the remote interaction are statistically summed to obtain the duration of the interaction anomaly.

[0079] Specifically, speech activity detection is performed on the interviewer's questions and the candidate's answers to determine the end point of the question and the start point of the answer in each question-and-answer cycle. Speech activity detection distinguishes between speech segments containing human voices and silent or noisy segments from a continuous audio stream. In remote interview scenarios, speech activity detection can accurately segment the statements of the interviewer and the candidate, identifying the start and end boundaries of each question and answer. The audio stream is extracted through speech activity detection to obtain multiple speech segments to be classified. To distinguish between the interviewer and the candidate, reference voiceprints for both are obtained beforehand. For the candidate's reference voiceprint, the baseline voiceprint feature vector template established during the identity verification stage is used. For the interviewer's reference voiceprint, an audio segment can be captured during the opening remarks stage of the interview, such as when the interviewer introduces themselves or reads the interview rules, and the voiceprint features are extracted as the interviewer's voiceprint template. The similarity between the voiceprint feature vector of each speech segment to be classified and the candidate's and interviewer's voiceprint templates is calculated. Cosine similarity or Euclidean distance is used for similarity calculation. If the voiceprint of a speech segment has a higher similarity to the baseline voiceprint feature vector template, it is marked as a candidate speaking; if it has a higher similarity to the interviewer's voiceprint template, it is marked as an interviewer speaking. Using this method, all speech segments are divided into two categories, forming a time series such as interviewer from 0 to 12 seconds, candidate from 15 to 28 seconds, interviewer from 30 to 45 seconds, and candidate from 48 to 65 seconds. This series clearly demonstrates the question-and-answer interaction structure of the interview.

[0080] Based on the speaker separation results, each question-and-answer cycle is identified. A question-and-answer cycle is defined as the complete interaction process from the interviewer's question to the candidate's answer, specifically including the interviewer's question segment and the immediately following candidate's answer segment. Pattern matching is used to find patterns in the time series where the interviewer's voice segment is immediately followed by the candidate's voice segment. Each such pattern found constitutes a question-and-answer cycle. For example, in the time series above, the first question-and-answer cycle is the interviewer asking questions from 0 to 12 seconds and the candidate answering from 15 to 28 seconds, and so on. It should be noted that more complex interaction patterns may exist in actual interviews, such as the interviewer interrupting to ask follow-up questions during the candidate's answer or the interviewer remaining silent to think after the candidate finishes answering. Rules can be set such that a question-and-answer cycle is only considered to exist when the time interval between the start time of the candidate's voice segment and the end time of the previous interviewer's voice segment is within a reasonable range, such as 1 to 10 seconds.

[0081] For each identified question-and-answer cycle, the end time of the question and the start time of the answer are determined. The end time T2 of the question is the timestamp of the interviewer's speech segment ending, i.e., the moment the interviewer stops speaking; the start time T1 of the answer is the timestamp of the candidate's speech segment beginning, i.e., the moment the candidate begins speaking. The difference between the end time of the question and the start time of the answer is used as the response delay for the current question-and-answer cycle. The formula for calculating the response delay Δt is Δt = T1 - T2. This difference represents the time elapsed from the end of the interviewer's question to the beginning of the candidate's answer, which is the candidate's thinking time or preparation time. In the example of the first question-and-answer cycle, the response delay is 15.2 seconds minus 12.0 seconds, which equals 3.2 seconds, meaning that the candidate begins answering 3.2 seconds after the interviewer's question ends.

[0082] When candidates use AI assistants to cheat, the response latency exhibits an unusual pattern: the latency falls within a specific range, and the variance of latency across different questions is very small. This is because the AI ​​assistant's answer generation process involves a fixed procedure. After hearing the interviewer's question, the candidate transmits the question to an off-site assistant via a hidden earpiece or other means, or directly inputs it into the AI ​​dialogue interface. The AI ​​model receives the question and begins reasoning to generate an answer. For large language models deployed in the cloud, the latency from receiving the question to outputting the first answer is typically between 1 and 3 seconds. The total latency in generating a complete answer depends on the answer length and the model's reasoning speed. Due to the relative stability of AI reasoning speed and network transmission, the latency of different questions exhibits a small variance. For example, the latency for five consecutive questions is 5.2 seconds, 4.8 seconds, 5.5 seconds, 5.1 seconds, and 4.9 seconds, with a standard deviation of only 0.28 seconds.

[0083] In contrast, the delay time of natural human thinking has greater randomness and question dependence. For areas familiar to the candidate, the delay may be as short as 1 to 2 seconds; for questions requiring in-depth thinking, the delay may be as long as 15 seconds or even longer. Furthermore, candidates may exhibit observable behavior during the thinking process. Therefore, by analyzing the statistical characteristics of response delay, such as mean, variance, and distribution shape, it is possible to distinguish between normal thinking and AI assistance. If the response delay in multiple consecutive question-and-answer cycles falls within a narrow range of 3 to 5 seconds with extremely small variance, this regularity strongly suggests that the candidate is relying on an external system to generate answers. Since the regularity of response delay alone is insufficient to definitively determine AI cheating, language complexity analysis is also necessary. By comparing the complexity of the candidate's language expression at different stages, abnormal abrupt changes in complexity can be detected. Such abrupt changes may indicate that the candidate used high-quality AI-generated text in some answers, and the language level of this text significantly exceeds normal expressive ability.

[0084] In the initial stage, candidates' initial remarks are collected and processed through speech recognition to obtain baseline text. The initial stage refers to the self-introduction segment at the beginning of the interview. One or more segments of the candidate's remarks are selected from this stage, and speech recognition is performed on these remarks to convert the speech signals into baseline text. The linguistic complexity of the baseline text is calculated to obtain a baseline complexity score. Linguistic complexity is a comprehensive indicator of text language proficiency, encompassing multiple dimensions such as lexical richness, syntactic complexity, and logical coherence. Natural language processing tools are used to analyze the baseline text. First, word segmentation is performed to divide the text into a list of words. Then, part-of-speech tagging is performed to identify the part of speech of each word, such as noun, verb, or adjective. Finally, syntactic analysis is performed to construct a syntactic dependency tree. Based on these analytical results, a series of language feature indicators were calculated. These indicators and weights can be customized according to different job requirements and interview language. Lexical richness indicators include: lexical density (defined as the number of content words divided by the total number of words) and average word length; syntactic complexity indicators include: average sentence length (total number of words divided by the number of sentences), clause ratio, and average syntactic tree depth; logical coherence indicators include: conjunction density (the number of occurrences of conjunctions such as "because," "therefore," and "but" divided by the total number of sentences), pronoun consistency, and topic shift frequency. The values ​​of these indicators were normalized to a range of 0 to 1, and a weighted average was used to obtain a comprehensive initial language complexity score, also ranging from 0 to 1. 0 indicates extremely simple language, close to casual conversation, while 1 indicates highly complex language, close to written language or academic text.

[0085] Once the interview enters the formal technical questioning phase, speech recognition is performed on the candidate's responses during the current question-and-answer period to obtain the response text. The linguistic complexity of the response text is then calculated to obtain a real-time complexity score. The processing flow is the same as the initial stage and will not be described in detail here. The initial linguistic complexity score is compared with the real-time complexity score. If the real-time complexity score changes abruptly, it indicates that the candidate's language level in this response significantly exceeds their natural expression ability. Under normal circumstances, a person's language expression style is relatively stable. In a tense interview environment, a candidate's linguistic complexity is usually slightly lower than or at the baseline level in a relaxed state, and will not suddenly increase significantly. Only when the candidate reads or paraphrases high-quality text written by others will the linguistic complexity show such a leap.

[0086] In addition to the absolute value of language complexity, the text's stylistic consistency is analyzed. By comparing the differences between the baseline text and the response text in terms of vocabulary habits, sentence structure preferences, and tone, the presence of AI-generated traces is further verified. Taking into account the detection results of both response latency and language complexity, the current question-and-answer cycle is judged to constitute an interaction anomaly. Specifically, when the response latency exceeds the response latency threshold and the ratio of the real-time complexity score to the baseline complexity score exceeds a preset complexity gain threshold, the current question-and-answer cycle is marked as an interaction anomaly. The response latency threshold can be set to a dynamic value that can be dynamically adjusted based on different stages. In the initial stage, a personalized baseline latency is calculated based on the candidate's response latency to simple questions, such as the average plus a standard deviation. Then, the threshold is set to 1.5 to 2 times the baseline latency. For example, if the candidate's average response latency in the initial stage is 2.5 seconds with a standard deviation of 0.8 seconds, then the baseline latency is 2.5 plus 0.8 equals 3.3 seconds, and the response latency threshold is set to 3.3 multiplied by 1.5, approximately equal to 5 seconds. This dynamic threshold can adapt to the differences in the thinking pace of different candidates.

[0087] A preset complexity gain threshold defines the magnitude of a jump in language complexity that is considered abnormal; this threshold can be set to 3.0. When the conditions for an interaction anomaly are met in the current question-and-answer cycle, the current cycle is marked as an interaction anomaly event, and the corresponding response latency is recorded. The response latency of all interaction anomalies identified during remote interaction is statistically summed to obtain the duration of the interaction anomaly. Here, the response latency of each anomaly event is summed, rather than the total answer time, because response latency better reflects the time cost of AI processing problems. For example, in a 40-minute interview, if six interaction anomalies are detected with response latency of 4.8 seconds, 5.2 seconds, 4.5 seconds, 6.1 seconds, 5.0 seconds, and 4.9 seconds respectively, these latency durations are summed to obtain an interaction anomaly duration of 30.5 seconds.

[0088] S107: Calculate the number of switching, duration of audio abnormality, duration of visual abnormality, and duration of interaction abnormality to obtain a comprehensive risk value. Generate an evaluation result based on the comprehensive risk value and send the evaluation result to the interviewer.

[0089] In step S107 above, the number of switching attempts, duration of audio anomalies, duration of visual anomalies, and duration of interaction anomalies are calculated to obtain a comprehensive risk value. Specifically, this includes: assigning risk weights to the number of switching attempts, duration of audio anomalies, duration of visual anomalies, and duration of interaction anomalies, respectively, wherein the risk weight corresponding to the number of switching attempts is greater than the risk weights corresponding to the duration of audio anomalies, duration of visual anomalies, and duration of interaction anomalies; multiplying the number of switching attempts by the risk weights corresponding to the number of switching attempts to obtain a switching risk score; processing the duration of audio anomalies, duration of visual anomalies, and duration of interaction anomalies based on a nonlinear mapping function to obtain audio severity coefficients, visual severity coefficients, and interaction severity coefficients; multiplying the audio severity coefficients, visual severity coefficients, and interaction severity coefficients by their respective risk weights to obtain audio risk scores, visual risk scores, and interaction risk scores; and summing the switching risk scores, audio risk scores, visual risk scores, and interaction risk scores to obtain a comprehensive risk value.

[0090] Specifically, risk weights are assigned to the number of identity switching attempts, the duration of audio anomalies, the duration of gaze anomalies, and the duration of interaction anomalies. These risk weights are set based on the actual impact of each anomaly type on the fairness of the interview, as well as its prevalence and severity in real-world cheating cases. Identity switching, specifically candidate changes detected by continuous facial feature matching, is the most serious form of cheating; therefore, the number of identity switching attempts is assigned the highest risk weight, which can be set to 100. This significantly exceeds the combined weight of all other anomaly types, ensuring that if an identity switching is detected even once, the overall risk value immediately exceeds the high-risk threshold, triggering a veto mechanism. This reflects the core role of identity verification in the interview integrity system.

[0091] The risk weight corresponding to the duration of audio anomalies is second only to identity switching but still quite high because audio anomalies usually indicate two serious types of cheating. First, if a voice other than the candidate's is detected, i.e., the voiceprint features do not match the baseline voiceprint template and the energy is higher than the background noise, it means that there is a second speaker in the interview environment, i.e., a ghostwriter providing answers from outside. The candidate may be repeating the ghostwriter's answer or the ghostwriter may be speaking directly to the interviewer through a hidden earpiece. Second, if AI-synthesized audio is detected, i.e., the synthesis probability output by the fake speech detection model is greater than 0.9, it indicates that the candidate used speech synthesis technology, such as a text-to-speech (TTS) system, to convert the AI-generated answer text into speech and play it to the interviewer, or used real-time voice changing technology to try to cover up the real voice. Whether it is ghostwriter assistance or AI-synthesized voice, it is a behavior that seriously distorts the interview evaluation results through technical means. Therefore, the weight of audio anomalies can be set to 80. The risk weight for the duration of abnormal interaction, i.e., the cumulative delay time suspected of AI-assisted responses, is at a medium level. While abnormal interaction strongly suggests that the candidate is using external AI tools to obtain answers, the inference based on statistical characteristics—the regularity of response delays and abrupt changes in language complexity—rather than direct observation of cheating tools, carries a certain possibility of misjudgment. Therefore, the weight of abnormal interaction needs to be set cautiously to avoid excessive punishment; the weight is set to 40. The risk weight for the duration of abnormal gaze, i.e., the cumulative time when the candidate's head posture or pupil direction deviates from the normal range for an extended period, is the lowest. This is because gaze deviation is the most ambiguous and easily influenced by normal behavior; therefore, the weight is set to 10.

[0092] The constraint for setting the above weights is that the risk weight corresponding to the number of switching attempts must be greater than the risk weights corresponding to the duration of audio anomalies, visual anomalies, and interaction anomalies. The switching risk score is obtained by multiplying the number of switching attempts by their respective risk weights. This calculation method is characterized by linear accumulation; that is, for each additional identity switch, the switching risk score increases by 100 points. The number of switching attempts is a discrete integer variable that records the total number of identity mismatch events detected by the system during the interview process. For example, if the system detects one identity switch in an interview, the switching risk score is 1 multiplied by 100, which equals 100 points.

[0093] To transform the continuous variable of duration into a severity coefficient between 0 and 1, thereby achieving comparability and additivity among different anomaly types, a nonlinear mapping function is used to process the duration of audio anomalies, visual anomalies, and interaction anomalies to obtain audio severity coefficients, visual anomaly severity coefficients, and interaction severity coefficients, respectively. The nonlinear mapping function can be the Sigmoid function, whose mathematical expression is f(x) = 1 / (1+e^(-1 / 2)). （-x）The sigmoid function outputs 0 when the input x approaches negative infinity, 1 when the input x approaches positive infinity, and 0.5 when the input x equals 0. The function curve grows most rapidly and exhibits a steep upward trend near x equals 0, gradually slowing down and eventually saturating in the region far from 0. This characteristic makes the sigmoid function very suitable for mapping duration to a severity coefficient. Besides the sigmoid function, other functions such as the tanh function and piecewise linear functions that can map duration to a severity coefficient can also be used.

[0094] The audio severity coefficient, gaze severity coefficient, and interaction severity coefficient are multiplied by their respective risk weights to obtain the audio risk score, gaze risk score, and interaction risk score, respectively. The calculation formulas are: Audio risk score = Audio severity coefficient multiplied by the weight corresponding to the duration of audio anomaly; gaze risk score = Gauge severity coefficient multiplied by the weight corresponding to the duration of gaze anomaly; and interaction risk score = Interaction severity coefficient multiplied by the weight corresponding to the duration of interaction anomaly. In the example above, the audio risk score is 0.42 multiplied by 80, approximately equal to 33.6 points; the gaze risk score is 0.38 multiplied by 10, approximately equal to 3.8 points; and the interaction risk score is 0.88 multiplied by 40, approximately equal to 35.2 points. The switching risk score, audio risk score, gaze risk score, and interaction risk score are then added together to obtain the comprehensive risk value. The comprehensive risk value is calculated as: Comprehensive risk value = Switching risk score + Audio risk score + Gauge risk score + Interaction risk score. This is a simple linear addition because each risk score has been appropriately normalized and scaled using weights and severity coefficients, and can be directly added together.

[0095] Furthermore, an assessment result is generated based on the comprehensive risk value and sent to the interviewer. Specifically, this includes: comparing the comprehensive risk value with a first risk threshold and a second risk threshold, where the first risk threshold is less than the second risk threshold; if the comprehensive risk value is less than the first risk threshold, a low-risk assessment result is generated; if the comprehensive risk value is greater than or equal to the first risk threshold and less than the second risk threshold, a medium-risk assessment result is generated and linked to a manual review instruction; if the comprehensive risk value is greater than or equal to the second risk threshold, a high-risk assessment result is generated and linked to a handling recommendation.

[0096] Specifically, the comprehensive risk value is compared with a first risk threshold and a second risk threshold. The first risk threshold is lower than the second risk threshold. These two thresholds divide the continuous comprehensive risk value space into three discrete intervals, each corresponding to a risk level. The selection of the thresholds needs to comprehensively consider multiple factors, including the actual distribution characteristics of cheating behavior, the company's tolerance for integrity risks, the availability of manual review resources, and the cost of misjudgment. The first risk threshold can be set at 30 points, and the second risk threshold at 70 points. These two values ​​are determined based on statistical analysis of a large number of actual interview cases and simulation testing of the risk assessment model.

[0097] If the overall risk value is less than the first risk threshold, a low-risk assessment result is generated. The low-risk assessment result includes the assessment object and a text description. Although the overall risk value is low, there may still be some minor anomalies. Therefore, a brief text description is generated, such as "the candidate's behavior is normal and no significant integrity risk is found." The low-risk assessment result follows the principle of minimum interference, that is, for candidates who perform normally, there should be no additional review burden or psychological pressure. After receiving the low-risk assessment result, the interviewer can focus on assessing the candidate's professional ability and soft skills without having to be distracted by checking integrity issues, thereby improving the efficiency of the interview and the candidate's experience.

[0098] If the overall risk value is greater than or equal to the first risk threshold and less than the second risk threshold, a medium-risk assessment result is generated and associated with a manual review instruction. The manual review instruction associated with the medium-risk assessment result contains all the information and tools needed by the reviewer. The review instruction contains a reference to a detailed chain of evidence. The chain of evidence is a data structure containing details of all anomalous events. It is formatted as an array, with each element representing an anomalous event. It includes the event type, such as identity switching, audio anomaly, visual anomaly, and interaction anomaly; the timestamp of the event; the duration of the event; the confidence level or severity of the event; and evidence files related to the event, such as screenshot URLs, audio clip URLs, and video clip URLs. These evidence files are not automatically deleted when the medium-risk assessment result is generated, but are retained in cloud storage with access permissions set. Only authorized reviewers can access them.

[0099] The manual review instruction also includes a standardized review process to ensure that the judgment standards of different reviewers are relatively consistent. The review process is divided into evidence review, comprehensive analysis, and output. That is, the reviewers examine the abnormal events in the evidence chain one by one, confirm whether each event actually occurred to rule out algorithmic false detections, and assess the severity of each event. The reviewers put all abnormal events together for overall evaluation, analyzing whether there are any correlations between abnormal events, such as whether gaze deviation and interaction delay occur simultaneously, suggesting that the candidate is viewing AI-generated answers on a certain device. Based on the above analysis, the reviewers make a final judgment. There are three judgment options: downgrade to low risk, which means that although the abnormal event exists, it does not constitute a substantial cheating threat and may be a normal behavioral variation or algorithmic false detection; maintain medium risk, which means that the abnormal event is indeed suspicious but the evidence is not yet sufficient, and further investigation is recommended, such as arranging a telephone follow-up or an in-person interview; upgrade to high risk, which means that the abnormal event constitutes clear evidence of cheating, and it is recommended to reject the candidate or take other measures.

[0100] If the overall risk value is greater than or equal to the second risk threshold, a high-risk assessment result is generated along with corresponding action recommendations. A high-risk assessment result means that upon detecting a serious integrity risk, relevant personnel will be immediately notified and clear action recommendations will be provided. The action recommendations associated with the high-risk assessment result include key action recommendations, automatically generated based on the specific composition of the overall risk value and the company's integrity policy. If the overall risk value is caused by a single identity switching event (i.e., the switching risk score is 100 points or higher), the key action recommendation is to immediately terminate the interview and reject the candidate, as identity switching is the most serious integrity violation and completely contradicts the fundamental premise of an interview.

[0101] If the overall risk score is caused by a combination of multiple high-weight anomalies, such as the superposition of audio and interaction anomalies, the main action recommendation is to suspend the current interview and arrange an in-person or video-monitored interview. This is because, although strong cheating signals were detected, considering the complexity of the remote environment and the error of the detection algorithm, it is reasonable to give the candidate an opportunity to prove themselves under more stringent conditions. If the overall risk score exceeds 70 points but mainly comes from the accumulation of a large number of low-to-medium weight anomalies, such as a large number of eye deviations and interaction anomalies, the main action recommendation is to require the candidate to provide additional supporting materials or conduct targeted skills tests. This is because this situation may indicate that the candidate used some auxiliary means in the interview, but their true abilities still need to be verified.

[0102] The recommendations also include a risk analysis report, which details why the candidate was deemed high-risk. The report is structured with four parts: an executive summary, key findings, evidence details, and recommended actions. The executive summary briefly outlines the core conclusions of the risk assessment. For example, during a software engineer interview, candidate Zhang was found to have committed multiple serious integrity violations, including suspected proxy assistance (3 times totaling 48 seconds) and suspected AI tool assistance (8 times totaling 43.2 seconds of response delay). The overall risk score is 96.8, far exceeding the high-risk threshold of 70, and it is recommended that the candidate be immediately rejected and added to the integrity blacklist. The key findings section lists all the anomalies that led to the high-risk rating, ordered by severity, with each anomaly including a brief description. The evidence includes the time of occurrence, duration, risk contribution value, and confidence level. For example, the first key finding is that three audio anomalies were detected, with non-candidate voiceprints. The first segment occurred at 8 minutes and 32 seconds into the interview and lasted for 15 seconds. The Euclidean distance between the voiceprint feature and the baseline voiceprint was 2.3, which is greater than the threshold of 1.5. The audio energy was higher than the background noise, indicating the presence of a second speaker. The risk contribution value of this anomaly was 61.6 points, with a confidence level of 0.92. The evidence details section provides access links and previews for all relevant evidence documents, including waveform and spectrograms of audio anomalies, video screenshots of gaze deviation with head posture angles marked, and text comparisons of interactive anomalies showing the jumps in delay time and language complexity. The above evidence has been automatically organized and labeled to facilitate quick verification by reviewers.

[0103] For example, candidate Li applied for a senior data scientist position at a technology company. During the remote interview, his behavior was monitored in real time and a comprehensive risk score was calculated. At the 35-minute mark of the interview, events such as eye deviation, non-baseline voiceprints, and abnormal interactions were detected. These anomalies were calculated as follows: the cumulative duration of the eye deviation was approximately 80 seconds, corresponding to a severity coefficient of approximately 0.31 and an eye deviation risk score of approximately 3.1; the number of abnormal interactions was 4, corresponding to a severity coefficient of approximately 0.62 and an interaction risk score of approximately 24.8; and the duration of the audio anomaly was approximately 10 seconds, but the confidence level was very high, corresponding to a severity coefficient of approximately 0.21 and an audio risk score of approximately 16.8. The comprehensive risk score S equals 0 + 16.8 + 3.1 + 24.8, which equals 44.7. The overall risk score of 44.7 points falls within the medium-risk range of 30 to 70 points, generating a medium-risk assessment result and a manual review instruction. The review instruction is sent to the reviewer Zhang's workbench. Zhang opens the review interface to view the evidence chain, plays video clips with abnormal gaze, statistical charts with abnormal interactions, and the language style during the response. Considering the relevance and consistency of all the evidence, Zhang believes that it should be upgraded to high risk. Therefore, Zhang clicks the "upgrade to high risk" button on the review interface and fills in the review opinion, suggesting that the candidate be rejected or required to participate in an offline interview to verify their true abilities.

[0104] After receiving Zhang's review decision, Li's risk rating was automatically updated to high risk, triggering a disposal recommendation generation process. The option to reject the candidate immediately was available. Simultaneously, a formal risk analysis report was generated. The report's executive summary stated that candidate Li, during a senior data scientist interview at a certain time, was confirmed by system detection and manual review to have engaged in serious cheating behavior, including using external devices to view answers (suspected AI tool assistance) and having an off-site substitute. The overall risk score, after manual review, was upgraded to high risk, with immediate rejection recommended and the candidate placed on a blacklist, prohibiting re-application for 12 months. The report was then sent to interviewer Wang's client so that Wang could assess Li's integrity based on the risk analysis report and make an appropriate choice.

[0105] This application also provides a system for remote interactive authentication and abnormal behavior recognition. Figure 2 This is a schematic diagram of the structure of a remote interactive authentication and abnormal behavior recognition system provided in an embodiment of this application. (Refer to...) Figure 2 The system includes a receiving unit 201, a first processing unit 202, a second processing unit 203, and an evaluation unit 204. The receiving unit 201 receives the ID card image uploaded by the candidate and collects the candidate's facial video information and initial audio data in real time. The first processing unit 202 generates a baseline face feature template based on the ID card image and face video information, and processes the initial audio data to obtain a baseline voiceprint feature vector template. The second processing unit 203 collects video stream data and audio stream data in real time during remote interaction. It extracts facial feature vectors from the video stream data and continuously compares them with a baseline facial feature template to identify candidate identity switching events and determines the number of switching events based on the number of times these events occur. It also compares audio feature vectors extracted from the audio stream data with a baseline voiceprint feature vector template to identify non-baseline voiceprints that do not match the template and accumulates the duration of non-baseline voiceprints to determine the duration of audio anomalies. Furthermore, it extracts facial key points from the video stream data to estimate the candidate's gaze direction and accumulates the time when the gaze direction deviates from a preset normal gaze range to determine the duration of gaze anomalies. Finally, it determines the response delay in multiple question-and-answer cycles based on the interviewer's questions and the candidate's answers and accumulates the duration when the response delay exceeds a response delay threshold to determine the duration of interaction anomalies. Evaluation unit 204 calculates the number of switching, duration of audio abnormality, duration of visual abnormality, and duration of interaction abnormality to obtain a comprehensive risk value. Based on the comprehensive risk value, it generates an evaluation result and sends the evaluation result to the interviewer.

[0106] In one possible implementation, the first processing unit 202 is used to extract the ID card face image from the ID card image, perform liveness detection on the face video information to determine that the candidate is a live object; if the liveness detection is successful, the ID card face feature vector and the real-time face feature vector in the face video information are extracted respectively; the similarity between the ID card face feature vector and the real-time face feature vector is calculated to obtain an initial similarity score; when the initial similarity score is greater than a preset similarity score, the candidate's identity verification is confirmed to be successful, and multiple face image frames are selected from the face video information, and the basic face feature vectors of the multiple face image frames are fused to obtain a baseline face feature template; the initial audio data of the candidate reading a preset text is collected, and speech activity detection is performed on the initial audio data to obtain a valid speech segment; acoustic features are extracted from the valid speech segment to obtain a voiceprint feature vector, and a baseline voiceprint feature vector template is generated based on the voiceprint feature vector.

[0107] In one possible implementation, the second processing unit 203 is used to extract facial feature vectors from video stream data, continuously compare the facial feature vectors with a reference facial feature template to identify candidate identity switching events, and determine the number of switching events based on the number of occurrences of candidate identity switching events. Specifically, this includes: selecting video stream data based on a preset frequency to obtain the current face image frame to be detected; extracting facial feature vectors from the current face image frame to be detected and calculating the similarity between the facial feature vectors and the reference facial feature template to obtain a real-time similarity score; determining a single match when the real-time similarity score is greater than or equal to a preset matching threshold; determining a single mismatch when the real-time similarity score is less than the preset matching threshold; counting the consecutive number of single mismatches, and confirming and marking it as a candidate identity switching event when the consecutive number reaches a preset number; and counting the number of occurrences of candidate identity switching events during remote interaction and using the occurrence number as the number of switching events.

[0108] In one possible implementation, the receiving unit 201 is used to perform voice activity detection on the audio stream data to obtain multiple real-time voice segments; the second processing unit 203 is used to extract acoustic features from each real-time voice segment to obtain a real-time voiceprint feature vector; calculate the distance between the real-time voiceprint feature vector and the reference voiceprint feature vector template to obtain a real-time voiceprint distance value; determine whether the real-time voiceprint distance value is greater than a preset voiceprint distance threshold; when the real-time voiceprint distance value is less than or equal to the preset voiceprint distance threshold, mark the real-time voice segment corresponding to the real-time voiceprint distance value as the reference voiceprint; when the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, mark the real-time voice segment corresponding to the real-time voiceprint distance value as a non-reference voiceprint; and count and accumulate the duration of all non-reference voiceprint segments identified in the remote interaction to obtain the duration of audio anomalies.

[0109] In one possible implementation, the receiving unit 201 is used to extract the candidate's facial key points from the video stream data in real time, and calculate the candidate's head pose angle based on the facial key points and a preset 3D face model; the second processing unit 203 is used to determine whether the head pose angle continuously exceeds a preset normal line of sight; when the time when the head pose angle continuously exceeds the preset normal line of sight reaches a preset single duration threshold, it is confirmed as a line of sight abnormality event, and the duration of the line of sight abnormality event is recorded; the duration of all line of sight abnormality events identified during the remote interaction is statistically summed to obtain the line of sight abnormality duration.

[0110] In one possible implementation, the receiving unit 201 is used to perform voice activity detection on the interviewer's questions and the candidate's answers to determine the end time of the question and the start time of the answer in each question-and-answer cycle, and to use the difference between the end time of the question and the start time of the answer as the response delay of the current question-and-answer cycle; in the initial stage, the candidate's initial speech is collected, and speech recognition is performed on the initial speech to obtain the baseline text, and the language complexity of the baseline text is calculated to obtain the baseline complexity score; the second processing unit 203 is used to perform speech recognition on the candidate's answers in the current question-and-answer cycle to obtain the answer text, and to calculate the language complexity of the answer text to obtain the real-time complexity score; when the response delay is greater than the response delay threshold, and the ratio of the real-time complexity score to the baseline complexity score is greater than the preset complexity gain threshold, the current answer cycle is marked as an interaction abnormal event, and the response delay duration corresponding to the interaction abnormal event is recorded; the response delay durations of all interaction abnormal events identified in the remote interaction process are statistically summed to obtain the interaction abnormal duration.

[0111] In one possible implementation, the evaluation unit 204 is used to assign risk weights to the number of switching attempts, the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies, respectively, wherein the risk weight corresponding to the number of switching attempts is greater than the risk weights corresponding to the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies; multiplying the number of switching attempts by the risk weights corresponding to the number of switching attempts yields a switching risk score; processing the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies based on a nonlinear mapping function yields an audio severity coefficient, a visual anomaly coefficient, and an interaction severity coefficient; multiplying the audio severity coefficient, the visual anomaly coefficient, and the interaction severity coefficient by their respective risk weights yields an audio risk score, a visual anomaly risk score, and an interaction risk score; and summing the switching risk score, the audio risk score, the visual anomaly risk score, and the interaction risk score yields a comprehensive risk value.

[0112] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0113] This application also discloses an electronic device. (See reference...) Figure 3 , Figure 3 This application provides a schematic diagram of the structure of an electronic device. The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 302, and at least one communication bus 305.

[0114] The communication bus 305 is used to enable communication between these components.

[0115] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0116] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0117] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 302, and by calling data stored in memory 302. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and application requests; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0118] The memory 302 may include random access memory (RAM) or read-only memory. Optionally, the memory 302 may include a non-transitory computer-readable storage medium. The memory 302 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 302 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc. The data storage area may store data involved in the various method embodiments described above. Optionally, the memory 302 may also be at least one storage device located remotely from the aforementioned processor 301.

[0119] like Figure 3 As shown, the memory 302, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application for remote interactive authentication and abnormal behavior recognition.

[0120] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an interface for users to input data and obtain user input data; while the processor 301 can be used to call the application stored in the memory 302 for remote interactive authentication and abnormal behavior recognition. When executed by one or more processors, the electronic device performs one or more of the methods described in the above embodiments.

[0121] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some service interfaces; indirect couplings or communication connections between devices or units may be electrical or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0127] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and the disclosure of practical truths. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure.

Claims

1. A method for remote interactive authentication and abnormal behavior identification, characterized in that, The method includes: Receive the ID card image uploaded by the candidate, and collect the candidate's facial video information and initial audio data in real time; A baseline facial feature template is generated based on the ID card image and the facial video information, and a baseline voiceprint feature vector template is obtained by processing the initial audio data. During remote interaction, video stream data and audio stream data are collected in real time. Facial feature vectors are extracted from the video stream data. The facial feature vectors are continuously compared with the baseline facial feature template to identify candidate identity switching events. The number of switching events is determined based on the number of times the candidate identity switching events occur. The audio feature vector extracted from the audio stream data is compared with the reference voiceprint feature vector template to identify non-reference voiceprints that do not match the reference voiceprint feature vector template, and the duration of the non-reference voiceprints is accumulated to determine the duration of audio anomalies. Facial key points are extracted from the video stream data to estimate the candidate's gaze direction, and the time when the gaze direction deviates from the preset normal gaze range is accumulated to determine the duration of the gaze abnormality. Based on the interviewer's questions and the candidate's answers, the response delay in each of the multiple question-and-answer cycles is determined, and the duration of the response delay exceeding the response delay threshold is accumulated to determine the duration of the interaction anomaly. The number of switching, the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies are calculated to obtain a comprehensive risk value. An evaluation result is generated based on the comprehensive risk value and sent to the interviewer.

2. The method according to claim 1, characterized in that, The step of generating a baseline facial feature template based on the ID card image and the facial video information, and processing the initial audio data to obtain a baseline voiceprint feature vector template, specifically includes: Extract the facial image from the ID card image, perform liveness detection on the facial video information, and determine that the candidate is a live object; If the liveness detection passes, the facial feature vector of the ID card image and the real-time facial feature vector of the facial video information are extracted respectively. The similarity between the facial feature vector on the document and the real-time facial feature vector is calculated to obtain an initial similarity score; When the initial similarity score is greater than the preset similarity score, the candidate identity verification is confirmed to be successful, and multiple face image frames are selected from the face video information. The basic face feature vectors of the multiple face image frames are fused to obtain the baseline face feature template. The initial audio data of the candidate reading the preset text is collected, and speech activity detection is performed on the initial audio data to obtain effective speech segments; Acoustic features are extracted from the effective speech segments to obtain voiceprint feature vectors, and the reference voiceprint feature vector template is generated based on the voiceprint feature vectors.

3. The method according to claim 2, characterized in that, The step of extracting facial feature vectors from the video stream data, continuously comparing the facial feature vectors with the baseline facial feature template to identify candidate identity switching events, and determining the number of switching events based on the number of occurrences of the candidate identity switching events specifically includes: The video stream data is selected based on a preset frequency to obtain the current face image frame to be detected; The face feature vector is extracted from the current face image frame to be detected, and the similarity between the face feature vector and the reference face feature template is calculated to obtain a real-time similarity score. When the real-time similarity score is greater than or equal to the preset matching threshold, it is determined to be a single match; When the real-time similarity score is less than the preset matching threshold, it is determined as a single mismatch; The number of consecutive mismatches is counted, and when the number of consecutive mismatches reaches a preset number, it is confirmed as a candidate identity switching event. The number of times the candidate identity switching event occurs during the remote interaction is counted, and the number of occurrences is taken as the switching count.

4. The method according to claim 2, characterized in that, The step of comparing the audio feature vector extracted from the audio stream data with the reference voiceprint feature vector template to identify non-reference voiceprints that do not match the reference voiceprint feature vector template, and accumulating the duration of the non-reference voiceprints to determine the duration of audio anomalies, specifically includes: Speech activity detection is performed on the audio stream data to obtain multiple real-time speech segments; Acoustic features are extracted from each of the real-time speech segments to obtain real-time speaker feature vectors; Calculate the distance between the real-time voiceprint feature vector and the reference voiceprint feature vector template to obtain the real-time voiceprint distance value; Determine whether the real-time voiceprint distance value is greater than a preset voiceprint distance threshold; When the real-time voiceprint distance value is less than or equal to the preset voiceprint distance threshold, the real-time speech segment corresponding to the real-time voiceprint distance value is marked as the reference voiceprint. When the real-time voiceprint distance value is greater than the preset voiceprint distance threshold, the real-time speech segment corresponding to the real-time voiceprint distance value is marked as the non-reference voiceprint. The duration of the audio anomaly is obtained by statistically analyzing and summing the durations of all segments corresponding to the non-reference voiceprints identified in the remote interaction.

5. The method according to claim 1, characterized in that, The step of extracting facial key points from the video stream data to estimate the candidate's gaze direction, and accumulating the time when the gaze direction deviates from a preset normal gaze range to determine the duration of gaze abnormality, specifically includes: The facial key points of the candidate are extracted in real time from the video stream data. Based on the facial key points and the preset 3D face model, the head pose angle of the candidate is calculated. Determine whether the head posture angle continuously exceeds the preset normal line of sight range; When the head posture angle continuously exceeds the preset normal line of sight for a period of time that reaches a preset single duration threshold, it is confirmed as a line of sight abnormality event, and the duration of the line of sight abnormality event is recorded. The duration of all the abnormal gaze events identified during the remote interaction is calculated and summed to obtain the duration of the abnormal gaze event.

6. The method according to claim 1, characterized in that, The method involves determining the response latency in multiple question-and-answer cycles based on the interviewer's questions and the candidate's answers, and accumulating the duration of response latency exceeding a response latency threshold to determine the duration of interaction anomalies. Specifically, this includes: Voice activity detection is performed on the interviewer's questions and the candidate's answers to determine the end time of the question and the start time of the answer in each question-and-answer cycle, and the difference between the end time of the question and the start time of the answer is used as the response delay of the current question-and-answer cycle. In the initial stage, the initial speech of the candidate is collected, the initial speech is subjected to speech recognition to obtain the baseline text, and the language complexity of the baseline text is calculated to obtain the baseline complexity score. The candidate's response utterance in the current question-and-answer cycle is subjected to speech recognition to obtain the response text, and the speech complexity of the response text is calculated to obtain a real-time complexity score. When the response delay is greater than the response delay threshold, and the ratio of the real-time complexity score to the baseline complexity score is greater than the preset complexity gain threshold, the current response period is marked as an interaction anomaly event, and the response delay duration corresponding to the interaction anomaly event is recorded. The duration of the interaction anomaly is obtained by statistically summing the response delays of all the interaction anomalies identified during the remote interaction process.

7. The method according to claim 1, characterized in that, The calculation of the number of switching operations, the duration of the audio anomaly, the duration of the visual anomaly, and the duration of the interaction anomaly yields a comprehensive risk value, specifically including: Each of the following is assigned a risk weight: the number of switching, the duration of audio anomaly, the duration of visual anomaly, and the duration of interaction anomaly. The risk weight corresponding to the number of switching is greater than the risk weight corresponding to the duration of audio anomaly, the duration of visual anomaly, and the duration of interaction anomaly. Multiply the number of handovers by the risk weight corresponding to the number of handovers to obtain the handover risk score; The duration of the audio anomaly, the duration of the visual anomaly, and the duration of the interaction anomaly are processed based on a nonlinear mapping function to obtain the audio severity coefficient, the visual anomaly severity coefficient, and the interaction severity coefficient. The audio severity coefficient, the line-of-sight severity coefficient, and the interaction severity coefficient are multiplied by their respective risk weights to obtain the audio risk score, the line-of-sight risk score, and the interaction risk score. The combined risk value is obtained by adding the switching risk score, the audio risk score, the line-of-sight risk score, and the interaction risk score together.

8. A system for remote interactive authentication and abnormal behavior recognition, characterized in that, The system includes a receiving unit, a first processing unit, a second processing unit, and an evaluation unit. The receiving unit receives the ID card image uploaded by the candidate and collects the candidate's facial video information and initial audio data in real time. The first processing unit generates a baseline facial feature template based on the ID card image and the facial video information, and processes the initial audio data to obtain a baseline voiceprint feature vector template. The second processing unit collects video stream data and audio stream data in real time during remote interaction. It extracts facial feature vectors from the video stream data and continuously compares the facial feature vectors with the baseline facial feature template to identify candidate identity switching events. It also determines the number of switching events based on the number of times the candidate identity switching events occur. Furthermore, it compares the audio feature vectors extracted from the audio stream data with the baseline voiceprint feature vector template to identify non-baseline voiceprints that do not match the baseline voiceprint feature vector template. It also accumulates the duration of the non-baseline voiceprints to determine the duration of audio anomalies. Facial key points are extracted from the video stream data to estimate the candidate's gaze direction, and the time when the gaze direction deviates from the preset normal gaze range is accumulated to determine the duration of the gaze abnormality. Based on the interviewer's questions and the candidate's answers, the response delay in each of the multiple question-and-answer cycles is determined, and the duration of the response delay exceeding the response delay threshold is accumulated to determine the duration of the interaction anomaly. The evaluation unit calculates the number of switching operations, the duration of audio anomalies, the duration of visual anomalies, and the duration of interaction anomalies to obtain a comprehensive risk value. Based on the comprehensive risk value, it generates an evaluation result and sends the evaluation result to the interviewer.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images