Cheating detection method and system applied to online interview

Abstract

The invention discloses a cheating detection method and system applied to online interview, relates to the technical field of computer vision, and solves the problems that in the prior art, multiple behavior characteristics in interview personnel are often adopted for monitoring in the whole process, multi-data analysis in the presence of non-cheating behaviors causes waste of computing resources, and the monitoring efficiency is high. And the cheating detection method is low in efficiency. A gaze anomaly coefficient is generated according to gaze data; when the fixation abnormal coefficient is greater than the abnormal threshold value, generating a multi-modal feature according to the monitoring video; generating a multi-modal decision weight according to the interview stage label; fusing the multi-modal decision-making weight and the multi-modal features to obtain multi-modal decision-making data; and inputting the multi-modal decision data into the cheating judgment model to obtain a monitoring result, carrying out tendency evaluation on cheating monitoring, and carrying out multi-modal dynamic fusion when the cheating tendency exists so as to accurately judge the cheating condition, so that the cheating monitoring efficiency and the cheating monitoring accuracy are improved.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, specifically a cheating detection method and system applied to online interviews. Background Art

[0002] With the rapid development of internet technology and the impact of the COVID-19 pandemic, more and more companies and universities are adopting online interviews as a key method for evaluating candidates. However, the openness and uncontrollability of online environments also pose new challenges to the fairness of interviews. Candidates may cheat in various ways, such as using electronic devices to access information or using AI to assist in answering questions. These behaviors not only undermine the fairness of the selection process but also undermine the credibility of the entire recruitment or admissions process.

[0003] Existing technologies for cheating detection in online interviews often monitor multiple behavioral characteristics of interviewees throughout the entire process. Multi-data analysis in the presence of non-cheating behaviors wastes computing resources, making the cheating detection method less efficient. Therefore, further improvements are needed to detect cheating in online interviews. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art; to this end, the present application proposes a cheating detection method and system for online interviews, which is used to solve the technical problem that the prior art often uses multiple behavioral characteristics of interviewees for monitoring throughout the entire process, and the multi-data analysis in the presence of non-cheating behavior will cause a waste of computing resources, making the cheating detection method less efficient.

[0005] To achieve the above objectives, the first aspect of the present application provides a cheating detection method for online interviews, comprising:

[0006] Obtain monitoring videos and interview stage labels within a time window; the time window refers to the time range between asking the current question and asking the next question during the interview process; the monitoring video refers to the video of the interviewee during the online interview;

[0007] Extract gaze data from monitoring videos using eye tracking technology;

[0008] generating a gaze abnormality coefficient based on the gaze data;

[0009] When there is a gaze anomaly coefficient greater than an anomaly threshold, a multimodal feature is generated based on the monitoring video; the anomaly threshold is set based on the historical gaze anomaly coefficient;

[0010] Generate multimodal decision weights based on interview stage labels;

[0011] Multimodal decision data is obtained by fusing multimodal decision weights and multimodal features;

[0012] The multimodal decision data is input into a cheating discrimination model to obtain a monitoring result; the cheating discrimination model is constructed using an artificial intelligence model.

[0013] Through the above steps, this application extracts the interviewee's gaze data and accurately quantifies the degree of abnormality of the gaze data to identify cheating tendencies. Once cheating tendencies are found, multimodal dynamic fusion technology is used to accurately identify cheating situations, which not only improves the efficiency of cheating monitoring, but also enhances the accuracy of identification.

[0014] Furthermore, generating a gaze abnormality coefficient according to the gaze data includes:

[0015] Obtaining gaze data; the gaze data includes gaze coordinates and gaze duration;

[0016] Extract gaze data whose gaze coordinates are not within a preset gaze area; the preset gaze area refers to the normal gaze area range of the interviewee when not answering questions;

[0017] A plurality of gaze regions are obtained by clustering the gaze data according to the gaze coordinates; the gaze regions include region IDs and gaze data; the gaze data include the total number of gazes, gaze time points and their corresponding gaze durations;

[0018] Several adjacent fixation time intervals are obtained by performing difference calculation on adjacent fixation time points;

[0019] The average time interval PSJ and the time interval standard deviation SJB are obtained by calculating several adjacent fixation time intervals;

[0020] By calculating several fixation durations, we can obtain the average duration PS and the standard deviation SB of the duration;

[0021] The abnormal function YF(PSJ, SJB, PS, SB, ZS) is constructed through the nonlinear relationship between the average time interval PSJ, the standard deviation of the time interval SJB, the average duration PS, the standard deviation of the duration SB, the total number of fixations ZS and the fixation abnormality coefficient respectively;

[0022] Substitute the average time interval, time interval standard deviation, average duration, duration standard deviation, and total number of gazes in several gaze areas into the abnormal function to calculate the gaze abnormality coefficient corresponding to the gaze area; the gaze abnormality coefficient refers to the quantitative degree of abnormal behavior.

[0023] Furthermore, the generating of multimodal features based on the monitoring video includes:

[0024] Obtain monitoring video and gaze data;

[0025] The facial monitoring area is identified by the MTCNN algorithm in the monitoring video, and the facial micro-motion features are extracted by optical flow calculation;

[0026] Extract audio data from the monitoring video, perform noise reduction on it, and extract audio features; the audio features include MFCC features and fundamental frequency features;

[0027] Extract question and answer data from the denoised audio data through the ASR model;

[0028] Generate question-answer anomaly coefficient based on question-answer data;

[0029] Use gaze data as eye features;

[0030] The multimodal features include facial micro-movement features, audio features, question-answering abnormality coefficients and eye features.

[0031] When conducting cheating monitoring, this application not only takes into account several behavioral characteristics of the interviewee, but also takes into account the degree of abnormality of the interviewee's answer data, and uses it as one of the multimodal features of cheating monitoring, thereby improving the accuracy of the cheating monitoring results.

[0032] Furthermore, generating a question-answer anomaly coefficient based on the question-answer data includes:

[0033] Obtain question and answer data; the question and answer data includes question content, question level, answer content and thinking time SS;

[0034] Inputting the question content and the answer content into an AI similarity generation model to obtain content AI similarity; the AI ​​similarity generation model is constructed using a large model;

[0035] Extract the expected thinking time YSS corresponding to the problem level;

[0036] The question-answering function WF(NSD, SS, YSS) is constructed based on the nonlinear relationship between the content AI similarity NSD, thinking time SS, expected thinking time YSS and the question-answering anomaly coefficient;

[0037] The question-answer anomaly coefficient is calculated by substituting the content AI similarity, thinking time, and expected thinking time into the question-answer function; the question-answer anomaly coefficient refers to the quantitative degree of abnormal behavior of the interviewee when answering.

[0038] This application quantifies the degree of abnormality in the interviewees' answer data by analyzing the interviewees' answer data. It not only considers the similarity between the answer results and the AI ​​results, but also further considers the factor of the interviewees' thinking time when answering questions, thereby improving the quantitative accuracy of the abnormality of the answer data and providing accurate data support for subsequent cheating monitoring.

[0039] Furthermore, the AI ​​similarity generation model is constructed through a large model, including:

[0040] Obtain the AI ​​similarity of several historical questions, historical answers, and their corresponding historical content;

[0041] Divide the historical question content, historical answer content, and their corresponding historical content AI similarity into training data, verification data, and test data; and perform data preprocessing on the training data, verification data, and test data to obtain a training set, a verification set, and a test set;

[0042] Select the large model as the base model;

[0043] Train the basic model using the training set, and adjust the learning rate and other hyperparameters on the validation set to obtain the pre-trained model;

[0044] By verifying the pre-trained model on the test set, we finally obtained an AI similarity generation model whose input is question content and answer content and output is content AI similarity.

[0045] Furthermore, generating multimodal decision weights based on interview stage labels includes:

[0046] Obtain interview stage labels and multimodal features;

[0047] Obtaining the initial weight CQ corresponding to the multimodal feature from a preset feature weight distribution table based on the interview stage label; the preset feature weight distribution table indicates the importance of multimodal features for identifying interview cheating at different interview stages and is set by experts based on experience;

[0048] Generate modal confidence degree MZD based on multimodal features;

[0049] Construct a weight calculation function QF(MZD, CQ) based on the nonlinear relationship between the modal confidence degree MZD, the initial weight CQ corresponding to the multimodal feature, and the multimodal decision weight corresponding to the multimodal feature under the current interview stage label;

[0050] Substitute the modal confidence MZD and the initial weight CQ corresponding to the multimodal feature into the weight calculation function to obtain the multimodal decision weight corresponding to the multimodal feature under the current interview stage label; the multimodal decision weight refers to the degree of influence of the multimodal feature when performing cheating judgment.

[0051] Furthermore, the generating of the modal confidence metric MZD according to the multimodal features includes:

[0052] Acquiring multimodal features; the multimodal features include facial micro-movement features, audio features, question-answer anomaly coefficients, and eye features;

[0053] Extract the tracking failure frames SZ and the total frames ZZ corresponding to the facial micro-motion features;

[0054] The calculation of facial confidence MBD satisfies the following formula:

[0055] Extract the audio signal-to-noise ratio (SNR) corresponding to the audio features;

[0056] The YZD calculation of audio confidence satisfies the following formula: Wherein, BSNR represents the standard signal-to-noise ratio;

[0057] Question and answer confidence is expressed as the question and answer anomaly coefficient;

[0058] Eye confidence is expressed as the maximum value of gaze abnormality coefficients in several gaze areas;

[0059] The modality confidence includes facial confidence, audio confidence, question-answering confidence, and eye confidence.

[0060] Furthermore, the abnormal threshold is set according to the historical gaze abnormality coefficient, including:

[0061] Obtain several historical monitoring videos with cheating behaviors;

[0062] Extracting a number of historical gaze anomaly coefficients from the historical monitoring video;

[0063] Selecting several maximum values ​​of historical gaze abnormality coefficients as gaze abnormality values ​​corresponding to the historical monitoring video;

[0064] The minimum value among the gaze anomaly values ​​corresponding to several historical monitoring videos is selected as the anomaly threshold.

[0065] Furthermore, the cheating identification model is constructed using an artificial intelligence model, including:

[0066] Obtaining several historical multimodal decision-making data and their corresponding historical monitoring results;

[0067] Divide a number of historical multimodal decision data and their corresponding historical monitoring results into training data, verification data, and test data; and perform data preprocessing on the training data, verification data, and test data to obtain a training set, a verification set, and a test set;

[0068] Select an artificial intelligence model as the base model;

[0069] Train the basic model using the training set, and adjust the learning rate and other hyperparameters on the validation set to obtain the pre-trained model;

[0070] By verifying the pre-trained model on the test set, we finally obtained a cheating discrimination model whose input is multimodal decision data and output is monitoring results.

[0071] Another aspect of the present invention provides a cheating detection system for online interviews, comprising: a data acquisition module and a data analysis module connected to each other;

[0072] The data acquisition module acquires monitoring videos and interview stage labels within a time window through a data acquisition device; the time window refers to the time range from the time the current question is asked to the time the next question is asked during the interview process;

[0073] The data analysis module: extracts gaze data from the monitoring video through eye tracking technology; generates a gaze anomaly coefficient based on the gaze data; when there is a gaze anomaly coefficient greater than an anomaly threshold, generates multimodal features based on the monitoring video; the anomaly threshold is set according to the historical gaze anomaly coefficient; generates multimodal decision weights based on interview stage labels; obtains multimodal decision data by fusing the multimodal decision weights and multimodal features; inputs the multimodal decision data into a cheating discrimination model to obtain monitoring results; the cheating discrimination model is constructed using an artificial intelligence model.

[0074] Compared with the prior art, the present invention has the following advantages:

[0075] 1. This application generates a gaze anomaly coefficient based on gaze data; when the gaze anomaly coefficient is greater than the anomaly threshold, generates multimodal features based on the monitoring video; generates multimodal decision weights based on the interview stage labels; obtains multimodal decision data by fusing the multimodal decision weights and multimodal features; inputs the multimodal decision data into a cheating discrimination model to obtain monitoring results, conducts a tendency assessment on cheating monitoring, and performs multimodal dynamic fusion when there is a cheating tendency to accurately identify cheating, thereby improving the efficiency and accuracy of cheating monitoring.

[0076] 2. This application considers the gaze data formed by the interviewees during the interview and compares it with the preset gaze area in the initial stage of the interview. It quantifies the abnormality of the gaze data for cheating tendencies, so that when monitoring cheating tendencies, it can be identified more accurately, providing accurate data support for improving the cheating monitoring results of the overall online interview.

[0077] 3. This application searches for the initial weight corresponding to each modal feature in the preset feature weight distribution table according to the interview stage label, and considers the modal confidence corresponding to the multimodal feature as one of the factors affecting the decision weight, so that the multimodal features are fused with fixed weights, and the combination of multimodal features is realized through adaptive weight fusion, so that the fused multimodal decision data can provide more effective information for cheating monitoring and improve the accuracy of cheating monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0079] Figure 1 This is a flowchart of the cheating detection method applied to online interviews in this application;

[0080] Figure 2 This is a schematic diagram of the cheating detection system used in online interviews in this application. DETAILED DESCRIPTION

[0081] The following will clearly and completely describe the technical solutions of this application in conjunction with the embodiments. Obviously, the embodiments described are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0082] See also Figure 1 The first embodiment of the present application provides a cheating detection method for online interviews, comprising:

[0083] Obtain monitoring videos and interview stage labels within a time window; a time window refers to the time range between asking the current question and asking the next question during the interview process;

[0084] Extract gaze data from monitoring videos using eye tracking technology;

[0085] generating a gaze abnormality coefficient based on the gaze data;

[0086] When the gaze anomaly coefficient is greater than the anomaly threshold, multimodal features are generated based on the monitoring video; the anomaly threshold is set based on the historical gaze anomaly coefficient;

[0087] Generate multimodal decision weights based on interview stage labels;

[0088] Multimodal decision data is obtained by fusing multimodal decision weights and multimodal features. Multimodal decision data refers to analytical data used for cheating identification.

[0089] The multimodal decision data is input into the cheating discrimination model to obtain the monitoring results; the cheating discrimination model is constructed through an artificial intelligence model; the monitoring results include cheating behavior and non-cheating behavior.

[0090] In this embodiment, the gaze abnormality coefficient is generated according to the gaze data, including:

[0091] Obtain gaze data; gaze data includes gaze coordinates and gaze duration;

[0092] Extract gaze data whose gaze coordinates are not within the preset gaze area; the preset gaze area refers to the normal gaze area range of the interviewee when not answering questions;

[0093] The gaze data is clustered according to the gaze coordinates to obtain several gaze areas; the gaze area includes the area ID and gaze data; the gaze data includes the total number of gazes, the gaze time point and the corresponding gaze duration;

[0094] Several adjacent fixation time intervals are obtained by performing difference calculation on adjacent fixation time points;

[0095] The average time interval PSJ and the time interval standard deviation SJB are obtained by calculating several adjacent fixation time intervals;

[0096] By calculating several fixation durations, we can obtain the average duration PS and the standard deviation SB of the duration;

[0097] The abnormal function YF(PSJ, SJB, PS, SB, ZS) is constructed through the nonlinear relationship between the average time interval PSJ, the standard deviation of the time interval SJB, the average duration PS, the standard deviation of the duration SB, the total number of fixations ZS and the fixation abnormality coefficient respectively;

[0098] The exception function satisfies the following formula:

[0099] Wherein, α represents the sensitivity coefficient, α>0, and the setting of α is to control the growth rate of the abnormal gaze coefficient. The specific value is set based on experience. In this embodiment, α is set to 1.2;

[0100] Substitute the average time interval, standard deviation of time interval, average duration, standard deviation of duration, and total number of fixations in several fixation areas into the anomaly function to calculate the fixation anomaly coefficient corresponding to the fixation area; the fixation anomaly coefficient refers to the quantitative degree of abnormal behavior.

[0101] This embodiment comprehensively considers the interviewee's eye gaze data formed during the interview process, and compares it with the preset gaze area set in the initial stage of the interview, so as to quantify the abnormality of the gaze behavior in terms of cheating tendency, so as to achieve more accurate identification when monitoring cheating tendency, and provide reliable data support for improving the accuracy of the overall online interview cheating monitoring results.

[0102] In this embodiment, the multimodal features are generated based on the monitoring video, including:

[0103] Obtain monitoring video and gaze data;

[0104] The facial monitoring area is identified by the MTCNN algorithm in the monitoring video, and the facial micro-motion features are extracted by optical flow calculation;

[0105] Extract audio data from the monitoring video, perform noise reduction on it, and extract audio features; audio features include MFCC features and fundamental frequency features;

[0106] The ASR model is used to extract question-and-answer data from the noise-reduced audio data. Question-and-answer data refers to the relevant data from the time the interviewer asks a question to the time the interviewee finishes answering the question.

[0107] Generate question-answer anomaly coefficient based on question-answer data;

[0108] Use gaze data as eye features;

[0109] Multimodal features include facial micro-movement features, audio features, question-answer anomaly coefficients, and eye features.

[0110] When conducting cheating monitoring, this embodiment not only comprehensively considers the various behavioral characteristics of the interviewees, but also incorporates the analysis of the abnormality level of their answers, and uses this abnormality level as one of the multimodal features. Through the fusion judgment of multimodal features, the accuracy and reliability of the cheating monitoring results are effectively improved.

[0111] In this embodiment, the question-answer anomaly coefficient is generated based on the question-answer data, including:

[0112] Obtain question and answer data; question and answer data includes question content, question level, answer content and thinking time SS;

[0113] The question content and answer content are input into the AI ​​similarity generation model to obtain the content AI similarity; the AI ​​similarity generation model is constructed through a large model;

[0114] Extract the expected thinking time YSS corresponding to the question level; the expected thinking time is set by experts based on the question level. The higher the question level, the longer the expected thinking time. For example, if the question level is 1, the corresponding expected thinking time is 10 seconds; if the question level is 3, the corresponding expected thinking time is 60 seconds.

[0115] The question-answering function WF(NSD, SS, YSS) is constructed based on the nonlinear relationship between the content AI similarity NSD, thinking time SS, expected thinking time YSS and the question-answering anomaly coefficient;

[0116] The question-answering function satisfies the following formula:

[0117] The question-answering anomaly coefficient is calculated by substituting the content AI similarity, thinking time, and expected thinking time into the question-answering function; the question-answering anomaly coefficient refers to the quantitative degree of abnormal behavior of the interviewee when answering.

[0118] This embodiment quantifies the degree of abnormality in the interviewee's answers through in-depth analysis of the interviewee's answer data. During the evaluation process, not only the similarity between the answer results and the AI-generated answers is considered, but also the factor of the interviewee's thinking time when answering the questions is further combined. Through the comprehensive analysis of multi-dimensional indicators, the accuracy of the judgment of the degree of abnormality of the answers is effectively improved, providing more accurate and reliable data support for subsequent cheating monitoring.

[0119] The AI ​​similarity generation model in this embodiment is constructed using a large model, including:

[0120] Obtain the AI ​​similarity of several historical questions, historical answers, and their corresponding historical content;

[0121] Divide the historical question content, historical answer content, and their corresponding historical content AI similarity into training data, verification data, and test data;

[0122] The training data, validation data and test data are preprocessed to obtain the training set, validation set and test set; the ratio between the training set, test set and validation set is 7:2:1;

[0123] Select the large model as the base model; the large model in this embodiment is the DeepSeek model;

[0124] Train the basic model using the training set, and adjust the learning rate and other hyperparameters on the validation set to obtain the pre-trained model;

[0125] By verifying the pre-trained model on the test set, we finally obtained an AI similarity generation model whose input is question content and answer content and output is content AI similarity.

[0126] In this embodiment, the multimodal decision weights are generated based on the interview stage labels, including:

[0127] Obtain interview stage labels and multimodal features;

[0128] Based on the interview stage labels, the initial weights (CQs) corresponding to the multimodal features are obtained from a preset feature weight distribution table. The preset feature weight distribution table indicates the importance of multimodal features for identifying interview cheating at different interview stages and is set by experts based on their experience.

[0129] The preset feature weight distribution table in this embodiment is as follows:

[0130]

[0131] In the preset feature weight distribution table, the first column is the interview stage label, including self-introduction, technical questions, and programming test; the corresponding row represents the initial weight of the multimodal feature;

[0132] Generate modal confidence degree MZD based on multimodal features;

[0133] Construct a weight calculation function QF(MZD, CQ) based on the nonlinear relationship between the modal confidence degree MZD, the initial weight CQ corresponding to the multimodal feature, and the multimodal decision weight corresponding to the multimodal feature under the current interview stage label;

[0134] The weight calculation function satisfies the following formula:

[0135] Wherein, β represents the confidence sensitivity coefficient, β∈(0,1), and the specific value is set according to experience. In this embodiment, β is set to 0.5; i and j both represent the modal number corresponding to the multimodal feature;

[0136] Substitute the modal confidence MZD and the initial weight CQ corresponding to the multimodal feature into the weight calculation function to obtain the multimodal decision weight corresponding to the multimodal feature under the current interview stage label; the multimodal decision weight refers to the degree of influence of the multimodal feature when making cheating judgments.

[0137] This embodiment searches for the initial weight corresponding to each modal feature in a preset feature weight distribution table based on the label information during the interview stage; at the same time, the modal confidence of the multimodal feature is used as one of the important factors affecting the decision weight. When performing multimodal feature fusion, it is no longer limited to a fixed weight distribution, but instead adopts an adaptive weight fusion method to dynamically combine the modal features, so that the fused multimodal decision data can provide more dimensional effective information for cheating monitoring, thereby significantly improving the accuracy and reliability of cheating behavior identification.

[0138] In this embodiment, the modal confidence metric MZD is generated based on the multimodal features, including:

[0139] Acquire multimodal features; multimodal features include facial micro-movement features, audio features, question-answer anomaly coefficients, and eye features;

[0140] Extract the tracking failure frames SZ and the total frames ZZ corresponding to the facial micro-motion features;

[0141] The calculation of facial confidence MBD satisfies the following formula:

[0142] Extract the audio signal-to-noise ratio (SNR) corresponding to the audio features;

[0143] The YZD calculation of audio confidence satisfies the following formula: BSNR represents the standard signal-to-noise ratio. The specific value is set based on experience. In this embodiment, BSNR is set to 20 dB.

[0144] Question and answer confidence is expressed as the question and answer anomaly coefficient;

[0145] Eye confidence is expressed as the maximum value of gaze abnormality coefficients in several gaze areas;

[0146] Modality confidence includes facial confidence, audio confidence, question-answering confidence, and eye confidence.

[0147] The abnormality threshold in this embodiment is set according to the historical gaze abnormality coefficient, including:

[0148] Obtain several historical monitoring videos with cheating behaviors;

[0149] Extract several historical gaze anomaly coefficients from historical monitoring videos;

[0150] Select several maximum values ​​of historical gaze anomaly coefficients as gaze anomaly values ​​corresponding to historical monitoring videos;

[0151] The minimum value among the gaze anomaly values ​​corresponding to several historical monitoring videos is selected as the anomaly threshold.

[0152] The cheating identification model in this embodiment is constructed using an artificial intelligence model, including:

[0153] Obtaining several historical multimodal decision-making data and their corresponding historical monitoring results;

[0154] Several historical multimodal decision-making data and their corresponding historical monitoring results are divided into training data, verification data, and test data; and data preprocessing is performed on the training data, verification data, and test data to obtain training sets, verification sets, and test sets; the ratio between the training set, test set, and verification set is 7:2:1;

[0155] Select an artificial intelligence model as the basic model; artificial intelligence models include convolutional neural network models, etc.

[0156] Train the basic model using the training set, and adjust the learning rate and other hyperparameters on the validation set to obtain the pre-trained model;

[0157] By verifying the pre-trained model on the test set, we finally obtained a cheating discrimination model whose input is multimodal decision data and output is monitoring results.

[0158] See also Figure 2 ,Another aspect of the present application provides a cheating detection system for online interviews, comprising: a data acquisition module and a data analysis module connected to each other;

[0159] Data acquisition module: This module uses data acquisition equipment to obtain monitoring videos and interview stage labels within a time window. The time window refers to the time between the current question and the next question in the interview process. The data acquisition equipment includes several sensors.

[0160] Data analysis module: extract gaze data from monitoring videos through eye tracking technology; generate gaze anomaly coefficients based on gaze data; when there is a gaze anomaly coefficient greater than the anomaly threshold, generate multimodal features based on the monitoring video; the anomaly threshold is set according to the historical gaze anomaly coefficients; generate multimodal decision weights based on interview stage labels; obtain multimodal decision data by fusing multimodal decision weights and multimodal features; input the multimodal decision data into the cheating discrimination model to obtain monitoring results; the cheating discrimination model is constructed through an artificial intelligence model.

[0161] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0162] The working principle of this application is as follows: by obtaining monitoring videos and interview stage labels within a time window; extracting gaze data from the monitoring video through eye tracking technology; generating a gaze anomaly coefficient based on the gaze data; when there is a gaze anomaly coefficient greater than the anomaly threshold, generating multimodal features based on the monitoring video; generating multimodal decision weights based on the interview stage labels; obtaining multimodal decision data by fusing the multimodal decision weights and multimodal features; inputting the multimodal decision data into a cheating discrimination model to obtain monitoring results, performing a tendency assessment on cheating monitoring, and performing multimodal dynamic fusion when there is a cheating tendency to accurately identify cheating situations, thereby improving the efficiency and accuracy of cheating monitoring, and avoiding the problem that the existing technology often uses multiple behavioral characteristics of interviewees for monitoring throughout the process, and multi-data analysis in the presence of non-cheating behavior will cause a waste of computing resources, making the cheating detection method less efficient.

[0163] The above embodiments are only used to illustrate the technical method of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present application.

Claims

1. A cheating detection method for online interviews, characterized in that: include: Obtain monitoring videos and interview stage labels within a time window; the time window refers to the time range between asking the current question and asking the next question during the interview process; Extract gaze data from monitoring videos using eye tracking technology; generating a gaze abnormality coefficient based on the gaze data; When there is a gaze anomaly coefficient greater than the anomaly threshold, a multimodal feature is generated based on the monitoring video; The abnormal threshold is set according to the historical gaze abnormality coefficient; Generate multimodal decision weights based on interview stage labels; Multimodal decision data is obtained by fusing multimodal decision weights and multimodal features; Multimodal decision data refers to analytical data used for cheating identification; The multimodal decision data is input into a cheating discrimination model to obtain a monitoring result; the cheating discrimination model is constructed using an artificial intelligence model.

2. The cheating detection method for online interviews according to claim 1, characterized in that: Generating a gaze abnormality coefficient according to the gaze data includes: Obtaining gaze data; the gaze data includes gaze coordinates and gaze duration; Extract gaze data whose gaze coordinates are not within a preset gaze area; the preset gaze area refers to the normal gaze area range of the interviewee when not answering questions; A plurality of gaze regions are obtained by clustering the gaze data according to the gaze coordinates; the gaze regions include region IDs and gaze data; the gaze data include the total number of gazes, gaze time points and their corresponding gaze durations; Several adjacent fixation time intervals are obtained by performing difference calculation on adjacent fixation time points; The average time interval PSJ and the time interval standard deviation SJB are obtained by calculating several adjacent fixation time intervals; By calculating several fixation durations, we can obtain the average duration PS and the standard deviation SB of the duration; The abnormal function YF(PSJ, SJB, PS, SB, ZS) is constructed through the nonlinear relationship between the average time interval PSJ, the standard deviation of the time interval SJB, the average duration PS, the standard deviation of the duration SB, the total number of fixations ZS and the fixation abnormality coefficient respectively; Substitute the average time interval, time interval standard deviation, average duration, duration standard deviation, and total number of gazes in several gaze areas into the abnormal function to calculate the gaze abnormality coefficient corresponding to the gaze area; the gaze abnormality coefficient refers to the quantitative degree of abnormal behavior.

3. The cheating detection method for online interviews according to claim 1, characterized in that: Generating multimodal features based on the monitoring video includes: Obtain monitoring video and gaze data; The facial monitoring area is identified by the MTCNN algorithm in the monitoring video, and the facial micro-motion features are extracted by optical flow calculation; Extract audio data from the monitoring video, perform noise reduction on it, and extract audio features; the audio features include MFCC features and fundamental frequency features; Extract question-and-answer data from the noise-reduced audio data using an ASR model; the question-and-answer data refers to the relevant data from the time the interviewer asks a question to the time the interviewee finishes answering the question during the interview; Generate question-answer anomaly coefficient based on question-answer data; Use gaze data as eye features; The multimodal features include facial micro-movement features, audio features, question-answering abnormality coefficients and eye features.

4. The cheating detection method for online interviews according to claim 3, characterized in that: Generating a question-answer anomaly coefficient based on the question-answer data includes: Obtain question and answer data; the question and answer data includes question content, question level, answer content and thinking time SS; Inputting the question content and the answer content into an AI similarity generation model to obtain content AI similarity; the AI ​​similarity generation model is constructed using a large model; Extract the expected thinking time YSS corresponding to the problem level; the expected thinking time YSS is set by the expert according to the problem level; The question-answering function WF(NSD, SS, YSS) is constructed based on the nonlinear relationship between the content AI similarity NSD, thinking time SS, expected thinking time YSS and the question-answering anomaly coefficient; The question-answer anomaly coefficient is calculated by substituting the content AI similarity, thinking time, and expected thinking time into the question-answer function; the question-answer anomaly coefficient refers to the quantitative degree of abnormal behavior of the interviewee when answering.

5. The cheating detection method for online interviews according to claim 4, characterized in that: The AI ​​similarity generation model is constructed through a large model, including: Obtain the AI ​​similarity of several historical questions, historical answers, and their corresponding historical content; Divide the historical question content, historical answer content, and their corresponding historical content AI similarity into training data, verification data, and test data; and perform data preprocessing on the training data, verification data, and test data to obtain a training set, a verification set, and a test set; Select the large model as the base model; Train the basic model using the training set, and adjust the learning rate and other hyperparameters on the validation set to obtain the pre-trained model; By verifying the pre-trained model on the test set, we finally obtained an AI similarity generation model whose input is question content and answer content and output is content AI similarity.

6. The cheating detection method for online interviews according to claim 1, characterized in that: The generation of multimodal decision weights based on interview stage labels includes: Obtain interview stage labels and multimodal features; Obtaining the initial weight CQ corresponding to the multimodal feature from a preset feature weight distribution table based on the interview stage label; the preset feature weight distribution table indicates the importance of multimodal features for identifying interview cheating at different interview stages and is set by experts based on experience; Generate modal confidence degree MZD based on multimodal features; Construct a weight calculation function QF(MZD, CQ) based on the nonlinear relationship between the modal confidence degree MZD, the initial weight CQ corresponding to the multimodal feature, and the multimodal decision weight corresponding to the multimodal feature under the current interview stage label; Substitute the modal confidence MZD and the initial weight CQ corresponding to the multimodal feature into the weight calculation function to obtain the multimodal decision weight corresponding to the multimodal feature under the current interview stage label; the multimodal decision weight refers to the degree of influence of the multimodal feature when performing cheating judgment.

7. The cheating detection method for online interviews according to claim 6, characterized in that: Generating modal confidence according to the multimodal features includes: Acquiring multimodal features; the multimodal features include facial micro-movement features, audio features, question-answer anomaly coefficients, and eye features; Extract the tracking failure frames SZ and the total frames ZZ corresponding to the facial micro-motion features; The calculation of facial confidence MBD satisfies the following formula: Extract the audio signal-to-noise ratio (SNR) corresponding to the audio features; The YZD calculation of audio confidence satisfies the following formula: Wherein, BSNR represents the standard signal-to-noise ratio; Question and answer confidence is expressed as the question and answer anomaly coefficient; Eye confidence is expressed as the maximum value of gaze abnormality coefficients in several gaze areas; The modality confidence includes facial confidence, audio confidence, question-answering confidence, and eye confidence.

8. The cheating detection method for online interviews according to claim 1, characterized in that: The abnormal threshold is set according to the historical gaze abnormality coefficient, including: Obtain several historical monitoring videos with cheating behaviors; Extracting a number of historical gaze anomaly coefficients from the historical monitoring video; Selecting several maximum values ​​of historical gaze abnormality coefficients as gaze abnormality values ​​corresponding to the historical monitoring video; The minimum value among the gaze anomaly values ​​corresponding to several historical monitoring videos is selected as the anomaly threshold.

9. The cheating detection method for online interviews according to claim 1, characterized in that: The cheating identification model is constructed using an artificial intelligence model, including: Obtaining several historical multimodal decision-making data and their corresponding historical monitoring results; Divide a number of historical multimodal decision data and their corresponding historical monitoring results into training data, verification data, and test data; and perform data preprocessing on the training data, verification data, and test data to obtain a training set, a verification set, and a test set; Select an artificial intelligence model as the base model; Train the basic model using the training set, and adjust the learning rate and other hyperparameters on the validation set to obtain the pre-trained model; By verifying the pre-trained model on the test set, we finally obtained a cheating discrimination model whose input is multimodal decision data and output is monitoring results.

10. A cheating detection system for online interviews, applied to the cheating detection method for online interviews according to any one of claims 1 to 9, characterized in that: include: Interconnected data acquisition module and data analysis module; The data acquisition module acquires monitoring videos and interview stage labels within a time window through a data acquisition device; the time window refers to the time range from the time the current question is asked to the time the next question is asked during the interview process; The data analysis module extracts gaze data from the monitoring video using eye tracking technology; generates a gaze anomaly coefficient based on the gaze data; and generates multimodal features based on the monitoring video when the gaze anomaly coefficient is greater than an anomaly threshold. The abnormality threshold is set according to the historical gaze abnormality coefficient; a multimodal decision weight is generated according to the interview stage label; multimodal decision data is obtained by fusing the multimodal decision weight and the multimodal features; and the multimodal decision data is input into the cheating discrimination model to obtain the monitoring result; The cheating identification model is constructed using an artificial intelligence model.

Images