A training examination process data real-time grabbing and abnormal marking method and system

Abstract

The application is suitable for the technical field of online training, and provides a training examination process data real-time grabbing and abnormal marking method and system, which comprises the following steps: collecting real-time video stream of examinees and establishing a personalized head posture baseline model; calculating the head posture deviation degree, recording the suspicious time window and adjusting the sampling frame rate; performing eye gaze path tracking to generate a gaze coordinate sequence; accumulating the gaze duration in the non-answer area to generate a visual line drift trigger signal; performing hand region detection to generate a hand disappearance and visual line deviation coupling label; performing foreign matter detection on the desktop area and extracting a screenshot; outputting an emotion intensity score related to cheating through a facial micro-expression verification module; and aggregating the abnormal data in a cause-effect chain to generate a structured abnormality traceability report. Through multi-modal behavior coupling and cause-effect time sequence analysis, the application realizes accurate identification of cheating behavior and construction of a traceable evidence chain, and significantly improves the reliability of online examination monitoring.

Description

Technical Field

[0001] This invention belongs to the field of online training technology, and in particular relates to a method and system for real-time data capture and anomaly labeling in the training and assessment process. Background Technology

[0002] With the popularization of online education and remote training, the online nature of training and assessment is increasing. Traditional offline centralized proctoring models are unsuitable for distributed, large-scale online assessment scenarios. Current mainstream online proctoring systems typically rely on single camera image analysis or audio / video recording playback, using manual spot checks or simple behavioral threshold detection to detect cheating. However, as cheating methods become more covert and diverse, single image detection methods are insufficient to effectively cope with complex and ever-changing cheating scenarios. There is an urgent need for a proctoring technology that can integrate multimodal data and achieve intelligent anomaly recognition.

[0003] Existing technologies include methods for head posture detection or gaze tracking using video streams captured by cameras. Some solutions determine whether a candidate is turning their head by setting a preset threshold for head angle, or calculate whether the gaze point deviates from the screen area using eye-tracking algorithms. Other solutions use a separate hand detection module to identify whether the hands have left the work area, or use image segmentation technology to identify foreign objects on the desktop. These solutions typically operate each detection step independently, triggering an alarm only when a single dimension exceeds the limit.

[0004] Current technologies detect anomalies from a single dimension, lacking comprehensive analysis of multiple behavioral modalities such as head posture, gaze direction, hand movements, foreign objects on the desk, and facial expressions. This results in high false positive and false negative rates. Furthermore, existing methods cannot establish causal relationships between scattered abnormal behaviors over time, making it difficult to form a complete chain of evidence for cheating and rendering the judgments lack interpretability and credibility. In addition, using fixed thresholds to judge head deviation fails to adapt to individual differences among test-takers, easily misjudging habitual movements of normal test-takers as cheating. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for real-time data capture and anomaly labeling in the training and assessment process, aiming to solve the technical problems existing in the prior art as identified in the background section.

[0006] This invention is implemented as follows: a method for real-time data capture and anomaly labeling in the training and assessment process, the method comprising:

[0007] By capturing real-time video streams of candidates during the assessment process using cameras, a personalized head posture baseline model is established for each candidate.

[0008] During the assessment, the deviation of the candidate's current head posture from the personalized head posture baseline model is calculated in real time. When the deviation exceeds the threshold, a suspicious time window is recorded and the camera sampling frame rate is adjusted.

[0009] Eye gaze path tracking is performed within the suspected time window to generate a gaze coordinate sequence;

[0010] Simultaneously capture the screen display content, map the gaze coordinate sequence onto the screen display content, accumulate the total time the gaze point falls on the non-answer area, and generate a gaze drift trigger signal when the total time exceeds the threshold.

[0011] When a gaze drift trigger signal is detected, hand area detection is performed. When the duration of the hands being out of the camera's field of vision exceeds a threshold, a coupled label of hand disappearance and gaze deviation is generated.

[0012] When the tag of hand disappearance and line of sight deviation is identified, foreign object detection is performed on the desktop area in the video frame. When a risky foreign object is detected, a screenshot of the foreign object area is extracted.

[0013] The screenshot of the foreign object area and the close-up of the candidate's face within the suspicious time window are input into the facial micro-expression verification module, which outputs a score of the intensity of cheating-related emotions.

[0014] The suspicious time window, the gaze coordinate sequence, the coupling label of the disappearance of the hand and the deviation of the gaze, the appearance time of the screenshot of the foreign object area, and the cheating-related emotional intensity score are aggregated into a causal chain to generate a structured anomaly tracing report.

[0015] As a further aspect of the present invention, the establishment of a personalized head posture baseline model for each examinee specifically includes:

[0016] Facial landmark detection is performed on the first five minutes of video frames in a real-time video stream. The rotation angle and pitch angle of the head in three-dimensional space are extracted in each frame to generate rotation angle sequence and pitch angle sequence.

[0017] Calculate the standard deviation of the rotation angle sequence, and take twice the standard deviation as the upper and lower limits of the normal range of head rotation angles. Calculate the average value of the pitch angle sequence as the pitch angle mean.

[0018] The number of head-turning events within the first five minutes is counted, and the number of head-turning events per minute is taken as the upper limit of the head-turning frequency. The head-turning event is defined as the absolute value of the head rotation angle exceeding 30 degrees and the duration being greater than 0.5 seconds.

[0019] As a further aspect of the present invention, the real-time calculation of the deviation of the examinee's current head posture from the personalized head posture baseline model specifically includes:

[0020] Perform head pose estimation on the current video frame to obtain the current head rotation angle and the current pitch angle. Calculate the distance between the current head rotation angle and the normal range of the head rotation angle, and the difference between the current pitch angle and the mean pitch angle.

[0021] When the distance value exceeds the standard deviation of two rotation angle sequences, it is determined to be a significant deviation. The current moment is recorded as the starting point of the suspicious time window, and the ending point is recorded when the subsequent continuous deviation exceeds 1 second.

[0022] While recording suspicious time windows, the camera sampling frame rate is increased, and then restored after the suspicious time window ends.

[0023] As a further embodiment of the present invention, the generation of the gaze coordinate sequence specifically includes:

[0024] For each video frame within the suspected time window, facial key point detection and eye center localization are performed, and the pupil center coordinates and corneal reflection point coordinates are extracted.

[0025] The gaze direction vector is calculated using the central pupil corneal reflection method. The intersection of the gaze direction vector with the screen plane equation is then obtained to get the two-dimensional coordinates of the gaze point on the screen.

[0026] The gaze coordinates corresponding to all video frames within the suspected time window are connected in chronological order to form a gaze coordinate sequence, and the dwell time corresponding to each gaze coordinate is recorded, wherein the dwell time is the length of time during which the gaze coordinate appears consecutively.

[0027] As a further embodiment of the present invention, the generation of the gaze drift trigger signal specifically includes:

[0028] Simultaneously capture screenshots of the current screen display content, identify the layout of the answer interface, and determine the boundary boxes of the valid answer area outside the screen edge and the camera blind spot;

[0029] Each gaze coordinate in the gaze coordinate sequence is compared with each bounding box to determine the region to which the gaze coordinate belongs. When the gaze coordinate falls outside the screen edge or the camera blind spot, the dwell time corresponding to the gaze coordinate is added to the total time in the non-answer area.

[0030] When the total time in the non-answer area exceeds 10 seconds, a gaze drift trigger signal is generated, and the gaze drift trigger signal is output together with the timestamp of the current suspicious time window.

[0031] As a further aspect of the present invention, the generation of the hand disappearance and gaze deviation coupling label specifically includes:

[0032] When a gaze drift trigger signal is detected, the hand region detection module is activated. Using an ellipse model based on skin color segmentation and a contour detection method, the bounding boxes of the left and right hand regions are located in the video frame.

[0033] When the areas of both the left and right hand regions are smaller than the preset area threshold and the state lasts for more than 1 second, it is determined that both hands are completely out of the camera's field of vision. The start timestamp of the time of separation is recorded, and the end timestamp is recorded when the hands reappear.

[0034] During the period when both hands are out of sight, the gaze coordinates in the gaze coordinate sequence are continuously monitored. If the gaze coordinates are still within the screen display area and the time of separation exceeds 2 seconds, a hand disappearance and gaze deviation coupling label containing the time of separation value and the gaze coordinate subsequence within the time of separation is generated.

[0035] As a further aspect of the present invention, the step of extracting a screenshot of the foreign object region when a risky foreign object is detected specifically includes:

[0036] When the tag of hand disappearance and gaze deviation is detected, the coordinate range of the desktop area in the video frame is calculated as the desktop detection sub-image based on the lower edge position of the candidate's face bounding box in the current video frame.

[0037] The desktop detection sub-image is segmented, and the presence of risky objects is identified.

[0038] When the confidence level of a risky object exceeds a preset threshold, the smallest bounding rectangle of the area where the risky object is located is captured as a screenshot of the foreign object area, and the category label of the risky object is recorded at the same time.

[0039] As a further aspect of the present invention, the output of the cheating-related emotional intensity score specifically includes:

[0040] Extract close-up image sequences of the examinee’s face from video frames within the suspicious time window, and input the screenshot of the foreign object area as auxiliary context into the facial micro-expression verification module;

[0041] Each frame in the close-up face image sequence is encoded with facial action units, and the intensity value of the frowning action unit is extracted, wherein the frowning action unit corresponds to the amplitude of the action of the eyebrows converging towards the center of the eyebrows;

[0042] When the intensity value of the frowning action unit is greater than the preset frowning threshold, the output of the cheating-related emotion intensity score is a tense state; otherwise, the output of the calm state is output.

[0043] As a further aspect of the present invention, the generation of the structured anomaly tracing report specifically includes:

[0044] Using the suspicious time window as the time axis reference, the non-answer area gaze period in the gaze coordinate sequence, the hand separation period in the hand disappearance and gaze deviation coupling label, the appearance time of the foreign object area screenshot, and the tension period in the cheating-related emotional intensity score are arranged in chronological order.

[0045] The time periods are connected in chronological order to form a causal evolution chain, in which the previous time period serves as the triggering premise for the next time period. Each link in the chain is assigned a confidence percentage based on the consistency of upstream and downstream time, and a structured anomaly tracing report is generated.

[0046] Another objective of this invention is to provide a real-time data capture and anomaly labeling system for the training and assessment process, the system comprising:

[0047] The video acquisition and baseline modeling module is used to acquire real-time video streams of candidates during the assessment process through cameras and establish a personalized head posture baseline model for each candidate.

[0048] The deviation detection module is used to calculate the degree of deviation of the examinee's current head posture from the personalized head posture baseline model in real time during the assessment process. When the deviation exceeds the threshold, a suspicious time window is recorded and the camera sampling frame rate is adjusted.

[0049] The gaze path tracking module is used to track the eye gaze path within the suspected time window and generate a gaze coordinate sequence.

[0050] The gaze drift detection module is used to synchronously capture the screen display content, map the gaze coordinate sequence onto the screen display content, accumulate the total time the gaze point falls on the non-answer area, and generate a gaze drift trigger signal when the total time exceeds the threshold.

[0051] The hand detection module is used to perform hand area detection when a gaze drift trigger signal is detected. When the duration of the hands being out of the camera's field of view exceeds a threshold, a coupled label of hand disappearance and gaze deviation is generated.

[0052] The foreign object detection module is used to detect foreign objects in the desktop area of ​​the video frame when the disappearance of the hand and the deviation of the line of sight are detected. When a risky foreign object is detected, a screenshot of the foreign object area is extracted.

[0053] The facial expression verification module is used to input the screenshot of the foreign object area and the close-up of the candidate's face within the suspicious time window into the facial micro-expression verification module, and output a score of the intensity of cheating-related emotions.

[0054] The anomaly tracing report module aggregates the suspicious time window, the gaze coordinate sequence, the coupling label of the disappearance of the hand and the deviation of the gaze, the appearance time of the screenshot of the foreign object area, and the cheating-related emotional intensity score by performing causal chain aggregation to generate a structured anomaly tracing report.

[0055] The beneficial effects of this invention are:

[0056] This invention achieves accurate identification and traceable analysis of cheating behavior by candidates through multimodal data fusion and causal chain aggregation. Establishing a personalized head posture baseline model eliminates the impact of individual differences on detection, significantly reducing the false alarm rate. A strategy of triggering suspicious time windows based on head deviation and dynamically adjusting the sampling frame rate ensures data collection density during critical periods while conserving computational resources. Through the hierarchical coupling of gaze drift, hand disappearance, desktop foreign object detection, and facial micro-expression verification, a closed-loop evidence system from behavioral anomalies to emotional confirmation is formed. The resulting structured anomaly tracing report connects scattered abnormal events in chronological causal order, assigning a confidence percentage to each link, giving cheating determination a complete logical chain and interpretability, facilitating manual review and dispute resolution, and significantly improving the intelligence and reliability of online training and assessment proctoring systems. Attached Figure Description

[0057] Figure 1 A flowchart of a method and system for real-time data capture and anomaly labeling during training and assessment provided in an embodiment of the present invention;

[0058] Figure 2 A flowchart for establishing a personalized head posture baseline model for each examinee, provided for embodiments of the present invention;

[0059] Figure 3 A flowchart for recording suspicious time windows and adjusting the camera sampling frame rate provided in an embodiment of the present invention;

[0060] Figure 4 A flowchart for generating gaze coordinate sequences provided in an embodiment of the present invention;

[0061] Figure 5 This is a flowchart for generating a gaze drift trigger signal provided in an embodiment of the present invention;

[0062] Figure 6 A flowchart for generating a tag coupling hand disappearance and line of sight deviation provided in an embodiment of the present invention;

[0063] Figure 7 This is a flowchart of the extraction of the foreign object region screenshot provided in an embodiment of the present invention;

[0064] Figure 8 A flowchart for outputting a cheating-related emotional intensity score provided in an embodiment of the present invention;

[0065] Figure 9 A flowchart for generating a structured anomaly tracing report provided in an embodiment of the present invention;

[0066] Figure 10 The diagram below shows the structure of a method and system for real-time data capture and anomaly labeling during the training and assessment process, as provided in this embodiment of the invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] Figure 1 A flowchart of a method and system for real-time data capture and anomaly labeling in a training and assessment process, as provided in an embodiment of the present invention, is shown below. Figure 1 As shown, the method includes:

[0069] The S100 uses a camera to capture real-time video streams during the candidate's assessment process and establishes a personalized head posture baseline model for each candidate.

[0070] During the training and assessment process, cameras are used to capture the complete video stream of each candidate during the assessment in real time. Based on the first five minutes of normal answering video frames, a personalized head posture baseline model is established for each candidate. The core purpose of this step is to eliminate the impact of individual differences on the detection of abnormal behavior, because different candidates have different heights, sitting postures, and habitual head angles. Using a uniform threshold can easily lead to misjudgments or missed judgments.

[0071] The three-dimensional rotation and pitch angle sequences of the head are extracted from consecutive video frames. The standard deviation of the rotation angle and the mean of the pitch angle are calculated to construct the normal range of head movement and the baseline pitch angle for the examinee. At the same time, the upper limit of the frequency of head turning events is calculated to distinguish between normal screen viewing and suspicious gaze deviation behavior.

[0072] S200: During the assessment process, the deviation of the candidate's current head posture from the personalized head posture baseline model is calculated in real time. When the deviation exceeds the threshold, a suspicious time window is recorded and the camera sampling frame rate is adjusted.

[0073] During the assessment, the system continuously monitors the deviation between the candidate's current head posture and the established personalized baseline model. By estimating the head posture for each frame of video, the system obtains the current head rotation angle and pitch angle, and calculates the distance of the rotation angle from the normal range and the difference between the pitch angle and the mean baseline.

[0074] When the deviation of the rotation angle exceeds two standard deviations, it is considered a significant deviation, and the current moment is recorded as the starting point of the suspicious time window. If the deviation continues for more than one second, the end point of the window is recorded. This design can effectively distinguish between purposeful, continuous cheating behavior and brief, unconscious head movements, because any conscious action takes at least one second to complete, thus avoiding false alarms triggered by normal actions such as blinking or adjusting posture.

[0075] Once a suspicious time window is triggered, the system immediately increases the camera's sampling frame rate, and then restores the original frame rate after the window ends. The purpose of dynamically adjusting the frame rate is to collect more intensive video data during critical periods when abnormal behavior may occur, providing high temporal resolution image input for subsequent eye gaze path tracking, hand detection, and micro-expression analysis, while maintaining a low frame rate during normal periods to save computing resources and network bandwidth.

[0076] S300, perform eye gaze path tracking within the suspected time window and generate a gaze coordinate sequence;

[0077] Within a suspicious time window, the system initiates eye gaze path tracking, quantifying the examinee's gaze behavior into an analyzable sequence of gaze coordinates. Specifically, this involves detecting facial key points and locating the eye center in each frame of video within the window, extracting the pupil center and corneal reflection point, calculating the gaze direction vector using the pupil-center-corneal reflection method, and then finding the intersection of the gaze vector with the screen plane equation to obtain the two-dimensional coordinates of the gaze point on the screen. The gaze coordinates of all frames are connected chronologically, and the duration of each coordinate's dwell time is recorded to form complete gaze trajectory data.

[0078] Transforming invisible line-of-sight movements into spatially localizable and temporally traceable quantitative information provides direct evidence for determining whether a candidate's gaze has strayed from the answer area. The advantage of using the central pupillary corneal reflex method lies in its ability to effectively compensate for errors caused by minute head movements by utilizing the corneal reflex point as a reference, resulting in more stable and accurate fixation point calculations. Recording the duration of fixation further reveals the candidate's level of attention engagement in different areas; simple fixation coordinates only indicate where the gaze was focused, while fixation duration reflects how long it lasted.

[0079] By generating a gaze coordinate sequence, the system can completely capture the candidate's visual attention trajectory, making it impossible to hide any gaze behavior that deviates from the answer area, thus constructing a quantitative evidence chain of gaze drift in the time dimension.

[0080] S400, synchronously capture the screen display content, map the gaze coordinate sequence onto the screen display content, accumulate the total time the gaze point falls on the non-answer area, and generate a gaze drift trigger signal when the total time exceeds the threshold.

[0081] After obtaining the gaze coordinate sequence, the system simultaneously captures a screenshot of the current screen display and identifies the layout of the answer interface, determining the bounding boxes of the valid answer area, the outside of the screen edge, and the camera blind spot. Each gaze coordinate in the gaze coordinate sequence is compared with each bounding box to determine the region to which the gaze point belongs. When a gaze coordinate falls outside the screen edge or the camera blind spot, the dwell time corresponding to that gaze coordinate is added to the total time outside the answer area.

[0082] When the total time spent outside the answer area exceeds a threshold, a gaze drift trigger signal is generated and output along with the timestamp of the current suspicious time window. The core function of this step is to align the eye's gaze trajectory with the actual content space on the screen, thereby accurately determining whether the candidate's gaze has truly left the answer area. The area outside the screen edge corresponds to the candidate looking at the external environment beyond the screen, and the camera blind spot corresponds to corner areas not covered by the camera. Both of these are high-risk areas for peeking at external materials or communicating with others. The valid answer area, timer area, and other normal functional areas within the screen are excluded to prevent normal viewing of questions or remaining time from being mistakenly judged as cheating.

[0083] By triggering the signal through the total duration of fixation on non-answer areas rather than a single, brief fixation, unconscious eye movements or brief external distractions caused by the test-taker's thinking can be effectively filtered out. Only continuous or repeated focus on non-answer areas is considered suspicious. The generation of the gaze drift trigger signal marks the system's transition from head posture abnormality screening to gaze behavior verification. This signal provides a precise activation opportunity for the subsequent hand detection module, forming a progressive abnormality judgment chain of head deviation → gaze drift.

[0084] When the S500 detects a gaze drift trigger signal, it performs hand area detection. When the duration of the hands being out of the camera's field of vision exceeds a threshold, it generates a coupled label of hand disappearance and gaze deviation.

[0085] Once the gaze drift trigger signal is detected, the hand region detection module is immediately activated. Using an elliptical model based on skin color segmentation and contour detection methods, the bounding boxes of the left and right hand regions are located in the video frame. When the areas of both the left and right hand regions are smaller than a preset area threshold and this state lasts for more than one second, it is determined that both hands have completely left the camera's field of vision. The start timestamp of the timeout is recorded, and the end timestamp is recorded when the hands reappear. During the period when the hands are out of the field of vision, the gaze coordinates in the gaze coordinate sequence are continuously monitored. If the gaze coordinates still fall within the screen display area and the timeout exceeds two seconds, a hand disappearance and gaze deviation coupling label containing the timeout duration and the gaze coordinate subsequence within the timeout period is generated.

[0086] By jointly verifying abnormal gaze and hand behavior, a dual-modal coupled anomaly detection criterion is formed. Gaze drift itself suggests the test-taker may be looking at areas outside the answer area, but drawing conclusions solely based on insufficient gaze is insufficient, as the test-taker may only be briefly distracted. When gaze drift occurs simultaneously with both hands completely out of the camera's field of view, and the duration of this separation exceeds a threshold, the likelihood of cheating increases significantly. Setting a one-second threshold filters out brief hand movements during normal answering, such as page turning or adjusting the keyboard and mouse; only sustained hand disappearance is associated with actively retrieving concealed items or viewing external materials. Furthermore, it requires that the gaze coordinates remain within the screen during the hand separation, rather than moving with the hands, excluding legitimate behaviors such as looking down at notes, further focusing on abnormal activities in the screen direction. The generated hand disappearance and gaze deviation coupling label packages the anomaly information from both modalities, including the duration of separation and the gaze trajectory subsequence.

[0087] S600 detects foreign objects in the desktop area of ​​the video frame when it recognizes the tag that the hand disappears and the line of sight deviates from the coupling. When a risky foreign object is detected, it extracts a screenshot of the foreign object area.

[0088] Once the coupling label of missing hand and gaze deviation is detected, foreign object detection is immediately performed on the desktop area in the video frame. Specifically, based on the lower edge position of the candidate's face bounding box in the current video frame, the coordinate range of the desktop area in the video frame is calculated as a desktop detection sub-image. This sub-image is then segmented to identify the presence of risky objects. When the confidence level of a detected risky object exceeds a preset threshold, the smallest bounding rectangle of the area containing the object is cropped as a foreign object region screenshot, and the category label of the risky object is recorded.

[0089] The previously accumulated multimodal anomaly signals, such as head deviation, gaze drift, and hand detachment, are focused on specific foreign object evidence in physical space. The coupling of missing hands and gaze deviation strongly suggests that the candidate may be retrieving or viewing hidden items. At this point, targeted foreign object detection is performed on the desktop area to directly verify the presence of cheating tools such as mobile phones, books, and notes. The desktop area is estimated by inferring the lower edge of the face bounding box, utilizing prior knowledge of human spatial relationships—that the area below the face is usually the desktop area in front of the candidate. This dynamic positioning method does not require pre-calibration of the camera angle and can adapt to different candidate sitting heights and camera installation positions. Once a risky foreign object is detected, the smallest bounding rectangle screenshot is immediately extracted, preserving the precise location and visual characteristics of the foreign object in the video, providing intuitive visual evidence for subsequent micro-expression verification and report generation.

[0090] Transforming abstract abnormal behavior signals into concrete, viewable images of foreign objects upgrades cheating detection from behavioral inference to physical evidence confirmation, greatly enhancing the credibility and traceability of the results. Simultaneously, recording risk category labels facilitates subsequent statistical analysis and classification, such as distinguishing between different cheating methods like electronic devices and paper materials.

[0091] S700, input the screenshot of the foreign object area and the close-up of the candidate's face in the suspicious time window into the facial micro-expression verification module, and output the cheating-related emotion intensity score;

[0092] After obtaining a screenshot of the foreign object area, the system will combine the sequence of close-up images of the examinee's face within the suspicious time window with the screenshot of the foreign object. Figure 1 The data is then input into a facial micro-expression verification module. This module encodes facial motion units for each frame in the close-up facial sequence, focusing on extracting the intensity value of the frowning motion unit. This intensity reflects the amplitude of the movement as the brow contracts towards the center of the forehead. When the frowning intensity exceeds a preset threshold, a cheating-related emotional intensity score is output as a tense state; otherwise, a calm state is output.

[0093] The system cross-validates visual behavioral evidence with the candidate's psychophysiological state. While behavioral cues such as head deviation, gaze drift, disappearing hands, and foreign objects on the desk are highly suspicious, they theoretically still have plausible explanations. By introducing facial micro-expression analysis, the system can capture the candidate's genuine emotional reactions such as tension and anxiety during cheating. These reactions are usually beyond conscious control, thus adding a physiological dimension independent of behavioral actions to the cheating determination. The frowning action unit is highly correlated with emotional states of tension and impaired thinking, and is more stable to detect than other facial units such as smiling and blinking, and is less affected by lighting and head angle.

[0094] Using screenshots of foreign object areas as supplementary contextual input helps the module understand the current situation. For example, when a phone appears on the desktop, an increase in the intensity of frowning significantly increases the confidence level of the tension score. By outputting binary classification scores of tension and calmness, the system avoids the computational burden and instability caused by complex emotion classification.

[0095] This step extends the entire anomaly detection chain from behavioral observation to psychological verification, effectively preventing misjudgments caused by candidates' habitual small movements or camera angles. Only when behavioral abnormalities and emotional tension occur simultaneously can the inference of cheating have a complete evidentiary loop.

[0096] S800, the suspicious time window, the gaze coordinate sequence, the coupling label of the disappearance of the hand and the deviation of the gaze, the appearance time of the screenshot of the foreign object area, and the cheating-related emotional intensity score are aggregated into a causal chain to generate a structured anomaly tracing report.

[0097] like Figure 2 As shown, establishing a personalized head pose baseline model for each examinee specifically includes:

[0098] S110 performs facial landmark detection on the first five minutes of video frames in the real-time video stream, extracts the rotation angle and pitch angle of the head in three-dimensional space in each frame, and generates rotation angle sequence and pitch angle sequence.

[0099] Five minutes before the start of the exam, candidates are usually in a normal answering state. The head rotation and pitch angle data collected at this time can represent the candidate's normal behavior pattern, providing a personalized reference standard for subsequent abnormal head movement detection and avoiding misjudgment due to individual differences.

[0100] S120, calculate the standard deviation of the rotation angle sequence, and take twice the standard deviation as the upper and lower limits of the normal range of head rotation angles. Calculate the average value of the pitch angle sequence as the pitch angle mean.

[0101] Standard deviation of rotation angle sequence: ;

[0102] Normal range of head rotation angle: ;

[0103] The average value of the pitch angle sequence: ;

[0104] in:

[0105] The standard deviation of the rotation angle sequence reflects the dispersion of the head rotation angle.

[0106] This represents the total number of video frames in the first five minutes.

[0107] For the first The detected three-dimensional rotation angle value of the head in the frame;

[0108] This is the arithmetic mean of the rotation angle sequence;

[0109] For the first The head pitch angle value detected in the frame;

[0110] The average value of the pitch angle sequence is used as the baseline pitch angle under normal answering conditions.

[0111] S130, count the number of head turning events in the first five minutes, and use the number of head turning events per minute as the upper limit of head turning frequency. The head turning event is defined as the absolute value of the head rotation angle exceeding 30 degrees and the duration being greater than 0.5 seconds.

[0112] According to ergonomic research, during normal test-taking, test-takers typically rotate their heads no more than 25 degrees left or right to see the screen content clearly. Rotations exceeding 30 degrees indicate looking outside the screen, which is highly correlated with cheating behavior such as peeking at the answers of other test-takers or external materials. A 0.5-second time threshold effectively filters out unintentional rapid head turns, instantaneous head turns caused by sudden external sounds, and slight head movements during blinking. Both conditions must be met simultaneously for a valid head turn event to be considered, ensuring a high detection rate for genuine cheating while significantly reducing false alarms.

[0113] like Figure 3 As shown, the real-time calculation of the deviation of the examinee's current head posture from the personalized head posture baseline model specifically includes:

[0114] S210, perform head pose estimation on the current video frame to obtain the current head rotation angle and the current pitch angle, calculate the distance between the current head rotation angle and the normal range of the head rotation angle, and the difference between the current pitch angle and the average pitch angle.

[0115] S220, when the distance value exceeds the standard deviation of two rotation angle sequences, it is determined to be a significant deviation, the current moment is recorded as the starting point of the suspicious time window, and the ending point is recorded when the subsequent continuous deviation exceeds 1 second.

[0116] One second is the shortest time threshold for a human to complete a conscious action. Any intentional cheating behavior requires at least one second to complete. This eliminates the possibility of momentary head deviation caused by accidental factors such as instantaneous distraction, eye fatigue causing blinking, or adjusting posture.

[0117] The S230 increases the camera's sampling frame rate while recording suspicious time windows, and resumes recording after the suspicious time window ends.

[0118] like Figure 4 As shown, the generation of the gaze coordinate sequence specifically includes:

[0119] S310, perform facial key point detection and eye center localization on each video frame within the suspicious time window, and extract the pupil center coordinates and corneal reflection point coordinates;

[0120] S320, the gaze direction vector is calculated using the central corneal reflection method of the pupil, and the intersection point of the gaze direction vector with the screen plane equation is obtained to obtain the two-dimensional coordinates of the gaze point on the screen;

[0121] ;

[0122] in:

[0123] The normalized unit vector of the line of sight;

[0124] The two-dimensional coordinates of the pupil center in the image coordinate system;

[0125] This represents the two-dimensional coordinates of the corneal reflector (Pulchin spot) in the image coordinate system.

[0126] The Euclidean distance between the center of the pupil and the corneal reflex point;

[0127] The direction of human eye's line of sight can be calculated by the relative positional relationship between the center of the pupil and the corneal reflex point.

[0128] S330, connect the gaze coordinates corresponding to all video frames within the suspicious time window in chronological order to form a gaze coordinate sequence, and record the dwell time corresponding to each gaze coordinate, wherein the dwell time is the length of time the gaze coordinate appears consecutively.

[0129] Simply focusing on a coordinate point only reflects "where the test taker looked," not "how long they looked." Dwell time is the key indicator for judging whether attention is abnormal. When answering questions normally, the test taker's eyes will move quickly in the question area, and the dwell time at a single coordinate point is usually short. However, when cheating, the test taker will stare at a fixed area outside the screen for a long time. Dwell time data can be used to calculate the proportion of attention in different areas.

[0130] like Figure 5 As shown, the generation of the gaze drift trigger signal specifically includes:

[0131] The S410 simultaneously captures screenshots of the currently displayed content on the screen, identifies the layout of the answer interface, and determines the boundary boxes outside the effective answer area of ​​the screen and the camera blind spot.

[0132] S420, compare each gaze coordinate in the gaze coordinate sequence with each bounding box to determine the region to which the gaze coordinate belongs. When the gaze coordinate falls outside the screen edge or the camera blind spot, add the dwell time corresponding to the gaze coordinate to the total time in the non-answer area.

[0133] The edges of the screen and the camera blind spot are high-risk areas where candidates' eyes are off the answer interface, directly corresponding to cheating behaviors such as peeking at external materials or communicating with others; while the timer area and other areas inside the screen are functional areas that need to be viewed during normal answering, excluding them can greatly reduce the probability of normal operation being mistaken for cheating.

[0134] S430, when the total time in the non-answer area exceeds 10 seconds, a gaze drift trigger signal is generated, and the gaze drift trigger signal is output together with the timestamp of the current suspicious time window.

[0135] like Figure 6 As shown, the generation of the hand disappearance and gaze deviation coupling label specifically includes:

[0136] S510: When a gaze drift trigger signal is detected, the hand region detection module is activated. Using an ellipse model based on skin color segmentation and contour detection method, the bounding boxes of the left and right hand regions are located in the video frame.

[0137] S520: When the areas of both the left and right hand regions are smaller than the preset area threshold and the state lasts for more than 1 second, it is determined that both hands are completely out of the camera's field of vision. The start timestamp of the time of separation is recorded, and the end timestamp is recorded when the hands reappear.

[0138] During normal test-taking, candidates may make brief hand movements, usually lasting less than one second. Genuine cheating requires both hands to be completely out of the camera's field of vision for a sustained period. A one-second time threshold effectively filters out these normal, brief hand movements, preventing them from being misjudged as abnormal, while ensuring the detection of genuine cheating.

[0139] S530, during the period when both hands are out of sight, continuously monitor the gaze coordinates in the gaze coordinate sequence. If the gaze coordinates are still within the screen display area and the time of separation exceeds 2 seconds, generate a hand disappearance and gaze deviation coupling label containing the time of separation value and the gaze coordinate subsequence within the time of separation.

[0140] like Figure 7 As shown, the step of extracting a screenshot of the foreign object area when a risky foreign object is detected specifically includes:

[0141] S610, when the tag of hand disappearance and gaze deviation is detected, the coordinate range of the desktop area in the video frame is calculated as the desktop detection sub-image based on the lower edge position of the candidate's face bounding box in the current video frame;

[0142] S620, perform image segmentation on the desktop detection sub-image and identify whether there are risky objects;

[0143] S630, when the confidence level of a risky object exceeds a preset threshold, the smallest bounding rectangle of the area where the risky object is located is captured as a screenshot of the foreign object area, and the category label of the risky object is recorded at the same time.

[0144] like Figure 8 As shown, the output of the cheating-related emotional intensity score specifically includes:

[0145] S710, extract the candidate's facial close-up image sequence from the video frames within the suspicious time window, and input the screenshot of the foreign object area as auxiliary context into the facial micro-expression verification module;

[0146] S720, each frame in the close-up face image sequence is encoded with facial action units, and the intensity value of the frowning action unit is extracted, wherein the frowning action unit corresponds to the amplitude of the action of the eyebrows converging towards the center of the eyebrows.

[0147] ;

[0148] in:

[0149] This is the intensity value of the frowning action unit (ActionUnit4), ranging from 0 to 1;

[0150] This is the baseline distance between the key points of the left and right eyebrows in a neutral expression state.

[0151] This represents the actual distance between the left and right eyebrow key points in the current frame.

[0152] When the brows contract towards the center of the brow... Less than The intensity value increases accordingly, reflecting the amplitude of the frowning action.

[0153] Frowning (AU4) is the most typical and easily detected facial expression when humans are tense, anxious, or have difficulty thinking, and it is highly correlated with the psychological state when cheating. Other facial action units, such as smiling and blinking, have low correlation with cheating emotions and are easily interfered with. Extracting the frowning action separately can reduce computational complexity while ensuring detection effectiveness.

[0154] S730, when the intensity value of the frowning action unit is greater than the preset frowning threshold, the cheating-related emotion intensity score is output as a tense state; otherwise, a calm state is output.

[0155] like Figure 9 As shown, the generation of the structured anomaly tracing report specifically includes:

[0156] S810, using the suspicious time window as the time axis reference, arrange the non-answer area gaze period in the gaze coordinate sequence, the hand separation period in the hand disappearance and gaze deviation coupling label, the appearance time of the foreign object area screenshot, and the tension period in the cheating-related emotional intensity score in chronological order;

[0157] S820 connects the various time periods in chronological order to form a causal evolution chain, where the previous time period serves as the triggering premise for the next time period. It assigns a confidence percentage to each link in the chain based on the consistency of upstream and downstream time, and generates a structured anomaly tracing report.

[0158] ;

[0159] in:

[0160] For the first in the chain The credibility percentage of each step;

[0161] For the first in the causal evolution chain Each link (upstream link);

[0162] For the first in the causal evolution chain Each step (current step);

[0163] This refers to the overlap between the end time of the upstream stage and the start time of the current stage.

[0164] This represents the total duration of the current segment;

[0165] The credibility of causal relationships between upstream and downstream processes can be assessed by evaluating the degree of time overlap between them.

[0166] By connecting scattered abnormal behaviors in chronological order, a causal logical chain of "abnormal action → gaze deviation → emotional tension" is formed. Each link serves as a triggering premise for the next, which can verify the rationality and relevance of the behavior, avoid misjudgment caused by a single abnormal point, and upgrade the cheating judgment from "single point evidence" to "logical closed loop".

[0167] Figure 10 A structural block diagram of a real-time data capture and anomaly labeling system for the training and assessment process provided in this embodiment of the invention is shown below. Figure 10 As shown, the system comprises:

[0168] The video acquisition and baseline modeling module 100 is used to acquire real-time video streams during the candidate's assessment process via camera and establish a personalized head posture baseline model for each candidate.

[0169] The deviation detection module 200 is used to calculate the degree of deviation of the candidate's current head posture from the personalized head posture baseline model in real time during the assessment process. When the deviation exceeds the threshold, a suspicious time window is recorded and the camera sampling frame rate is adjusted.

[0170] The gaze path tracking module 300 is used to perform eye gaze path tracking within the suspected time window and generate a gaze coordinate sequence.

[0171] The gaze drift detection module 400 is used to synchronously capture the screen display content, map the gaze coordinate sequence onto the screen display content, accumulate the total time the gaze point falls on the non-answer area, and generate a gaze drift trigger signal when the total time exceeds a threshold.

[0172] The hand detection module 500 is used to perform hand area detection when a gaze drift trigger signal is detected. When the duration of the hands being out of the camera's field of view exceeds a threshold, a hand disappearance and gaze deviation coupling label is generated.

[0173] The foreign object detection module 600 is used to detect foreign objects in the desktop area of ​​the video frame when the tag of hand disappearance and line of sight deviation is detected. When a risky foreign object is detected, a screenshot of the foreign object area is extracted.

[0174] The facial expression verification module 700 is used to input the screenshot of the foreign object area and the close-up of the candidate's face within the suspicious time window into the facial micro-expression verification module, and output a score of the intensity of cheating-related emotions.

[0175] The anomaly tracing report module 800 aggregates the suspicious time window, the gaze coordinate sequence, the coupling label of the disappearance of the hand and the deviation of the gaze, the appearance time of the screenshot of the foreign object area, and the cheating-related emotional intensity score by performing causal chain aggregation to generate a structured anomaly tracing report.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time data capture and anomaly labeling in a training and assessment process, characterized in that, The method includes: By capturing real-time video streams of candidates during the assessment process using cameras, a personalized head posture baseline model is established for each candidate. During the assessment, the deviation of the candidate's current head posture from the personalized head posture baseline model is calculated in real time. When the deviation exceeds the threshold, a suspicious time window is recorded and the camera sampling frame rate is adjusted. Eye gaze path tracking is performed within the suspected time window to generate a gaze coordinate sequence; Simultaneously capture the screen display content, map the gaze coordinate sequence onto the screen display content, accumulate the total time the gaze point falls on the non-answer area, and generate a gaze drift trigger signal when the total time exceeds the threshold. When a gaze drift trigger signal is detected, hand area detection is performed. When the duration of the hands being out of the camera's field of vision exceeds a threshold, a coupled label of hand disappearance and gaze deviation is generated. When the tag of hand disappearance and line of sight deviation is identified, foreign object detection is performed on the desktop area in the video frame. When a risky foreign object is detected, a screenshot of the foreign object area is extracted. The screenshot of the foreign object area and the close-up of the candidate's face within the suspicious time window are input into the facial micro-expression verification module, which outputs a score of the intensity of cheating-related emotions. The suspicious time window, the gaze coordinate sequence, the coupling label of the disappearance of the hand and the deviation of the gaze, the appearance time of the screenshot of the foreign object area, and the cheating-related emotional intensity score are aggregated into a causal chain to generate a structured anomaly tracing report.

2. The method according to claim 1, characterized in that, The establishment of a personalized head posture baseline model for each candidate specifically includes: Facial landmark detection is performed on the first five minutes of video frames in a real-time video stream. The rotation angle and pitch angle of the head in three-dimensional space are extracted in each frame to generate rotation angle sequence and pitch angle sequence. Calculate the standard deviation of the rotation angle sequence, and take twice the standard deviation as the upper and lower limits of the normal range of head rotation angles. Calculate the average value of the pitch angle sequence as the pitch angle mean. The number of head-turning events within the first five minutes is counted, and the number of head-turning events per minute is taken as the upper limit of the head-turning frequency. The head-turning event is defined as the absolute value of the head rotation angle exceeding 30 degrees and the duration being greater than 0.5 seconds.

3. The method according to claim 2, characterized in that, The real-time calculation of the deviation of the examinee's current head posture from the personalized head posture baseline model specifically includes: Perform head pose estimation on the current video frame to obtain the current head rotation angle and the current pitch angle. Calculate the distance between the current head rotation angle and the normal range of the head rotation angle, and the difference between the current pitch angle and the mean pitch angle. When the distance value exceeds the standard deviation of two rotation angle sequences, it is determined to be a significant deviation. The current moment is recorded as the starting point of the suspicious time window, and the ending point is recorded when the subsequent continuous deviation exceeds 1 second. While recording suspicious time windows, the camera sampling frame rate is increased, and then restored after the suspicious time window ends.

4. The method according to claim 3, characterized in that, The generation of the gaze coordinate sequence specifically includes: For each video frame within the suspected time window, facial key point detection and eye center localization are performed, and the pupil center coordinates and corneal reflection point coordinates are extracted. The gaze direction vector is calculated using the central pupil corneal reflection method. The intersection of the gaze direction vector with the screen plane equation is then obtained to get the two-dimensional coordinates of the gaze point on the screen. The gaze coordinates corresponding to all video frames within the suspected time window are connected in chronological order to form a gaze coordinate sequence, and the dwell time corresponding to each gaze coordinate is recorded, wherein the dwell time is the length of time during which the gaze coordinate appears consecutively.

5. The method according to claim 4, characterized in that, The generation of the gaze drift trigger signal specifically includes: Simultaneously capture screenshots of the current screen display content, identify the layout of the answer interface, and determine the boundary boxes of the valid answer area outside the screen edge and the camera blind spot; Each gaze coordinate in the gaze coordinate sequence is compared with each bounding box to determine the region to which the gaze coordinate belongs. When the gaze coordinate falls outside the screen edge or the camera blind spot, the dwell time corresponding to the gaze coordinate is added to the total time in the non-answer area. When the total time in the non-answer area exceeds 10 seconds, a gaze drift trigger signal is generated, and the gaze drift trigger signal is output together with the timestamp of the current suspicious time window.

6. The method according to claim 5, characterized in that, The specific components of generating the hand disappearance and gaze deviation coupling label include: When a gaze drift trigger signal is detected, the hand region detection module is activated. Using an ellipse model based on skin color segmentation and a contour detection method, the bounding boxes of the left and right hand regions are located in the video frame. When the areas of both the left and right hand regions are smaller than the preset area threshold and the state lasts for more than 1 second, it is determined that both hands are completely out of the camera's field of vision. The start timestamp of the time of separation is recorded, and the end timestamp is recorded when the hands reappear. During the period when both hands are out of sight, the gaze coordinates in the gaze coordinate sequence are continuously monitored. If the gaze coordinates are still within the screen display area and the time of separation exceeds 2 seconds, a hand disappearance and gaze deviation coupling label containing the time of separation value and the gaze coordinate subsequence within the time of separation is generated.

7. The method according to claim 6, characterized in that, The step of extracting a screenshot of the foreign object region when a risky foreign object is detected specifically includes: When the tag of hand disappearance and gaze deviation is detected, the coordinate range of the desktop area in the video frame is calculated as the desktop detection sub-image based on the lower edge position of the candidate's face bounding box in the current video frame. The desktop detection sub-image is segmented, and the presence of risky objects is identified. When the confidence level of a risky object exceeds a preset threshold, the smallest bounding rectangle of the area where the risky object is located is captured as a screenshot of the foreign object area, and the category label of the risky object is recorded at the same time.

8. The method according to claim 7, characterized in that, The output of the cheating-related emotional intensity score specifically includes: Extract close-up image sequences of the examinee’s face from video frames within the suspicious time window, and input the screenshot of the foreign object area as auxiliary context into the facial micro-expression verification module; Each frame in the close-up face image sequence is encoded with facial action units, and the intensity value of the frowning action unit is extracted, wherein the frowning action unit corresponds to the amplitude of the action of the eyebrows converging towards the center of the eyebrows; When the intensity value of the frowning action unit is greater than the preset frowning threshold, the output of the cheating-related emotion intensity score is a tense state; otherwise, the output of the calm state is output.

9. The method according to claim 8, characterized in that, The generation of the structured anomaly source tracing report specifically includes: Using the suspicious time window as the time axis reference, the non-answer area gaze period in the gaze coordinate sequence, the hand separation period in the hand disappearance and gaze deviation coupling label, the appearance time of the foreign object area screenshot, and the tension period in the cheating-related emotional intensity score are arranged in chronological order. The time periods are connected in chronological order to form a causal evolution chain, in which the previous time period serves as the triggering premise for the next time period. Each link in the chain is assigned a confidence percentage based on the consistency of upstream and downstream time, and a structured anomaly tracing report is generated.

10. A system for real-time data capture and anomaly labeling in the training and assessment process, characterized in that, The system includes: The video acquisition and baseline modeling module is used to acquire real-time video streams of candidates during the assessment process through cameras and establish a personalized head posture baseline model for each candidate. The deviation detection module is used to calculate the degree of deviation of the examinee's current head posture from the personalized head posture baseline model in real time during the assessment process. When the deviation exceeds the threshold, a suspicious time window is recorded and the camera sampling frame rate is adjusted. The gaze path tracking module is used to track the eye gaze path within the suspected time window and generate a gaze coordinate sequence. The gaze drift detection module is used to synchronously capture the screen display content, map the gaze coordinate sequence onto the screen display content, accumulate the total time the gaze point falls on the non-answer area, and generate a gaze drift trigger signal when the total time exceeds the threshold. The hand detection module is used to perform hand area detection when a gaze drift trigger signal is detected. When the duration of the hands being out of the camera's field of view exceeds a threshold, a coupled label of hand disappearance and gaze deviation is generated. The foreign object detection module is used to detect foreign objects in the desktop area of ​​the video frame when the disappearance of the hand and the deviation of the line of sight are detected. When a risky foreign object is detected, a screenshot of the foreign object area is extracted. The facial expression verification module is used to input the screenshot of the foreign object area and the close-up of the candidate's face within the suspicious time window into the facial micro-expression verification module, and output a score of the intensity of cheating-related emotions. The anomaly tracing report module aggregates the suspicious time window, the gaze coordinate sequence, the coupling label of the disappearance of the hand and the deviation of the gaze, the appearance time of the screenshot of the foreign object area, and the cheating-related emotional intensity score by performing causal chain aggregation to generate a structured anomaly tracing report.

Images