Student state visual analysis system for optimizing virtual human teaching

Abstract

The invention discloses a student state visual analysis system for optimizing virtual human teaching. The student state visual analysis system comprises a front-end sensing layer, a multi-modal feature extraction module, a time sequence analysis layer and a teaching decision visualization layer, the front-end sensing layer collects facial expressions and eye movement data of students in real time, and locates facial areas of the students in real time by adopting a parameter-optimized SSD target detection algorithm; the multi-modal feature extraction module extracts seven types of expressions and four types of eye movement features based on a RepVGG model through a multi-modal fusion method, and constructs a 11-dimensional time series data stream; the time sequence analysis layer dynamically models a cognitive state based on an LSTM network model, and outputs the cognitive state through a continuous behavior rule; the teaching decision visualization layer generates an interactive instrument board, realizes accurate mapping of a cognitive state and a course time axis through a thermodynamic diagram and a trend curve, and automatically marks a high-doubt section. The method breaks through the limitation of a single mode, improves the feedback efficiency, and is suitable for a virtual human asynchronous teaching scene.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and more specifically to a student status visualization and analysis system for optimizing virtual human teaching. Background Technology

[0002] With the development of computer technology, multimedia technology, and network technology, network-based interactive learning platforms and applications are becoming increasingly widespread. Currently, network-based interactive teaching platforms mainly adopt two modes: one is asynchronous on-demand mode, where lecture videos are stored on a server, and students download them to watch and learn independently; the other is real-time live streaming mode, where teachers conduct online lectures with students in real time through a server.

[0003] However, existing solutions have three technical bottlenecks, making it difficult for teachers to obtain effective learning feedback in a timely manner: (1) Limitation of modality: Traditional systems judge the learning state by basic facial expression recognition or eye tracking alone, and cannot capture the coordinated changes of facial expressions and eye movements, such as distinguishing between "thinking frown" and "confused frown", resulting in a high misjudgment rate of cognitive state; (2) Lack of temporal dynamic analysis: Instantaneous feature analysis models ignore the temporal evolution of cognitive state (such as the transformation path of "focus → doubt → distraction"), making it difficult to detect the learning focus and to provide early warning of the learning engagement; (3) Feedback vacuum hinders teaching optimization: In asynchronous teaching, teachers cannot obtain the students' learning state in real time by observing their facial expressions and state, and lack a data-driven content optimization mechanism. Summary of the Invention

[0004] To address the aforementioned technical issues, this technical solution provides a student status visualization analysis system for optimizing virtual human teaching. By combining multimodal fusion, temporal dynamic modeling, and closed-loop visualization decision-making, it can effectively solve the problems in the background technology.

[0005] This invention is achieved through the following technical solution:

[0006] A student status visualization and analysis system for optimizing virtual human teaching includes a front-end perception layer, a multimodal feature extraction module, a time series analysis layer, and a teaching decision visualization layer.

[0007] The aforementioned front-end perception layer collects students' facial expressions and eye movement data in real time, and uses a parameter-optimized SSD target detection algorithm to locate students' facial regions in real time.

[0008] The multimodal feature extraction module extracts 7 types of facial expressions and 4 types of eye-tracking features based on the RepVGG model through a multimodal fusion method, and constructs an 11-dimensional temporal data stream.

[0009] The temporal analysis layer dynamically models cognitive states based on an LSTM network model, converting instantaneous facial expressions and eye movement data into dynamic behavioral features with a time dimension, capturing the duration and dynamic transition patterns of dynamic behavioral features, and recording changes in cognitive states that reflect the degree of learning engagement; and outputting cognitive states through continuous behavioral rules.

[0010] The aforementioned teaching decision visualization layer generates an interactive dashboard that maps the course timeline, using heatmaps and trend curves to achieve a precise mapping between cognitive state and the course timeline, and automatically marks high-confusion sections.

[0011] Furthermore, the front-end perception layer employs a parameter-optimized SSD object detection algorithm to locate the student's facial region in real time. The parameters of the SSD object detection algorithm are configured as follows: the feature layer step size is set to [8, 16, 32, 64, 128, 256] pixels; the aspect ratio of the anchor box is set to multiple ratios such as [1:1, 1:2, 2:1, 1:1.5, 1.5:1]. After detection, redundant detection boxes are filtered out by the Non-Maximum Suppression (NMS) algorithm, and finally, accurate facial region coordinates are output. These coordinates are then cropped and used for subsequent feature extraction to locate the facial region in real time.

[0012] Furthermore, the RepVGG model executes tasks a and b in parallel, and the specific contents of tasks a and b are as follows:

[0013] Task a: Classify the cropped face image with a size of 100×100 pixels into 7 categories of expressions and output a 7-dimensional expression probability vector. The 7 categories of expressions are: happy, panicked, thinking, surprised, frowning, disgusted, and neutral.

[0014] Task b: Extract eye movement features based on the pupil-corneal reflex PCCR algorithm. The eye movement features include: fixation point coordinates (x,y), saccade direction θ, saccade speed v, and blink frequency f.

[0015] Furthermore, the time-series analysis layer receives an 11-dimensional time-series data stream from the front-end perception layer. Using a Long Short-Term Memory (LSTM) network structure, it performs time-series modeling on the input 11-dimensional time-series data stream, learns its dynamic evolution rules, and infers dynamic behavioral patterns that reflect specific cognitive states. The LSTM network has a sampling rate of 2Hz, 64 hidden units, data coverage of 5 seconds of context, and a length T = 10.

[0016] Furthermore, the temporal analysis layer outputs a state sequence or probability distribution representing the level of learning engagement or a specific cognitive state; the definition of the cognitive state includes:

[0017] 1) Focus: The association pattern between thinking expression and stable gaze (displacement < 0.1);

[0018] 2) Question: The co-occurrence pattern of facial expressions such as frowning lasting >3 seconds + blinking >0.8 Hz;

[0019] 3) Analysis: Based on the combined characteristics of a thinking expression and a scanning speed of 30-80px / s;

[0020] 4) Dispersion: Recognizes aversion / neutral expressions and coordinated changes with a scanning speed >100px / s.

[0021] Furthermore, the cognitive state is determined by using a 4-dimensional probability vector to represent the probability that a student is in the following cognitive states:

[0022] 1) Focus:

[0023] Facial expression requirement: A thoughtful expression lasting >0.7 seconds;

[0024] Eye movement requirements: stable fixation, standard deviation of eye movement σ≤0.1;

[0025] The calculation of gaze point displacement defines the gaze point displacement between adjacent time points:

[0026]

[0027] In the above formula, xk and yk are the coordinates of the gaze point at the current time in the normalized coordinate system, ranging from 0 to 1; xk-1 and yk-1 are the coordinates of the gaze point at the previous time; dk is the gaze displacement (Euclidean distance) between adjacent sampling points;

[0028] Mean gaze displacement:

[0029]

[0030] In the above formula, n is the number of sampling points within a 3-second time window, n = sampling rate × 3, the default sampling rate is 2Hz, n = 6; μd is the average gaze displacement, which is the arithmetic mean of all dk within the window;

[0031] The standard deviation of fixation stability is calculated using the following formula: σ is the standard deviation of fixation point displacement within a 3-second window.

[0032]

[0033] In the above formula, σ is the gaze stability index, a standardized value with no unit; n-1 is the Bessel correction coefficient;

[0034] Stability coefficient determination criteria:

[0035] When σ≤0.05, the stability coefficient Gs=1.0; when 0.05<σ≤0.1, the stability coefficient Gs=0.8; when σ>0.1, the stability coefficient Gs=0.

[0036] The formula for focusing probability is: F = P_thinking × G_s;

[0037] In the above formula, P_thinking represents the probability of a thinking expression, which is output by the RepVGG model;

[0038] 2) Question:

[0039] Triggering conditions: The co-occurrence pattern of frowning for more than 3 seconds + blinking for more than 0.8 Hz means that the two must be satisfied within the same 3-second time window;

[0040] Weight calculation:

[0041]

[0042] In the above formula, Wt is the duration weight, and Tf is the effective duration in seconds; when Tf = 3s, Wt = 1.0, and when Tf < 3s, it decays proportionally;

[0043]

[0044] In the above formula, Wb is the blink weight and Bh is the average blink frequency; when Bh = 0.8 Hz, Wb = 1.0, and when Bh < 0.8 Hz, it is attenuated proportionally.

[0045] The formula for the probability of doubt is: C = P_frowning × Wt × Wb;

[0046] 3) Analysis:

[0047] Triggering condition: Based on the combination of a thinking expression probability > 0.6 and a saccade speed of 30-80 px / s;

[0048] Conformity factor formula:

[0049] Parameter definition: In the above formula, v is the scanning speed, in pixels per second (px / s);

[0050] Analysis of the probability formula: A = P_thinking × F_speed;

[0051] 4) Disorganization:

[0052] Triggering conditions: Probability of recognizing aversion / neutral expressions > 0.8 + co-variation with a scanning speed > 100px / s;

[0053] Sagling factor formula:

[0054] Parameter definition: In the above formula, v is the saccadic velocity.

[0055] The formula for the probability of dispersion is: D = (P_averse + P_neutral) * F_scan.

[0056] Furthermore, the heatmap uses the X-axis for course time and the Y-axis for cognitive state, with color depth representing state intensity. It displays the intensity distribution of different cognitive states, such as focus and confusion, over time, and automatically marks high-confusion areas. If a student remains in a confused state for 3 minutes, a high-confusion area is triggered and marked as a high-confusion segment (such as a difficult area in a database principles course). The focus score is calculated by multiplying the focus probability by the duration. The trend curve shows the changing trends of key cognitive state indicators, such as average focus and confusion frequency, over time. The state markers directly mark specific cognitive state events, such as peak confusion areas and inattentiveness starting points, on the course timeline; and automatically mark high-confusion areas lasting more than 3 minutes.

[0057] Teachers can intuitively locate teaching difficulties and identify areas of high confusion by using the dashboard display, such as the dark red sections of a heat map. They can then identify the teaching difficulties and areas where students' attention has significantly decreased. They can then optimize and adjust the courseware content by breaking down knowledge points and adding interactive examples, forming an enhanced closed loop of "analysis-feedback-optimization".

[0058] Furthermore, the trend curve includes a main line for statistical focus score and an auxiliary line for statistical distraction frequency; the focus score is calculated by multiplying the focus probability by the duration; the distraction frequency is calculated as the number of times the eyes become unfocused every 5 minutes.

[0059] Beneficial effects

[0060] The present invention proposes a student status visualization and analysis system for optimizing virtual human teaching, which, compared with existing technologies, has the following advantages:

[0061] (1) This invention improves cognitive state and recognition accuracy through multimodal fusion, overcoming the limitation of single modality; by simultaneously integrating facial expression features (RepVGG classification) and eye movement features (PCCR algorithm), an 11-dimensional temporal data stream is constructed, overcoming the limitation of single-modality cognitive judgment, and accurately identifying similar facial expression states that are difficult to distinguish by traditional methods, such as "thinking frown" and "confused frown".

[0062] (2) This invention uses a dual-layer LSTM network to process 10 steps of time-series data and learn the spatiotemporal dynamic correlation of multimodal features. For example, continuous behavior recognition is based on the analysis of feature combination rules (such as "frowning lasts > 3 seconds + blinking > 0.8 Hz = confused state"); and state transition path tracking to identify typical cognitive evolution paths such as "focus → confusion → distraction", and to explore the focus score, thus making up for the lack of dynamic analysis in the prior art.

[0063] (3) Based on cognitive state mapping data visualization, this invention generates a multi-dimensional dashboard, which includes a heat map that marks high doubt areas with color depth, such as the dark red block in the 15th-20th minute; it uses trend curves to statistically analyze focus scores and distraction frequency; and it automatically marks high doubt areas, forming a teaching enhancement closed loop of "analysis-feedback-optimization", which solves the pain point that teachers cannot obtain real-time feedback in asynchronous teaching.

[0064] (4) This invention applies facial expression recognition-cognitive state mapping technology to the field of education, and provides visualization technology and automated data analysis to assist teaching decisions, further solving the pain point of "feedback vacuum" in asynchronous teaching, and providing a technical foundation for the equalization of educational resources. Attached Figure Description

[0065] Figure 1 This is a schematic diagram illustrating the specific operation steps in the example of this invention.

[0066] Figure 2 This is a schematic diagram of the cognitive state logic discrimination process in this invention.

[0067] Figure 3 This is a schematic diagram of the teaching decision visualization dashboard in this invention. Detailed Implementation

[0068] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. The described embodiments are merely some embodiments of the present invention, and not all embodiments. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the design concept of the present invention should fall within the protection scope of the present invention.

[0069] Example 1:

[0070] A student status visualization and analysis system for optimizing virtual human teaching includes a front-end perception layer, a multimodal feature extraction module, a time series analysis layer, and a teaching decision visualization layer.

[0071] The aforementioned front-end perception layer collects students' facial expressions and eye movement data in real time, and uses a parameter-optimized SSD object detection algorithm to locate the student's facial region in real time. The parameters of the SSD object detection algorithm are configured as follows: the feature layer step size is set to [8, 16, 32, 64, 128, 256] pixels; the aspect ratio of the anchor box is set to multiple ratios such as [1:1, 1:2, 2:1, 1:1.5, 1.5:1]. After detection, redundant detection boxes are filtered out by the non-maximum suppression (NMS) algorithm, and finally the accurate facial region coordinates are output. These coordinates are then cropped for subsequent feature extraction to locate the facial region in real time.

[0072] The multimodal feature extraction module extracts 7 types of facial expressions and 4 types of eye-tracking features based on the RepVGG model using a multimodal fusion method, constructing an 11-dimensional temporal data stream. The RepVGG model executes tasks a and b in parallel. The specific contents of tasks a and b are as follows:

[0073] Task a: Classify the cropped face image with a size of 100×100 pixels into 7 categories of expressions and output a 7-dimensional expression probability vector. The 7 categories of expressions are: happy, panicked, thinking, surprised, frowning, disgusted, and neutral.

[0074] Task b: Extract eye movement features based on the pupil-corneal reflex PCCR algorithm. The eye movement features include: fixation point coordinates (x,y), saccade direction θ, saccade speed v, and blink frequency f.

[0075] The time-series analysis layer receives an 11-dimensional time-series data stream from the front-end perception layer. It utilizes a long short-term memory (LSTM) network structure to dynamically model cognitive states based on the LSTM network model. The LSTM network has a sampling rate of 2Hz, 64 hidden units, and the data covers a 5-second context with a length T=10.

[0076] The system performs time-series modeling on the input 11-dimensional time-series data stream, learns its dynamic evolution rules, and infers dynamic behavioral patterns that reflect specific cognitive states. It converts instantaneous facial expressions and eye movement data into dynamic behavioral features with a time dimension, captures the duration and dynamic transition patterns of dynamic behavioral features, and records changes in cognitive states that reflect the degree of learning engagement. Through continuous behavioral rules, it outputs cognitive states.

[0077] The definition of cognitive state includes:

[0078] 1) Focus: The association pattern between thinking expression and stable gaze (displacement < 0.1);

[0079] 2) Question: The co-occurrence pattern of facial expressions such as frowning lasting >3 seconds + blinking >0.8 Hz;

[0080] 3) Analysis: Based on the combined characteristics of a thinking expression and a scanning speed of 30-80px / s;

[0081] 4) Dispersion: Recognizes aversion / neutral expressions and coordinated changes with a scanning speed >100px / s.

[0082] The temporal analysis layer outputs a state sequence or probability distribution that represents the level of learning engagement or a specific cognitive state.

[0083] The cognitive state is determined by using a 4-dimensional probability vector to represent the probability that a student is in the following cognitive states:

[0084] 1) Focus:

[0085] Facial expression requirement: A thoughtful expression lasting >0.7 seconds;

[0086] Eye movement requirements: stable fixation, standard deviation of eye movement σ≤0.1;

[0087] The calculation of gaze point displacement defines the gaze point displacement between adjacent time points:

[0088]

[0089] In the above formula, xk and yk are the coordinates of the gaze point at the current time in the normalized coordinate system, ranging from 0 to 1; xk-1 and yk-1 are the coordinates of the gaze point at the previous time; dk is the gaze displacement (Euclidean distance) between adjacent sampling points;

[0090] Mean gaze displacement:

[0091]

[0092] In the above formula, n is the number of sampling points within a 3-second time window, n = sampling rate × 3, the default sampling rate is 2Hz, n = 6; μd is the average gaze displacement, which is the arithmetic mean of all dk within the window;

[0093] The standard deviation of fixation stability is calculated using the following formula: σ is the standard deviation of fixation point displacement within a 3-second window.

[0094]

[0095] In the above formula, σ is the gaze stability index, a standardized value with no unit; n-1 is the Bessel correction coefficient;

[0096] Stability coefficient determination criteria:

[0097] When σ≤0.05, the stability coefficient Gs=1.0; when 0.05<σ≤0.1, the stability coefficient Gs=0.8; when σ>0.1, the stability coefficient Gs=0.

[0098] The formula for focusing probability is: F = P_thinking × G_s;

[0099] In the above formula, P_thinking represents the probability of a thinking expression, which is output by the RepVGG model;

[0100] 2) Question:

[0101] Triggering conditions: The co-occurrence pattern of frowning for more than 3 seconds + blinking for more than 0.8 Hz means that the two must be satisfied within the same 3-second time window;

[0102] Weight calculation:

[0103]

[0104] In the above formula, Wt is the duration weight, and Tf is the effective duration in seconds; when Tf = 3s, Wt = 1.0, and when Tf < 3s, it decays proportionally;

[0105]

[0106] In the above formula, Wb is the blink weight and Bh is the average blink frequency; when Bh = 0.8 Hz, Wb = 1.0, and when Bh < 0.8 Hz, it is attenuated proportionally.

[0107] The formula for the probability of doubt is: C = P_frowning × Wt × Wb;

[0108] 3) Analysis:

[0109] Triggering condition: Based on the combination of a thinking expression probability > 0.6 and a saccade speed of 30-80 px / s;

[0110] Conformity factor formula:

[0111] Parameter definition: In the above formula, v is the scanning speed, in pixels per second (px / s);

[0112] Analysis of the probability formula: A = P_thinking × F_speed;

[0113] 4) Disorganization:

[0114] Triggering conditions: Probability of recognizing aversion / neutral expressions > 0.8 + co-variation with a scanning speed > 100px / s;

[0115] Sagling factor formula:

[0116] Parameter definition: In the above formula, v is the saccadic velocity.

[0117] The formula for the probability of dispersion is: D = (P_averse + P_neutral) * F_scan.

[0118] The aforementioned teaching decision visualization layer generates an interactive dashboard that maps the course timeline, using heatmaps and trend curves to achieve a precise mapping between cognitive state and the course timeline, and automatically marks high-confusion sections.

[0119] The heatmap uses the X-axis for course time and the Y-axis for cognitive state, with color depth representing state intensity. It displays the intensity distribution of different cognitive states, such as focus and confusion, over time, and automatically marks high-confusion areas. If a student remains in a confused state for 3 minutes, a high-confusion area is triggered and marked as a difficult area in the database principle. The focus score is calculated by multiplying the focus probability by the duration. The trend curve shows the changing trends of key cognitive state indicators, such as average focus and confusion frequency, over time. The state markers directly mark specific cognitive state events, such as peak confusion areas and initiation points of distraction, on the course timeline; and automatically mark high-confusion areas lasting more than 3 minutes.

[0120] Teachers can intuitively locate teaching difficulties and identify areas of high confusion by using the dashboard display, such as the dark red sections of a heat map. They can then identify the teaching difficulties and areas where students' attention has significantly decreased. They can then optimize and adjust the courseware content by breaking down knowledge points and adding interactive examples, forming an enhanced closed loop of "analysis-feedback-optimization".

[0121] The trend curve includes a main line for statistical focus score and an auxiliary line for statistical distraction frequency; the focus score is calculated by multiplying the focus probability by the duration; the distraction frequency is calculated as the number of times the eyes become unfocused every 5 minutes.

[0122] Case 1:

[0123] This example is applied to a virtual human asynchronous teaching platform (such as a database principles course), where students watch virtual teacher lecture videos via electronic devices. The system uses multimodal cognitive state analysis to help teachers identify teaching difficulties in the "index optimization" section (minutes 10-15). Specific operation methods are as follows... Figure 1 As shown, the steps include:

[0124] Step 1: Multimodal data acquisition: The system uses the camera and eye tracker on the student's learning electronic device to synchronously and in real time capture two types of key data when the student is learning asynchronously: one is a video image stream containing facial expression changes, and the other is high-precision eye movement data.

[0125] Step 2: Front-end perception layer processing. The collected raw data is sent to the front-end perception layer for real-time processing. This layer includes two sub-steps: face detection and multimodal feature extraction.

[0126] Step 2.1: Face detection. The system uses an SSD target detection algorithm with specific optimized parameters to process the input video image (the video image is uniformly processed to 300×300 pixels) to quickly and accurately locate the student's facial region.

[0127] To achieve optimal detection performance, the SSD algorithm is configured with the following parameters: the feature layer stride is set to [8, 16, 32, 64, 128, 256] pixels; the aspect ratio of the anchor box is set to various ratios such as [1:1, 1:2, 2:1, 1:1.5, 1.5:1] to adapt to face detection at different poses and distances.

[0128] After detection, redundant detection boxes are filtered out using the non-maximum suppression (NMS) algorithm (with an IoU threshold of 0.5), and the final output is the accurate face region coordinates (100x100 pixels), which are then cropped for subsequent feature extraction.

[0129] Step 2.2: Feature extraction. This step processes two tasks in parallel:

[0130] 1. Facial Expression Classification: The cropped facial images are input into a lightweight RepVGG neural network model. This pre-trained model efficiently classifies student facial expressions, outputting a probability vector with seven dimensions, corresponding to seven basic facial expressions: "happy," "panicked," "thinking," "surprised," "frowning," "disgusted," and "neutral." The model's inference speed, after optimization, can reach less than or equal to 10 milliseconds per frame, meeting the requirements for real-time analysis.

[0131] 2. Eye Movement Feature Extraction: The system uses the pupil-corneal reflection (PCCR) algorithm to analyze the data collected by the infrared eye tracker. This algorithm outputs a 4-dimensional eye movement feature vector, specifically including: fixation point (x, y) coordinates, saccade direction θ, saccade velocity v, and blink frequency f.

[0132] The facial and eye movement features described above together describe the distribution and dynamics of students' visual attention when viewing instructional content.

[0133] Step 3: Temporal Analysis and Cognitive State Inference. The instantaneous features extracted by the front-end perception layer are integrated and deeply analyzed in this module to infer the student's dynamic cognitive state; the specific operation method is as follows... Figure 2 As shown, the steps include:

[0134] Step 3.1: Multimodal data fusion:

[0135] The system aligns and concatenates 7-dimensional facial expression classification probability vectors with 5-dimensional eye-tracking feature vectors according to precise timestamps, thereby constructing an 11-dimensional fused temporal data stream. This data stream is continuously generated at a specific sampling rate (e.g., 2Hz), and each data point comprehensively reflects the student's combined facial expression and eye-tracking performance at a specific moment.

[0136] Step 3.2: LSTM Dynamic Modeling:

[0137] Input data into a two-layer LSTM network (hidden unit 64):

[0138] (1) Determining the state of focus:

[0139] Triggering conditions: Thinking expression > 0.7s + stable gaze (σ≤0.1) associated pattern;

[0140] Example: The probability of thinking is >0.75 (integrated over 0.7s) and the standard deviation of gaze displacement σ = 0.03;

[0141] Analysis of probability: When σ = 0.03 ≤ 0.05, then the stability coefficient Gs = 1.0;

[0142] The formula for the probability of focus is: F = P_thinking × Gs = 0.75 × Gs = 0.65 × 1.0 = 0.65;

[0143] Example output: The focus probability F is 0.75.

[0144] (2) Determining the state of confusion:

[0145] Triggering conditions: A co-occurrence pattern of frowning lasting >3 seconds + blinking >0.8Hz;

[0146] Example: If the frowning duration Tf = 2.1s (3-second average), and the blinking frequency is 0.9Hz (3-second window);

[0147] Analysis of probability: Although the duration of 2.1s does not exceed 3s, it meets the continuous characteristic. The blinking frequency is 0.9Hz, so the blinking frequency is >0.8Hz; the condition for triggering the doubt judgment (based on a 3-second window);

[0148] Then weight

[0149]

[0150] Blinking frequency:

[0151] If the RepVGG model outputs a frowning expression probability P_frowning = 0.7,

[0152] The probability formula for doubt is: C = P_frowning × W_t × W_b = 0.7 × 0.7 × 1.0 = 0.49;

[0153] Example output: The probability of confusion, C, is 0.49.

[0154] (3) Analysis of status determination:

[0155] Triggering conditions: Thinking expression > 0.6 and scanning speed 30-80px / s;

[0156] Example: Given that the probability of P thinking is 0.6 (average over 2 seconds) and the scanning speed v = 45px / s (belonging to the interval 30≤v<55), then;

[0157] Analyze the probability:

[0158] If the condition for triggering the analysis state determination is met, then the speed compliance factor is set to 0.6;

[0159] Analysis of the activity probability formula: A = P_probability of thinking × F_speed = 0.6 × 0.6 = 0.36;

[0160] Example output: The probability of activity A is 0.36.

[0161] (4) Determination of disorganized state:

[0162] Triggering conditions: Probability of combined aversion / neutrality > 0.9 (3-second average) and scanning speed v > 120px / s;

[0163] Example: Probability of dislike + neutrality is 0.9, scanning speed v = 120px / s;

[0164] Analysis of probabilities: The probability of being averse to + neutral is 0.9, which is greater than 0.8; the scanning speed v is 120px / s, which is greater than 100px / s; the condition for determining a scattered state has been triggered; when v = 120px / s,

[0165] Sagling Factor

[0166] The formula for the probability of dispersion is: D = (P_averse + P_neutral)^(total) × F_scan = 0.9 × 0.4 = 0.36;

[0167] Example output: The dispersion probability D is 0.36.

[0168] Step 3.3: State Decision and Storage:

[0169] For the 4-dimensional probability vector output by the LSTM network, the system adopts a strategy of taking the highest probability value, using the cognitive state with the highest probability as the student's final cognitive label at that moment. For example, if the output is [Focus: 0.75, Doubt: 0.49, Analysis: 0.36, Distraction: 0.36], then the current state is determined to be "Focus". This cognitive label, along with information such as timestamp, student ID, and course ID, is recorded and stored in the database at 0.5-second intervals, forming a complete student learning state sequence file.

[0170] Step 4: Generation of a Visual Dashboard for Instructional Decisions:

[0171] This step aims to present the cognitive state sequence data stored in the database to teachers or teaching administrators in an intuitive and multi-dimensional way.

[0172] Step 4.1: Cognitive State-Course Timeline Mapping. The system loads the sequence of students' cognitive states for a specified course from the database and precisely aligns it with the timeline of the teaching video to ensure that the cognitive state at each moment corresponds to a specific teaching content segment.

[0173] Step 4.2: Multi-dimensional visualization presentation. The system generates an interactive dashboard containing various visualization charts, such as... Figure 3 As shown:

[0174] Heatmap: With course time as the X-axis and cognitive state (such as focus, doubt, etc.) as the Y-axis, the intensity or concentration of a certain state within a specific time period is represented by the shade of color (e.g., shades of red). For example, a dark red "doubt" block (intensity 0.65) in the 10-15 minute interval visually indicates to the teacher that there may be teaching difficulties at this point.

[0175] Trend Curve: Plot the curves of key indicators over time. The main line can be "Attention Score" (probability of focus × ​​duration), and the secondary line can be "Distraction Frequency" (e.g., the distraction frequency reaches a peak of 2 times / minute at 10:22), helping teachers to grasp the overall fluctuation pattern of students' attention.

[0176] Event labeling: The system automatically marks key cognitive state events on the timeline, such as "high confusion zones" that occur continuously for a long time or "attention inflection points" where students' attention levels change significantly, so that teachers can quickly locate problems.

[0177] Step 4.3: Teacher-side Interactive Functions. The dashboard provides interactive functions, allowing teachers to filter specific time intervals for analysis (such as locating a segment explaining a particular knowledge point). Teachers can also compare changes in the cognitive states of different students within the same course view, enabling personalized analysis. For example, teachers can use the dashboard's filtering function to view only the teaching video replays for the "high-confusion zone" (3:15-5:30), and can also access comparison views of three other students taking the same course, discovering that these three students also showed varying degrees of confusion in this area. Based on this, the teacher determines that the "Database Principles and Applications" section needs to be redesigned, for example, by adding more basic pre-existing knowledge or supplementing the explanation with a more engaging animation.

[0178] Step 5: The final goal of this system is to create a closed-loop teaching optimization system. Based on data insights revealed by the visualization dashboard, such as identifying common teaching difficulties or areas where students are easily distracted, teachers or curriculum administrators can implement targeted optimization measures. These measures may include breaking down difficult knowledge points, adding supplementary case studies, or introducing interactive Q&A sessions in areas where students' attention is easily diverted. Through this closed loop of "analysis-feedback-optimization," the quality and effectiveness of asynchronous virtual human teaching are continuously improved.

[0179] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered by the present invention.

Claims

1. A student status visualization and analysis system for optimizing virtual human teaching, characterized in that: It includes a front-end perception layer, a multimodal feature extraction module, a time series analysis layer, and a teaching decision visualization layer; The aforementioned front-end perception layer collects students' facial expressions and eye movement data in real time, and uses a parameter-optimized SSD target detection algorithm to locate students' facial regions in real time. The multimodal feature extraction module extracts 7 types of facial expressions and 4 types of eye-tracking features based on the RepVGG model through a multimodal fusion method, and constructs an 11-dimensional temporal data stream. The temporal analysis layer dynamically models cognitive states based on an LSTM network model, converting instantaneous facial expressions and eye movement data into dynamic behavioral features with a time dimension, capturing the duration and dynamic transition patterns of dynamic behavioral features, and recording changes in cognitive states that reflect the degree of learning engagement; and outputting cognitive states through continuous behavioral rules. The aforementioned teaching decision visualization layer generates an interactive dashboard that maps the course timeline, using heatmaps and trend curves to achieve a precise mapping between cognitive state and the course timeline, and automatically marks high-confusion sections.

2. The student status visualization and analysis system for optimizing virtual human teaching according to claim 1, characterized in that: The aforementioned front-end perception layer employs a parameter-optimized SSD object detection algorithm to locate the student's facial region in real time. The parameters of the SSD object detection algorithm are configured as follows: the feature layer stride is set to [8, 16, 32, 64, 128, 256] pixels; the aspect ratio of the anchor box is set to multiple ratios such as [1:1, 1:2, 2:1, 1:1.5, 1.5:1]. After detection, redundant detection boxes are filtered out using the Non-Maximum Suppression (NMS) algorithm, and finally, accurate facial region coordinates are output. These coordinates are then cropped for subsequent feature extraction to locate the facial region in real time.

3. The student status visualization and analysis system for optimizing virtual human teaching according to claim 1, characterized in that: The RepVGG model described above executes tasks a and b in parallel. The specific contents of tasks a and b are as follows: Task a: Classify the cropped face image with a size of 100×100 pixels into 7 categories of expressions and output a 7-dimensional expression probability vector. The 7 categories of expressions are: happy, panicked, thinking, surprised, frowning, disgusted, and neutral. Task b: Extract eye movement features based on the pupil-corneal reflex PCCR algorithm. The eye movement features include: fixation point coordinates (x,y), saccade direction θ, saccade speed v, and blink frequency f.

4. The student status visualization and analysis system for optimizing virtual human teaching according to claim 1, characterized in that: The temporal analysis layer receives an 11-dimensional temporal data stream from the front-end perception layer, consisting of a 7-dimensional expression probability vector and a 4-dimensional eye-tracking feature vector. Using a Long Short-Term Memory (LSTM) network structure, it performs temporal modeling on the input 11-dimensional temporal data stream, learns its dynamic evolution rules, and infers dynamic behavioral patterns reflecting specific cognitive states. The LSTM network has a sampling rate of 2Hz, 64 hidden units, and data coverage of 5 seconds of context, with a length T=10.

5. The student status visualization and analysis system for optimizing virtual human teaching according to claim 1, characterized in that: The temporal analysis layer outputs a state sequence or probability distribution representing the level of learning engagement or a specific cognitive state; the definition of the cognitive state includes: 1) Focus: The association pattern between thinking expression and stable gaze (displacement < 0.1); 2) Question: The co-occurrence pattern of facial expressions such as frowning lasting >3 seconds + blinking >0.8 Hz; 3) Analysis: Based on the combined characteristics of a thinking expression and a scanning speed of 30-80px / s; 4) Dispersion: Recognizes aversion / neutral expressions and coordinated changes with a scanning speed >100px / s.

6. The student status visualization analysis system for optimizing virtual human teaching according to claim 5, characterized in that: The cognitive state is determined by using a 4-dimensional probability vector to represent the probability that a student is in the following cognitive states: 1) Focus: Facial expression requirement: A thoughtful expression lasting >0.7 seconds; Eye movement requirements: stable fixation, standard deviation of eye movement σ≤0.1; The calculation of gaze point displacement defines the gaze point displacement between adjacent time points: In the above formula, xk and yk are the coordinates of the gaze point at the current time in the normalized coordinate system, ranging from 0 to 1; xk-1 and yk-1 are the coordinates of the gaze point at the previous time; and dk is the gaze displacement between adjacent sampling points. Mean gaze displacement: In the above formula, n is the number of sampling points within a 3-second time window, n = sampling rate × 3, the default sampling rate is 2Hz, n = 6; μd is the average gaze displacement, which is the arithmetic mean of all dk within the window; The standard deviation of fixation stability is calculated using the following formula: σ is the standard deviation of fixation point displacement within a 3-second window. In the above formula, σ is the gaze stability index, a standardized value, without units; n-1 is the Bessel correction coefficient; Stability coefficient determination criteria: When σ≤0.05, the stability coefficient Gs=1.0; when 0.05<σ≤0.1, the stability coefficient Gs=0.8; when σ>0.1, the stability coefficient Gs=0. The formula for focusing probability is: F = P_thinking × G_s; In the above formula, P_thinking represents the probability of a thinking expression, which is output by the RepVGG model; 2) Question: Triggering conditions: The co-occurrence pattern of frowning for more than 3 seconds + blinking for more than 0.8 Hz means that the two must be satisfied within the same 3-second time window; Weight calculation: In the above formula, Wt is the duration weight, and Tf is the effective duration in seconds; when Tf = 3s, Wt = 1.0, and when Tf < 3s, it decays proportionally; In the above formula, Wb is the blink weight and Bh is the average blink frequency; when Bh = 0.8 Hz, Wb = 1.0, and when Bh < 0.8 Hz, it is attenuated proportionally. The formula for the probability of doubt is: C = P_frowning × Wt × Wb; 3) Analysis: Triggering condition: Based on the combination of a thinking expression probability > 0.6 and a saccade speed of 30-80 px / s; Conformity factor formula: Parameter definition: In the above formula, v is the scanning speed, in pixels per second (px / s); Analysis of the probability formula: A = P_thinking × F_speed; 4) Disorganization: Triggering conditions: Probability of recognizing aversion / neutral expressions > 0.8 + co-variation with a scanning speed > 100px / s; Sagling factor formula: Parameter definition: In the above formula, v is the saccadic velocity. The formula for the probability of dispersion is: D = (P_averse + P_neutral) * F_scan.

7. The student status visualization analysis system for optimizing virtual human teaching according to claim 1, characterized in that: The heatmap uses the X-axis for course time and the Y-axis for cognitive state, with color depth representing state intensity. It displays the intensity distribution of different cognitive states, such as focus and confusion, over time, and automatically marks high-confusion areas. If a student remains in a confused state for 3 minutes, a high-confusion area is triggered and marked as a difficult area in the database principle. The focus score is calculated by multiplying the focus probability by the duration. The trend curve shows the changing trends of key cognitive state indicators, such as average focus and confusion frequency, over time. The state markers directly mark specific cognitive state events, such as peak confusion areas and initiation points of distraction, on the course timeline. Automatically label areas of high suspicion that persist for more than 3 minutes; Teachers can intuitively locate teaching difficulties and identify areas of high confusion by using the dashboard display, such as the dark red sections of a heat map. They can then identify the teaching difficulties and areas where students' attention has significantly decreased. They can then optimize and adjust the courseware content by breaking down knowledge points and adding interactive examples, forming an enhanced closed loop of "analysis-feedback-optimization".

8. A student status visualization and analysis system for optimizing virtual human teaching according to claim 1 or 7, characterized in that: The trend curve includes a main line for statistical focus score and an auxiliary line for statistical distraction frequency; the focus score is calculated by multiplying the focus probability by the duration; the distraction frequency is calculated as the number of times the eyes become unfocused every 5 minutes.

Images