Online learning engagement real-time intervention method based on multi-modal data fusion

Abstract

This invention discloses a real-time intervention method for online learning engagement based on multimodal data fusion, relating to the fields of big data processing and online learning data processing. The method acquires visual data streams and behavioral interaction data streams during the online learning process, extracting visual feature sequences and behavioral feature sequences separately through a sliding window dual-stream temporal alignment mechanism. These sequences are then input into a cross-modal temporal attention network, calculating the attention weight distribution of the visual feature sequences to the behavioral events using the timestamps of the behavioral events as anchor points. The entropy value of this distribution is calculated; if the entropy value is lower than the dispersion threshold, feature alignment is deemed invalid and the current window data is discarded; otherwise, the weighted fused features are input into a dynamic engagement evaluation model. This invention eliminates the temporal misalignment between visual and behavioral data, reduces feature alignment errors during asynchronous data fusion, and avoids triggering invalid intervention commands due to data latency.

Description

Technical Field

[0001] This invention relates to the field of big data processing and online learning data processing technology, and discloses a method for real-time intervention of online learning engagement based on multimodal data fusion. Background Technology

[0002] In the field of online learning engagement recognition, existing conventional technical solutions typically employ a multimodal data fusion processing architecture. In practice, such solutions capture video images of learners via cameras and record behavioral logs such as mouse clicks and page dwell times via web scripts. During the data fusion phase, the system forcibly concatenates facial features extracted from the video images with operational features extracted from the behavioral logs at a unified timestamp, generating a joint feature vector. This joint feature vector is then input into a pre-trained classification network, which outputs a classification result of the learner's engagement state. Based on the classification result, corresponding intervention instructions are matched from a policy library and sent to the teaching terminal.

[0003] The core flaw in the aforementioned existing technology lies in the inherent difference between the acquisition frequency of video images and the generation frequency of behavior logs. Video images belong to a high-sampling-rate data stream, while behavior logs belong to a low-sampling-rate data stream. Forcibly stitching the data together using a uniform timestamp introduces a large number of redundant features from the high-sampling-rate visual data—features without corresponding behavioral events—into the joint feature vector, leading to feature mismatch in the temporal dimension. This temporal misalignment causes the feature vector received by the classification network to deviate from the true engagement state distribution, consequently causing the intervention instructions output by the system to deviate from the learner's actual state. Summary of the Invention

[0004] The purpose of this invention is to provide a solution to the problems described in the background section.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Real-time intervention methods for online learning engagement based on multimodal data fusion include: The visual data stream and behavioral interaction data stream during the online learning process are acquired, and the visual feature sequence and behavioral feature sequence are extracted respectively through a sliding window dual-stream temporal alignment mechanism. The visual feature sequence and the behavioral feature sequence are input into a cross-modal temporal attention network. Using the timestamps of behavioral events in the behavioral feature sequence as anchors, the attention weight distribution of the visual feature sequence to the behavioral events is calculated within the sliding window. Calculate the entropy value of the attention weight distribution. When the entropy value is lower than the preset dispersion threshold, it is determined that the feature alignment has failed and the current sliding window data is discarded. Otherwise, the weighted fused features are input into the dynamic engagement evaluation model. The engagement state vector output by the dynamic engagement assessment model is hashed and mapped with the instruction tags in the intervention strategy library to generate an intervention instruction stream to drive the teaching system to execute feedback.

[0006] Preferably, the extraction of visual feature sequences via the sliding window dual-stream temporal alignment mechanism includes: The visual data stream is downsampled in the temporal dimension to obtain a sparse video frame sequence; The sparse video frame sequence is used to locate facial regions, and the eye movement trajectory features of the facial regions are extracted using optical flow. The facial micro-expression features are then extracted using a facial key point tracking algorithm. The visual feature sequence is generated by arranging the eye movement trajectory features and the facial micro-expression change features in the chronological order of the sparse video frame sequence.

[0007] Preferably, the extraction of behavioral feature sequences via the sliding window dual-stream temporal alignment mechanism includes: Parse the document object model tree of the online learning platform, and listen for and capture interactive events in the behavioral interaction data stream; Extract the frequency characteristics of mouse click coordinate changes and the duration of page dwell on the current webpage element from the interactive events; Based on the occurrence sequence of the interaction events, the frequency feature of mouse click coordinate changes and the feature of page dwell time are serialized and concatenated to generate the behavior feature sequence.

[0008] Preferably, using the timestamps of behavioral events in the behavioral feature sequence as anchors includes: Identify the state transition event in the behavioral feature sequence, and set the timestamp of the transition event as the anchor point; Centered on the anchor point, the left and right boundaries of the sliding window are dynamically set according to the sampling frequency ratio of the visual data stream and the behavioral interaction data stream, so that the sliding window contains at least one complete visual feature extraction cycle.

[0009] Preferably, calculating the entropy value of the attention weight distribution includes: The attention weight values ​​assigned to each visual feature node within the sliding window are statistically analyzed to construct an attention probability distribution matrix; The entropy value is obtained by performing a logarithmic operation on the attention probability distribution matrix and summing the results. When feature alignment fails, the timestamp corresponding to the anchor point at the time of failure is recorded, and the sliding window is reinitialized starting from the timestamp to reset the hidden state vector of the cross-modal temporal attention network.

[0010] Preferably, the step of hashing the engagement state vector with the instruction tags in the intervention strategy library includes: The engagement state vector is binarized and quantized to generate a state hash code; Calculate the Hamming distance between the state hash code and the tag hash codes corresponding to each instruction tag pre-stored in the intervention strategy library; Select the target label hash code with the smallest Hamming distance, and encapsulate the intervention action corresponding to the target label hash code into the intervention instruction stream.

[0011] Preferably, the downsampling process of the visual data stream in the time dimension includes: The visual data stream is received at the edge computing node deployed on the student terminal; In the edge computing node, the frame extraction ratio of the downsampling process is adaptively adjusted according to the current inference stage of the dynamic input evaluation model; When the dynamic engagement evaluation model is in the feature extraction stage, a first frame extraction ratio is used. When the dynamic engagement evaluation model is in the state classification stage, it is switched to a second frame extraction ratio that is lower than the first frame extraction ratio.

[0012] Preferably, dynamically setting the left and right boundaries of the sliding window includes: Retrieve the current student's historical behavior and interaction logs, and calculate the historical average time interval between adjacent transition events; The product of the historical average time interval and the ratio of the sampling frequency is used as the base window length; Obtain the variance of the behavioral feature sequence within the basic window length. When the variance is greater than a preset variance threshold, shorten the basic window length to generate a shortened window, and use the boundary of the shortened window as the left boundary and the right boundary.

[0013] Preferably, resetting the hidden state vector of the cross-modal temporal attention network includes: Extract the final hidden state vector corresponding to the most recent valid sliding window that was not discarded before the reset; A decay factor is applied to the final hidden state vector to generate a compensated hidden state vector; The compensated hidden state vector is used as the initial input to the re-initialized cross-modal temporal attention network, and the compensated hidden state vector is used to perform forward inference on the dynamic engagement evaluation model within the reset window.

[0014] Preferably, selecting the target label hash code with the smallest Hamming distance includes: When there are at least two candidate label hash codes with equal Hamming distance and both being minimum, extract the model confidence that accompanies the output of the engagement state vector by the dynamic engagement evaluation model. Retrieve the historical intervention success rates associated with the candidate label hash codes in the intervention strategy library; The model confidence score and the historical intervention success rate are weighted and summed, and the candidate label hash code with the largest weighted sum is selected as the target label hash code.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention establishes a sliding window dual-stream temporal alignment mechanism. It calculates the attention weight distribution of visual feature sequences using the timestamps of behavioral events as anchor points, and determines the effectiveness of feature alignment by comparing the entropy value of the attention weight distribution with a dispersion threshold. This technique establishes an event-related weight mapping between high-sampling-rate visual data streams and low-sampling-rate behavioral data streams in the temporal dimension. Entropy values ​​are used to discard window data with temporal misalignment, preventing redundant visual features from entering the subsequent evaluation model. This eliminates the temporal feature mismatch caused by forced stitching, ensuring the consistency between the feature vector input to the dynamic engagement evaluation model and the actual engagement state distribution.

[0016] 2. By setting transition events as anchor points and dynamically setting window boundaries based on the sampling frequency ratio, the coverage of the sliding window is matched with the inherent frequency of the data; by applying a decay factor to the final hidden state vector to generate a compensating hidden state vector, the loss of temporal context information during the window reset process is avoided; in the hash mapping stage, the model confidence and historical intervention success rate are weighted and summed to handle cases where the distance is equal, reducing the probability of mismatched intervention instructions caused by hash collisions. Attached Figure Description

[0017] Figure 1 This is the main flowchart of the real-time intervention method for online learning engagement based on multimodal data fusion according to the present invention; Figure 2 This is a flowchart of the visual feature sequence extraction process of the present invention; Figure 3 This is a flowchart of the behavioral feature sequence extraction process of the present invention; Figure 4 This is a flowchart illustrating the dynamic setting of the sliding window in this invention. Figure 5 This is a flowchart of the feature alignment failure determination and window reset process of the present invention; Figure 6 This is a flowchart of the hash mapping and intervention instruction generation process of the present invention. Detailed Implementation

[0018] Please refer to Figure 1 This embodiment provides a real-time intervention method for online learning engagement based on multimodal data fusion. The online learning system is deployed in an architecture consisting of a student terminal and a cloud server. The student terminal is equipped with a high-definition camera and a web-based learning platform. The system acquires visual data streams and behavioral interaction data streams during the online learning process. The visual data stream is collected by the high-definition camera at a sampling frequency of 30 frames per second, which is a high sampling rate data stream. The behavioral interaction data stream is captured by the web-based learning platform through a script program, and its generation depends on the student's active operation, which is a low sampling rate data stream. After acquiring the above two types of data streams, the system extracts visual feature sequences and behavioral feature sequences respectively through a sliding window dual-stream temporal alignment mechanism. The core of this mechanism is to abandon the method of forced splicing according to a uniform timestamp in the existing technology, and instead establish a temporal mapping relationship based on event association. Specifically, the system inputs the visual feature sequences and behavioral feature sequences into a cross-modal temporal attention network. In the cross-modal temporal attention network, the attention weight distribution of the visual feature sequences to behavioral events is calculated within a sliding window, using the timestamp of the behavioral event in the behavioral feature sequence as the anchor point. Through this attention weight distribution, the system can quantify the true correlation between each feature node in high-sampling-rate visual data and low-sampling-rate behavioral events. Subsequently, the system calculates the entropy value of the attention weight distribution. When the entropy value is lower than a preset dispersion threshold, it indicates that the visual feature attention is highly concentrated on a few isolated nodes, resulting in temporal misalignment. In this case, feature alignment is deemed invalid, and the current sliding window data is discarded. Otherwise, the weighted fused features are input into the dynamic engagement assessment model. Finally, the engagement state vector output by the dynamic engagement assessment model is hashed and mapped with the instruction labels in the intervention strategy library to generate an intervention instruction stream to drive the teaching system to execute feedback.

[0019] In one embodiment, reference Figure 2 The specific process of extracting visual feature sequences using a sliding window dual-stream temporal alignment mechanism includes downsampling the visual data stream in the temporal dimension to obtain a sparse video frame sequence. After the edge computing node deployed on the student terminal receives the visual data stream, it adaptively adjusts the frame extraction ratio of the downsampling process based on the current inference stage of the dynamic engagement evaluation model. Specifically, the edge computing node maintains a double-buffered queue in its memory. When the dynamic engagement evaluation model is in the feature extraction stage, the model requires dense temporal information to capture changes in facial micro-expressions. At this time, a first frame extraction ratio is used, for example, extracting one frame every two frames. The first frame extraction ratio is set to... When the dynamic engagement assessment model is in the state classification stage, the model mainly relies on macroscopic features for decision-making. At this time, it switches to a second frame extraction ratio lower than the first frame extraction ratio, for example, extracting one frame every five frames, and setting the second frame extraction ratio to [missing information]. ,in By dynamically adjusting the frame extraction ratio, the computational load on edge computing nodes is reduced while ensuring feature extraction accuracy. This applies to the extracted sparse video frame sequences. The system performs facial region localization. Facial region localization employs a multi-task cascaded convolutional network, which consists of three cascaded network stages that output candidate bounding boxes, calibration boxes, and facial keypoint coordinates, respectively. This process obtains the bounding boxes of the facial regions. Subsequently, the system utilizes optical flow to extract eye movement trajectory features in the facial region. Optical flow is based on the assumptions of constant grayscale and neighborhood consistency, and is applicable to two adjacent frames in a sparse video frame sequence. and pixel coordinates in the facial region At this point, construct the optical flow constraint equation: ; in, This represents the grayscale gradient of the image in the horizontal direction. This represents the grayscale gradient of the image in the vertical direction. The gray-level gradient of the image over time. and These represent the optical flow velocity components of a pixel in the horizontal and vertical directions, respectively. To solve this overdetermined system of equations, the system employs the Lucas-Kanade algorithm, at the pixel level... local neighborhood Internally establish the weighted least squares objective function: ; in, This is a Gaussian window weighting function used to assign higher weights to the center pixel. By solving a system of equations where the partial derivatives of this objective function are zero, the eye movement trajectory feature vector is obtained. Simultaneously, the system incorporates a facial landmark tracking algorithm to extract facial micro-expression variation features. This algorithm employs a cascaded regression tree structure, taking a grayscale image patch of the facial region as input and outputting a set of two-dimensional coordinates for 68 pre-defined facial landmarks. To eliminate interference from rigid head movements, the system calculates the relative displacement of facial key points between adjacent frames. A subset of key points is selected from non-rigid regions such as the brow bone, corners of the eyes, and corners of the mouth. Calculate the deformation vectors of adjacent frames: ; The deformation vector This refers to facial micro-expression changes. The system incorporates eye movement trajectory features. Facial micro-expression changes Generate visual feature sequences by arranging the sparse video frame sequences in temporal order. ,in For the first The feature concatenation vector of the frame.

[0020] In one embodiment, reference Figure 3 The specific process of extracting behavioral feature sequences through a sliding window dual-stream timing alignment mechanism includes parsing the Document Object Model (DOM) tree of the online learning platform. The DOM tree is a tree structure composed of webpage nodes, each containing a tag name, attribute set, and event listeners. The system injects a script into the webpage and binds event listeners to all interactive nodes (such as buttons, hyperlinks, input boxes, and scrollbars) using a depth-first search algorithm by traversing the root node of the DOM tree. When listening for and capturing interactive events in the behavioral interaction data stream, the system uses an event delegation mechanism to bind event listeners to the top-level node of the DOM tree, capturing interactive events triggered by child nodes using the event bubbling phase. For the captured interactive events, the system extracts the frequency characteristics of mouse click coordinate changes and the page dwell time characteristics of the current webpage element. Specifically, the system maintains a time slice in memory with a length of... A circular buffer is used to store the most recently occurred mouse coordinate sequence. Calculate the first-order difference between adjacent coordinate points within this time slice: ; Set a displacement threshold ,when The displacement is determined to be valid upon clicking. Mouse click coordinate change frequency characteristics. The calculation formula is: ; in, This is an indicator function that takes the value when the condition is true. Otherwise, the value is For the duration of focus on the current webpage element, the system records the currently focused webpage element node in the Document Object Model (DOM) tree and starts a high-precision timer. When focus shifts, i.e., when a focus loss event is triggered, the focus loss timestamp is recorded. With focus timestamp The difference, page dwell time characteristics The calculation formula is: ; The system analyzes the frequency of mouse click coordinate changes based on the sequence of interactive events. Features of page dwell time Perform serialization and concatenation. During the concatenation process, if at a certain moment only mouse movement is triggered without focus shift, then the corresponding... The field is populated with the value of the previous valid dwell time; if only focus shift occurred without triggering mouse movement, then the corresponding... The field is filled with 0. This generates a behavioral feature sequence. .

[0021] In one embodiment, reference Figure 4 The specific implementation method using the timestamps of behavioral events in a behavioral feature sequence as anchors includes identifying transition events where states change abruptly within the behavioral feature sequence. In the behavioral feature sequence, each feature vector... It includes frequency and duration dimensions. The system calculates the Euclidean distance between adjacent feature vectors as the state change quantity. ; Set a transition threshold ,when At that time, the judgment of the first Each behavioral event is a state transition event. The system sets the occurrence time of this transition event as the anchor timestamp. After determining the anchor point, the system dynamically sets the left and right boundaries of the sliding window based on the ratio of the sampling frequencies of the visual data stream and the behavioral interaction data stream, using that anchor point as the center. The sampling frequency of the visual data stream is... The sampling frequency of the behavioral interaction data stream is The sampling frequency ratio is The system retrieves the current student's historical interaction logs and calculates the historical average time interval between adjacent transition events within a predetermined time period. The base window length is the product of the historical average time interval and the sampling frequency. : ; The base window length is displayed on the timeline as... Extending outwards from the center, the sliding window ensures that it covers the complete visual change cycle corresponding to a single behavioral event in the time dimension, thus containing at least one complete visual feature extraction cycle within the sliding window. After calculating the basic window length, the system obtains the feature variance of the behavioral feature sequence within that basic window length. All behavioral feature vectors within this window are then extracted. Calculate its mathematical expectation Then calculate the variance. : ; in, This refers to the number of behavioral feature vectors contained within the base window length. A preset variance threshold is set. ,when This indicates that the student's behavior is in a high-frequency, unstable state during that time period. If an excessively long base window is used in this case, a large amount of irrelevant, noisy visual features will be introduced into subsequent calculations. Therefore, the system shortens the base window length to generate a shortened window. Shortening factor It is inversely proportional to the variance: ; in, This is the lower limit of the minimum shortening factor to prevent information loss due to excessively narrow windows. Shorten the window length. Ultimately, the shortened window's boundary was used as the left boundary. With right boundary : , ; In one embodiment, after the sliding window is determined, the system inputs the visual feature sequence and the behavioral feature sequence into a cross-modal temporal attention network. The cross-modal temporal attention network consists of a multi-head self-attention layer and a feedforward neural network layer. In the multi-head self-attention layer, the system uses the behavioral feature sequence as the query input and the visual feature sequence as the key and value inputs. Specifically, for the behavioral feature matrix within the sliding window... With visual feature matrix ,in The length of the behavioral feature sequence. The length of the visual feature sequence. and These are the feature dimensions. A linear mapping layer transforms the behavioral feature matrix into a query matrix. Convert the visual feature matrix into a key matrix AND-value matrix .in, , , The weight parameter matrix is ​​a learnable matrix. This represents the dimension of the key vector. When calculating the attention weight distribution, the system uses the timestamp of the behavior event as the anchor point and only calculates the behavior feature vector corresponding to the anchor point. Attention score among all visual feature nodes within the sliding window: ; in, Key matrix The first in There are 10 visual feature vectors. The attention scores are then transformed into attention weight distributions using the Softmax activation function. ; The attention weight distribution It reflects the contribution or correlation of each visual feature node to the current behavioral event.

[0022] In one embodiment, reference Figure 5 The system calculates the entropy value of the attention weight distribution to determine the effectiveness of feature alignment. The attention weight values ​​assigned to each visual feature node within the sliding window are statistically analyzed, and the aforementioned attention weight distribution is obtained. This is the attention probability distribution matrix. Performing a logarithmic operation on this matrix and summing the results yields the entropy value. Entropy is calculated using the Shannon entropy formula: ; In this formula, Indicates the first The attention weight probability is assigned to each visual feature node. When there is a severe temporal misalignment between the high-sampling-rate visual data stream and the low-sampling-rate behavioral data stream, the true visual features corresponding to the behavioral event do not appear within the current sliding window. This causes the attention mechanism to fail to find effective associated targets, and the model's attention degenerates and concentrates on a few noisy visual nodes with extreme gradient responses within the window. At this time, the attention probability distribution matrix exhibits an extremely sharp shape, meaning that only a few... The value is close to The rest of the large amount The value is close to This leads to the calculated entropy value Extremely small. The system has a preset dispersion threshold. This threshold is determined by statistically analyzing the distribution of minimum entropy values ​​when temporal misalignment occurs on an offline validation set. When the calculated entropy value... Below the preset dispersion threshold When this occurs, the system determines that feature alignment has failed. At this point, the system discards the current sliding window data to prevent redundant and misaligned visual features from entering subsequent models. Simultaneously, the system records the timestamp corresponding to the anchor point at the time of failure. and with that timestamp Reinitialize the sliding window as the starting point and reset the hidden state vector of the cross-modal temporal attention network.

[0023] In one embodiment, the specific process of resetting the hidden state vector of a cross-modal temporal attention network includes extracting the final hidden state vector corresponding to the most recent valid sliding window that was not discarded before the reset. In a cross-modal temporal attention network, if the network structure includes recurrent neural network layers or introduces a temporal buffering mechanism with relative position encoding, the network will process the... When there are 10 effective sliding windows, a final hidden state vector will be output. ,in is the dimension of the hidden layer. This vector encodes historical temporal context information. When the When a window is deemed misaligned and discarded, directly setting the hidden state vector to zero would cause the model to lose its historical memory in the reinitialized window. Therefore, the system needs to adjust the final hidden state vector... Apply attenuation factor Generate compensated hidden state vector : ; Among them, attenuation factor The value range is set to Attenuation factor The size of this value is negatively correlated with the number of consecutively discarded windows. If the number of consecutively discarded windows is... ,but ,in This is the decay rate hyperparameter. The system will compensate for the hidden state vector. This serves as the initial input to the re-initialized cross-modal temporal attention network. In the specific forward inference computation, if the network employs a long short-term memory (LSTM) network structure, then... Assign the cell state at the initial moment of the new window. With hidden state : , ; By using the compensated hidden state vector to perform forward inference on the dynamic engagement evaluation model within the reset window, the model retains a certain degree of prior knowledge of historical context when processing new valid data, thus avoiding the complete break of temporal context information caused by sudden events.

[0024] In one embodiment, when the entropy value of the attention weight distribution... Not lower than the preset dispersion threshold At this point, the system determines that the feature alignment is valid. The system then uses the attention weight distribution to perform a weighted summation of the visual feature matrix, generating a weighted fused feature. : ; in, Value matrix The first in A visual feature vector. This weighted fusion feature. Redundant visual information with a high sampling rate irrelevant to the current behavioral event is suppressed, highlighting feature components that are truly aligned in the temporal dimension. The system inputs weighted fused features into the dynamic engagement evaluation model. The dynamic engagement evaluation model employs a combined architecture of temporal convolutional networks and fully connected layers. The temporal convolutional network contains multiple layers of one-dimensional causal convolutional layers and dilated convolutional layers. In the first... In layer dilated convolution, the dilation coefficient is set to... The kernel size is set to Causal convolution ensures that the model is accurate at computation time. When outputting, it only depends on time. Including previous input features, to prevent future information leakage. The output feature map of the temporal convolutional network is compressed in time dimension by a global average pooling layer to obtain a fixed-dimensional global feature vector. This global feature vector is then input into a classification head containing a multilayer perceptron, and after processing by the Softmax activation function, the input state vector is output. ,in This represents the total number of engagement categories. Engagement State Vector Each dimension corresponds to a probability value for a category of engagement, including deep focus, superficial browsing, cognitive overload, and distraction.

[0025] In one embodiment, reference Figure 6 The system performs a hash mapping between the engagement state vector output by the dynamic engagement assessment model and the instruction tags in the intervention strategy library, enabling rapid retrieval and generation of intervention instructions. The intervention strategy library is stored in an in-memory database and contains multiple intervention records, each consisting of intervention action text and corresponding instruction tags. During the hash mapping process, the system first hashes the engagement state vector... Binarization and quantization are performed to generate state hash codes. The system then determines the dimension index corresponding to the maximum probability value in the engagement state vector. And extract the probability value of that dimension. Set a binarization threshold. This threshold is determined based on the quantile of the distribution of the model's maximum probability values ​​on the training set. A sign function is used to binarize the engagement state vector: ; in, The first of the state hash codes _ bits. Concatenate all bits in dimensional order to generate a length of _ bits. State hash code Accordingly, during the preprocessing stage of the intervention strategy library, the system has applied the same binarization threshold to the baseline category probability vector corresponding to each instruction label. Quantization was performed, generating pre-stored tag hash codes. The system calculates the Hamming distance between the state hash code and the tag hash codes corresponding to each instruction tag pre-stored in the intervention strategy library. The Hamming distance is defined as the number of distinct bits at the same position in two binary strings of equal length. ; in, This represents the XOR operation. Indicates the first The hash code of the first tag Each bit. The system iterates through the intervention strategy library and selects the target label hash code with the smallest Hamming distance: ; The intervention actions corresponding to the target label hash codes are encapsulated into an intervention instruction stream. The intervention instruction stream is encapsulated in JSON format, including instruction type field, trigger timestamp field, and action parameter field. It is sent to the message queue of the teaching system via WebSocket long connection protocol to drive the teaching terminal to perform feedback actions such as highlighting reminders on the interface, popping up guidance prompts, or adjusting the difficulty of learning materials.

[0026] In one embodiment, during the selection of the target label hash code with the minimum Hamming distance, the system handles potential hash collisions. When there are at least two candidate label hash codes with equal and minimum Hamming distances, it indicates that the engagement state vector, after binarization, simultaneously matches multiple intervention strategies. In this case, the system extracts the model confidence associated with the dynamic engagement evaluation model's output engagement state vector. Model confidence. Defined as the maximum probability value in the engagement state vector: ; Simultaneously, the system retrieves the historical intervention success rates associated with candidate label hash codes from the intervention strategy library. For the first... The system retrieves the hash codes of each candidate label from the database, and records the percentage of times that the intervention restored the trainee's engagement to normal in historical execution records; this percentage is recorded as the historical intervention success rate. The system calculates a comprehensive score for candidate labels by weighting and summing the model confidence score with the historical intervention success rate. : ; in, The result of normalizing the model confidence score is scaled to... Interval. and The preset weighting coefficients satisfy... . This reflects the deterministic weights of the current model's judgment. This reflects the weighting of historical prior experience. The system selects the candidate label hash code with the largest weighted sum as the target label hash code: ; By introducing a weighted summation mechanism of model confidence and historical intervention success rate, when faced with fuzzy matching caused by hash collisions, the system can combine the certainty of the current identification with the actual effect of historical interventions to select the intervention instruction that best fits the current situation and has high reliability, thereby reducing the probability of mismatched intervention instructions caused by information loss due to binarization and quantization.

[0027] In one embodiment, for extracting eye movement trajectory features of the facial region using optical flow, the system employs an image pyramid multi-scale strategy to handle the large displacement caused by rapid eye rotation. The system constructs a Gaussian pyramid for single-frame facial images in a sparse video frame sequence. The number of pyramid layers is set to... , No. The first layer is the original image. Layer image by the first The layer image is obtained by Gaussian smoothing and downsampling at intervals. The resolution of the layer image becomes the first layer. Layer The initial process of optical flow calculation begins at the very top of the pyramid. Starting with the first layer, in this image layer, the apparent displacement of the eyeball is small, satisfying the linear assumption of the optical flow constraint equation. Initial optical flow estimates are calculated at the top layer. Then, multiply the estimated value by As the first The initial guess value of the first layer is passed to the next layer. In the layer calculation, the system iteratively solves the residual image. Let the layer be... The guessed optical flow of the layer is The spatial and temporal gradients between the current frame and the previous frame image after the speculative optical flow offset are calculated. An overdetermined system of equations is constructed and the incremental optical flow is solved using the least squares method. Update # The optical flow estimate of the layer is Repeat this iterative process until the incremental optical flow is less than a preset convergence threshold or the maximum number of iterations is reached. Finally, the bottom layer, i.e., the... The optical flow vector obtained by solving the layer This serves as the precise eye movement trajectory feature of the facial region within the current time interval. This multi-scale iterative strategy avoids computational drift caused by the failure of the first-order Taylor expansion approximation on a single-layer image, thus improving the robustness of eye movement feature extraction in high-sampling-rate visual data streams.

[0028] In one embodiment, for facial keypoint tracking algorithms to extract facial micro-expression variation features, after the system outputs initial coordinates from the cascaded regression tree, a Kalman filter is introduced to smooth the temporal keypoint coordinates. Since learners may experience slight head movements or be affected by lighting noise during online learning, directly calculating the deformation vectors of adjacent frames would introduce a large amount of high-frequency noise. The system assigns an independent Kalman filter to each of the 68 facial keypoints. For the... The key point has a state vector defined as follows: ,in and For the first The two-dimensional coordinates of the frame and Let be the rate of change of the coordinates. The state transition equation is set as a uniform motion model: ; in, Here is the state transition matrix. The process noise has the following covariance matrix: The observation vector is the coordinate output by the cascaded regression tree. The observation equation is: ; in, For the observation matrix, The covariance matrix of the observation noise is: The system performs a prediction and update step upon receiving the keypoint coordinates of each frame. In the prediction step, a prior state estimate is calculated. Covariance of prior error In the update step, the Kalman gain is calculated. And update the posterior state estimate. and posterior error covariance The system extracts the coordinate components from the updated posterior state estimate as the filtered keypoint coordinates. When calculating facial micro-surface change features, the relative displacement is calculated using filtered coordinates. By using state recursion with a Kalman filter, the system effectively filters out transient spike noise in the facial keypoint sequence, ensuring that the extracted micro-surface features purely reflect the real muscle movements of the face.

[0029] In one embodiment, for parsing the Document Object Model (DOM) tree of an online learning platform and capturing interactive events, the system employs an incremental tree differencing algorithm to reduce computational overhead. During the operation of the web-based learning platform, the DOM tree dynamically changes based on user actions such as page turning and expanding / collapsed menus. The system maintains a hash fingerprint mapping table of the DOM tree from the previous moment in memory. This mapping table hashes the unique identifier of each node in the DOM tree with the structural features of its subtrees to generate a fingerprint value. When the MutationObserver interface triggers a callback function to report a node change, the system does not perform a full traversal of the entire tree but instead locates the node that has changed. It extracts the current structural features of the changed node and recalculates the hash fingerprint. The new fingerprint is compared with the old fingerprint in the mapping table. If the fingerprint has changed, it indicates a substantial change in the subtree structure or attributes of the node, and the system marks the node as an active node and records the change timestamp. If the fingerprint has not changed, it is considered an invalid change, and the event is ignored. By using an incremental tree difference algorithm, the system filters out document object model tree change events that do not change the interactive semantics, such as style redrawing and hiding node display. It only captures the core interactive events that truly affect the learner's learning behavior trajectory, thereby reducing data redundancy in the behavioral interaction data stream and improving the signal-to-noise ratio of subsequent behavioral feature extraction.

[0030] In one embodiment, to extract the frequency characteristics of mouse click coordinate changes in interactive events, the system introduces a median filtering mechanism into the first-order difference result to remove invalid jitter displacements. During the student's static viewing of the screen, due to slight camera vibrations or electromagnetic interference from the mouse hardware, a large number of tiny coordinate fluctuations are recorded in the circular buffer. If the first-order difference is calculated directly, these tiny fluctuations will accumulate, leading to inaccurate frequency characteristics. The value is artificially high. The system sets a sliding median filter window, and the window size is set to [value missing]. For the difference sequence within the circular buffer The system slides a median filter window across the sequence and calculates the median of all differences within the window. And replace the difference value at the center of the window with this median: ; After median filtering, isolated spike noise is effectively removed, while continuous large-amplitude displacements caused by actual clicks or drags by the student are preserved. The system uses the filtered difference sequence when calculating frequency characteristics. The original sequence is replaced with an indicator function for accumulation. This processing mechanism ensures that the frequency of mouse click coordinate changes accurately reflects the learner's willingness to interact, avoids misjudging hardware noise as high-frequency interaction behavior, and thus guarantees the consistency between the behavioral feature sequence and the visual feature sequence in representing the learner's engagement.

[0031] In one embodiment, for dynamically setting the left and right boundaries of the sliding window, the system introduces a lower bound constraint in the time dimension when calculating the shortened window length. When the window shortening mechanism is triggered because the variance of the behavioral feature sequence exceeds a preset variance threshold, the system must ensure that the shortened window can still accommodate the complete visual feature extraction cycle. The visual feature extraction cycle is determined by the processing latency of the optical flow method and the facial keypoint tracking algorithm, and the minimum time span of this cycle is set to... The system calculates the shortened window length. Apply boundary constraints: ; when When this occurs, it indicates that the current shortening operation will cause the sliding window to be unable to cover a complete facial feature extraction process. In this case, the system will forcibly set the window length to [value]. Determining the final window length Then, the system recalculates the left and right boundaries based on this length. During the boundary resetting process, if the left boundary... If the time exceeds the start time of the current sparse video frame sequence, the system will hard truncate the left boundary to the start time of the sequence and correspondingly shift the right boundary to the right to maintain the window length. The system remains unchanged. By introducing a lower bound constraint and a boundary truncation compensation mechanism in the time dimension, the system can adapt to the dynamic window adjustment of high variance behavior while strictly ensuring the temporal integrity of the visual data input to the cross-modal temporal attention network, preventing incomplete feature input due to excessively narrow windows or boundary overflows.

[0032] In one embodiment, for a multi-head self-attention layer in a cross-modal temporal attention network, the system employs a grouped linear mapping mechanism to achieve multi-head parallel computation. The system sets the number of attention heads to be [number missing]. When converting the behavioral feature matrix and visual feature matrix into query matrix, key matrix, and value matrix, the system does not use a single, large weight matrix. Instead, it segments the input features along the last dimension. (Behavioral feature matrix) Segmented along the feature dimension Submatrix ,in The feature dimension of each submatrix is Similarly, the visual feature matrix Divided into Submatrix For the first Each attention head uses a dedicated weight parameter matrix. Perform a linear mapping to generate the first... Header query matrix Key matrix AND-value matrix Within each head, the attention weight distribution for the query is independently computed based on the anchor behavior features. With weighted fusion features After all heads have been computed, the system concatenates the weighted fused features of each head along the feature dimension: ; Subsequently, the concatenated features are input into an output linear mapping layer. Perform feature fusion: ; Through the multi-head attention mechanism, the system can capture the correlation weights between behavioral events and different visual representations (such as eye movement representations and micro-surface change representations) in different feature subspaces, which enriches the feature extraction capability of cross-modal temporal attention networks and improves the effectiveness of subsequent entropy value determination and feature alignment.

[0033] In one embodiment, for the logarithmic operation in the process of calculating the entropy value of the attention weight distribution, the system uses the natural constant. The logarithmic function with base is replaced by The system uses a logarithmic function with a base of 0 and adjusts the discreteness threshold accordingly. In the Shannon entropy formula, changing the base of the logarithm only affects the scale of the entropy value without altering its physical monotonicity. The entropy formula calculated using the natural logarithm is as follows: ; The corresponding dispersion threshold is set to .when When the feature alignment fails, the timestamp corresponding to the anchor point at the time of failure is recorded. Afterwards, the system not only resets the sliding window but also maintains a queue of expired timestamps in memory. This queue uses a first-in, first-out (FIFO) strategy, storing the most recent timestamps. The timestamp of the second expiration The system monitors the interval between adjacent timestamps in this queue. When there are multiple consecutive failures with intervals smaller than a preset dense failure threshold. If the system detects an extremely abnormal interaction pattern, causing the system to continuously fail to align features, it sends a degradation intervention command to the teaching system. This command bypasses the hash mapping process and directly instructs the teaching system to pause the current dynamic evaluation model inference. Instead, it sends basic environmental detection prompts to the student's terminal, such as suggesting the student adjust the lighting or posture. This mechanism prevents the system from getting stuck in an ineffective loop of repetitive calculations and window resets under extreme abnormal conditions, ensuring the overall operational stability of the online learning engagement intervention system.

[0034] In one embodiment, for the temporal convolutional network in the dynamic engagement evaluation model, the system specifies in detail the mathematical operations of causal convolution and dilated convolution. The input to the temporal convolutional network is a weighted fused feature sequence processed by a cross-modal temporal attention network. ,in This represents the length of the time step within the sliding window. In the... In layer causal dilated convolution, the dilation coefficient is set to... The kernel size is The weight tensor of this convolutional kernel is represented as: ,in For the input feature dimension, Let be the output feature dimension. For the th in the output feature map... Feature vectors at each time step Causal convolution is achieved by traversing the preceding sequence with a span of... Calculations are performed based on historical inputs: ; in, The first in the convolution kernel Weight matrix for each position, This is the bias vector. When hour, It is padded with zero vectors. This continues as the network layers increase. As the value increases, the expansion coefficient grows exponentially. This causes the size of the receptive field to increase exponentially, as shown in the formula: This structure enables temporal convolutional networks to capture long-range temporal dependencies within a sliding window without increasing network depth or the number of parameters. After each causal dilated convolutional layer, a weight normalization layer is added to accelerate model convergence, and a ReLU activation function is introduced to introduce nonlinearity. Finally, residual connections are used to add the input and output of the convolutional layer to prevent gradient degradation in deep networks. The output feature map after processing through multiple layers of temporal convolutional networks is then compressed into a one-dimensional feature vector through global average pooling and input to a fully connected classification head to output an engagement state vector.

[0035] In one embodiment, for the process of generating state hash codes by binarizing and quantizing the engagement state vector, the system employs an adaptive threshold determination mechanism based on the maximum inter-class variance method. During the model training phase or the system offline initialization phase, the system collects engagement state vectors from a large number of samples. For the engagement state vector of a single sample... The system extracts the maximum probability value. After collecting data containing... The maximum probability set of samples Then, the system uses the maximum inter-class variance method to find the optimal binarization threshold on this set. Set a candidate threshold. ,Will Divide into two sets: confidence set AND and non-confidence sets Calculate the within-class variance of two sets. ,in The sample ratio of the two sets. Let V be the variance of the two sets. System traversal. All possible values ​​are used as candidate thresholds Find the within-class variance Minimize, or equivalently minimize, the inter-class variance The largest threshold is used as the final adaptive binarization threshold. Through this mechanism, the system can assess the calibration degree of the model on the actual data distribution based on the dynamic engagement level, and automatically adjust the strictness of binarization, avoiding hash code generation deviations caused by fixed thresholds on different model versions or different student groups.

[0036] In one embodiment, for encapsulating the intervention action corresponding to the target label hash code into an intervention instruction stream, the system defines a message encapsulation protocol with priority queue characteristics. The JSON format message of the intervention instruction stream includes an instruction identifier field, a priority field, a trigger timestamp field, an action type field, and an action payload field. The instruction identifier is a unique string generated by the system using the MD5 algorithm based on the current timestamp and a random number. The priority field is set according to the probability value of the distracted / absent-minded category in the engagement state vector; the higher the probability value, the higher the priority value. The action type field includes interface highlighting, pop-up prompts, difficulty adjustment, and rest suggestions. The action payload field stores parameters corresponding to the action type, such as the XPath path of the document object model node for interface highlighting, or the target difficulty coefficient for difficulty adjustment. Before sending the intervention instruction stream to the teaching system, the system stores it in a priority sending queue located in memory. The queue is sorted in descending order of priority value. When the network communication module is idle, the system retrieves the instruction stream from the head of the queue and sends it via the WebSocket protocol. If the sending fails, the system re-inserts the instruction stream into the queue and triggers a backoff retransmission mechanism. Through a priority queue mechanism, the system ensures that when multiple intervention instructions are generated in a short period of time, the teaching system can prioritize the execution of high-priority instructions for severe inattentiveness, thus guaranteeing the rationality of the timing of intervention actions and the ability to respond urgently.

[0037] In one embodiment, for the weighted summation process when selecting the target label hash code with the minimum Hamming distance, the system dynamically configures the weight coefficients of model confidence and historical intervention success rate. The system bases this configuration on the duration of the current online learning session. Dynamic adjustment and In the initial stage of the session, the historical intervention success rate... It relies on global historical data and lacks adaptability to the specific state of the current learner. Therefore, in Less than the preset duration threshold At that time, the system settings For larger values, such as , The system primarily relies on the model confidence score of the current frame to make decisions. As the learning time increases, the system accumulates local intervention feedback data for that learner, at which point the system gradually increases... The weights. The specific dynamic adjustment formula is: , ; in, The initial minimum success rate weight. This is the weight for maximizing the success rate. Through this exponential decay function, the system further improves the accuracy of matching intervention commands with the distribution of individual student characteristics when handling hash collisions.

Claims

1. An online learning engagement real-time intervention method based on multi-modal data fusion, characterized in that, include: The visual data stream and behavioral interaction data stream during the online learning process are acquired, and the visual feature sequence and behavioral feature sequence are extracted respectively through a sliding window dual-stream temporal alignment mechanism. The visual feature sequence and the behavioral feature sequence are input into a cross-modal temporal attention network. Using the timestamps of behavioral events in the behavioral feature sequence as anchors, the attention weight distribution of the visual feature sequence to the behavioral events is calculated within the sliding window. Calculate the entropy value of the attention weight distribution. When the entropy value is lower than the preset dispersion threshold, it is determined that the feature alignment has failed and the current sliding window data is discarded. Otherwise, the weighted fused features are input into the dynamic engagement evaluation model. The engagement state vector output by the dynamic engagement assessment model is hashed and mapped with the instruction tags in the intervention strategy library to generate an intervention instruction stream to drive the teaching system to execute feedback.

2. The method of claim 1, wherein, The extraction of visual feature sequences via the sliding window dual-stream temporal alignment mechanism includes: The visual data stream is downsampled in the temporal dimension to obtain a sparse video frame sequence; The sparse video frame sequence is used to locate facial regions, and the eye movement trajectory features of the facial regions are extracted using optical flow. The facial micro-expression features are then extracted using a facial key point tracking algorithm. The visual feature sequence is generated by arranging the eye movement trajectory features and the facial micro-expression change features in the chronological order of the sparse video frame sequence.

3. The method of claim 1, wherein, The extraction of behavioral feature sequences via the sliding window dual-stream temporal alignment mechanism includes: Parse the document object model tree of the online learning platform, and listen for and capture interactive events in the behavioral interaction data stream; Extract the frequency characteristics of mouse click coordinate changes and the duration of page dwell on the current webpage element from the interactive events; Based on the occurrence sequence of the interaction events, the frequency feature of mouse click coordinate changes and the feature of page dwell time are serialized and concatenated to generate the behavior feature sequence.

4. The method of claim 1, wherein, The anchoring point for the timestamps of behavioral events in the behavioral feature sequence includes: Identify the state transition event in the behavioral feature sequence, and set the timestamp of the transition event as the anchor point; Centered on the anchor point, the left and right boundaries of the sliding window are dynamically set according to the sampling frequency ratio of the visual data stream and the behavioral interaction data stream, so that the sliding window contains at least one complete visual feature extraction cycle.

5. The method according to claim 1, characterized in that, The calculation of the entropy value of the attention weight distribution includes: The attention weight values ​​assigned to each visual feature node within the sliding window are statistically analyzed to construct an attention probability distribution matrix; The entropy value is obtained by performing a logarithmic operation on the attention probability distribution matrix and summing the results. When feature alignment fails, the timestamp corresponding to the anchor point at the time of failure is recorded, and the sliding window is reinitialized starting from the timestamp to reset the hidden state vector of the cross-modal temporal attention network.

6. The method according to claim 1, characterized in that, The step of hash mapping the engagement state vector with the instruction labels in the intervention strategy library includes: The engagement state vector is binarized and quantized to generate a state hash code; Calculate the Hamming distance between the state hash code and the tag hash codes corresponding to each instruction tag pre-stored in the intervention strategy library; Select the target label hash code with the smallest Hamming distance, and encapsulate the intervention action corresponding to the target label hash code into the intervention instruction stream.

7. The method according to claim 2, characterized in that, The downsampling process of the visual data stream in the time dimension includes: The visual data stream is received at the edge computing node deployed on the student terminal; In the edge computing node, the frame extraction ratio of the downsampling process is adaptively adjusted according to the current inference stage of the dynamic input evaluation model; When the dynamic engagement evaluation model is in the feature extraction stage, a first frame extraction ratio is used. When the dynamic engagement evaluation model is in the state classification stage, it is switched to a second frame extraction ratio that is lower than the first frame extraction ratio.

8. The method according to claim 4, characterized in that, The dynamic setting of the left and right boundaries of the sliding window includes: Retrieve the current student's historical behavior and interaction logs, and calculate the historical average time interval between adjacent transition events; The product of the historical average time interval and the ratio of the sampling frequency is used as the base window length; Obtain the variance of the behavioral feature sequence within the basic window length. When the variance is greater than a preset variance threshold, shorten the basic window length to generate a shortened window, and use the boundary of the shortened window as the left boundary and the right boundary.

9. The method according to claim 5, characterized in that, The resetting of the hidden state vector of the cross-modal temporal attention network includes: Extract the final hidden state vector corresponding to the most recently undiscarded valid sliding window before the reset; A decay factor is applied to the final hidden state vector to generate a compensated hidden state vector; The compensated hidden state vector is used as the initial input to the re-initialized cross-modal temporal attention network, and the compensated hidden state vector is used to perform forward inference on the dynamic engagement evaluation model within the reset window.

10. The method according to claim 6, characterized in that, The selection of the target label hash code with the smallest Hamming distance includes: When there are at least two candidate label hash codes with equal Hamming distance and both being minimum, extract the model confidence that accompanies the output of the engagement state vector by the dynamic engagement evaluation model. Retrieve the historical intervention success rates associated with the candidate label hash codes in the intervention strategy library; The model confidence score and the historical intervention success rate are weighted and summed, and the candidate label hash code with the largest weighted sum is selected as the target label hash code.

Images