Online training patrol method and system and storage medium

Abstract

The present disclosure relates to an online training patrol method and system and a storage medium, and particularly relates to the technical field of computers. A multi-modal data fusion technology is adopted, and the learning state of a student is comprehensively judged according to multi-dimensional data such as images (such as expressions / postures), audio (such as voices / environmental sounds), interactions (such as mouse / keyboards), texts (such as answers to questions / chats), etc., so as to improve the accuracy of judgment and reduce misjudgment. Furthermore, the feature vectors of multi-dimensional data are extracted and fused, and based on the fused feature vectors, the learning state of the student is automatically predicted, which can not only improve the patrol efficiency of online training, so that the patrol work can cover all students, but also can reduce the subjective influence when artificially judging, and improve the consistency of the patrol standard.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an online training inspection method, system, and storage medium. Background Technology

[0002] Current large-scale online training inspections rely heavily on single-modal data such as video or audio, resulting in significant limitations to the overall inspection system. For example, simply seeing a trainee sitting in front of a screen through video does not reveal whether the trainee is attentively watching the course or running games, reading other videos, or reading novels in the background.

[0003] At the inspection execution level, current methods still primarily rely on training administrators conducting manual spot checks. This inspection model is inefficient, and when there are many trainees, it is difficult to cover all of them. Furthermore, the subjectivity of manual judgment greatly affects the consistency and fairness of inspection standards, impacting the order of online training and the trainees' learning experience. Different administrators have significantly different definitions of "abnormal behavior": some administrators consider a trainee looking down for more than 30 seconds to be inattentive, while others may relax the standard to 1 minute. Summary of the Invention

[0004] This embodiment employs multimodal data fusion technology to comprehensively assess learners' learning status based on multi-dimensional data such as images (e.g., facial expressions / postures), audio (e.g., voice / ambient sound), interactions (e.g., mouse / keyboard), and text (e.g., answering questions / chatting), thereby improving the accuracy of the assessment and reducing misjudgments. Furthermore, it extracts and fuses feature vectors from the multi-dimensional data, and automatically predicts learners' learning status based on the fused feature vectors. This not only improves the efficiency of online training inspections, ensuring that inspections cover all learners, but also reduces the subjective influence of manual judgments and improves the consistency of inspection standards.

[0005] This disclosure proposes an online training monitoring method, comprising: collecting image information, audio information, interaction information, and text information from the student's terminal; extracting features from the image information, audio information, interaction information, and text information respectively to obtain image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors; performing cross-modal self-attention processing on the image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain fused feature vectors; and predicting the student's learning status based on the fused feature vectors.

[0006] In some embodiments, collecting image information from the student terminal includes: collecting video streams from the student terminal and extracting keyframe images from the video stream; when the structural similarity between any first keyframe image and a second keyframe image is greater than a preset value, collecting the first keyframe image and skipping the collection of the second keyframe image; or, collecting audio information from the student terminal includes: collecting audio streams from the student terminal and distinguishing between student speech and environmental noise in the audio stream; extracting Mel-frequency cepstral coefficients, speech activity, and emotional tendency of the student speech; or, collecting interaction information from the student terminal includes: monitoring student behavior information, including: mouse movement trajectory, keyboard events, and interface operation sequences; generating a student behavior summary vector based on the student behavior information, including: operation entropy value, number of actions per unit time, and abnormal behavior matching degree; or, collecting text information from the student terminal includes: real-time crawling of online training chat room questions, correct and incorrect records of the answer system, and bullet screen interactive text; based on the crawled information, using the BERT-mini model to perform semantic analysis on question difficulty classification, sentiment polarity detection, and knowledge point relevance.

[0007] In some embodiments, feature extraction is performed on image information, audio information, interaction information, and text information to obtain image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors, respectively. This includes: processing image information sequentially through convolutional layers, multi-level residual blocks, global average pooling, and fully connected layers to obtain image feature vectors; or processing audio information sequentially through an LSTM model and fully connected layers to obtain audio feature vectors; or processing interaction information sequentially through the input layer, hidden layer, output layer, and normalization layer of a deep feature encoder to obtain interaction feature vectors; or processing text information sequentially through global average pooling, linear layers, and normalization layers to obtain text feature vectors.

[0008] In some embodiments, cross-modal self-attention processing is performed on image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain a fused feature vector. This includes: concatenating image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain a feature matrix; using learnable query weight matrix, key weight matrix, and value weight matrix, linear transformations are performed on the feature matrix to obtain a query matrix, key matrix, and value matrix, respectively; a dot product operation is performed using the transpose of the query matrix and the key matrix, and the result of the dot product operation is normalized to obtain a weight matrix, which is used to characterize the correlation strength between modalities; and a weighted summation is performed on the value matrix using the weight matrix to obtain a fused feature vector, which is used to characterize the fusion information of cross-modal features.

[0009] In some embodiments, predicting a learner's learning state based on the fused feature vector includes at least one of the following: inputting the fused feature vector into a focus prediction head to obtain a learner's focus prediction result; inputting the fused feature vector into an emotional state prediction head to obtain a learner's emotional state prediction result; inputting the fused feature vector into a comprehension prediction head to obtain a learner's comprehension prediction result; inputting the fused feature vector into an abnormal behavior prediction head to obtain a learner's abnormal behavior prediction result; and inputting the fused feature vector into a learning outcome prediction head to obtain a learner's learning outcome prediction result.

[0010] In some embodiments, the attention prediction head, built based on a regression task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and finally multiply the processing result by a score cap to output an attention score within the range of 0 to the score cap; the emotion state prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, then apply a softmax activation function in dimension 1 to calculate and output a probability distribution representing multiple emotion state types; the comprehension prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function to output a comprehension probability within the range of 0 to 1; the abnormal behavior prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function to output an abnormality probability within the range of 0 to 1; the learning effect prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, then apply a softmax activation function in dimension 1... The above calculation outputs a probability distribution representing various learning effect types.

[0011] In some embodiments, the student terminal is configured to collect image information, audio information, interaction information, and text information from the student terminal side; extract features from the image information, audio information, interaction information, and text information respectively to obtain image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors, and upload them to the training server; the training server is configured to perform cross-modal self-attention processing on the image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain fused feature vectors; and predict the student's learning status based on the fused feature vectors.

[0012] In some embodiments, the feature extraction models for each student terminal are trained using a federated learning method.

[0013] In some embodiments, the method further includes: using gated attention to adjust the weights of feature vectors of different modalities.

[0014] In some embodiments, the method further includes: identifying group training risks based on multiple fusion feature vectors corresponding to multiple trainees.

[0015] In some embodiments, the method further includes: sending a network anomaly alarm message to the management terminal when a network service anomaly occurs during group training; or sending a prompt message to the teacher terminal to adjust the teaching content when a misunderstanding of knowledge points occurs during group training; or sending a cheating warning message to both the teacher terminal and the student terminal when cheating occurs during group training.

[0016] In some embodiments, the method further includes: sending a learning status prompt message to the student's terminal when the student's learning status meets preset prompt conditions.

[0017] This disclosure provides some embodiments of an online training inspection system, including one or more modules for performing online training inspection methods.

[0018] Some embodiments of this disclosure propose an online training inspection system, including: a memory; and a processor coupled to the memory, the processor being configured to execute an online training inspection method based on instructions stored in the memory.

[0019] Some embodiments of this disclosure propose a computer-readable storage medium storing computer instructions that, when executed by a processor, implement an online training and inspection method.

[0020] Some embodiments of this disclosure propose a computer program product, including computer instructions that, when executed by a processor, implement an online training and inspection method. Attached Figure Description

[0021] The accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. This disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings.

[0022] Obviously, the accompanying drawings described below are merely some embodiments of this disclosure. Those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0023] Figure 1 A flowchart illustrating an online training inspection method according to some embodiments of this disclosure is shown.

[0024] Figure 2 A schematic diagram of an online training and inspection system according to some embodiments of this disclosure is shown.

[0025] Figure 3A schematic diagram of an online training and inspection system according to some embodiments of this disclosure is shown. Detailed Implementation

[0026] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0027] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0028] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0029] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0030] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0031] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0032] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0033] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0034] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0035] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0036] Furthermore, in order to avoid obscuring this disclosure with unnecessary details, only processing steps and / or device structures closely related to the scheme at least according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0037] In online training scenarios, the online training system includes a training server and user terminals; user terminals include student terminals and teacher terminals. The training server, student terminals, and teacher terminals work collaboratively to support the efficient operation of the entire training process (from pre-class preparation and in-class interaction to post-class management). The training server is the "brain" and "data warehouse" of the online training system, responsible for processing requests from all terminals, storing critical data, and scheduling system resources. It is the core device ensuring the stability and security of training, and its functions can be divided into data management, business support, resource scheduling, and security assurance. Student terminals are the "entry point" for students to participate in online training, and need to meet the needs of "convenient learning, real-time interaction, and post-class review." Common forms include, but are not limited to, computers, smartphones, and tablets. Teacher terminals are the "toolboard" for teachers to conduct online teaching, and need to meet the needs of "course building, real-time teaching, interactive guidance, and post-class management." Common forms are mainly computers, with some scenarios supporting tablets and smartphones.

[0038] Figure 1 A flowchart illustrating some embodiments of the online training inspection method of this disclosure is shown. Figure 1 As shown, the online training inspection method of this embodiment includes steps 110-140.

[0039] In step 110, image information, audio information, interaction information, and text information are collected from the student's terminal.

[0040] In step 120, feature extraction is performed on the image information, audio information, interactive information, and text information respectively to obtain image feature vector, audio feature vector, interactive feature vector, and text feature vector.

[0041] In step 130, cross-modal self-attention processing is performed on the image feature vector, audio feature vector, interaction feature vector, and text feature vector to obtain a fused feature vector.

[0042] In step 140, the student's learning status is predicted based on the fused feature vector.

[0043] This embodiment employs multimodal data fusion technology to comprehensively assess learners' learning status based on multi-dimensional data such as images (e.g., facial expressions / postures), audio (e.g., voice / ambient sounds), interactions (e.g., mouse / keyboard), and text (e.g., answers / chat), thereby improving the accuracy of the assessment and reducing misjudgments. Furthermore, it extracts and fuses feature vectors from multi-dimensional data, and automatically predicts learners' learning status based on the fused feature vectors. This not only improves the efficiency of online training inspections, ensuring that inspections cover all learners, but also reduces the subjective influence of manual judgments and improves the consistency of inspection standards.

[0044] For example, in traditional inspection methods, if a student is seen looking down (taking notes) in video monitoring, it may be mistakenly judged as abnormal behavior of not paying attention in class. However, the inspection method disclosed in this paper judges normal behavior based on a comprehensive analysis of characteristics such as "note-taking behavior + correct answer rate + no abnormal voice".

[0045] Step 110, collecting image information from the student terminal, includes: acquiring the video stream from the student terminal and extracting keyframe images from the video stream; when the structural similarity (SSIM) of any first keyframe image and second keyframe image is greater than a preset value, acquiring the first keyframe image and skipping the acquisition of the second keyframe image. Thus, based on adaptive keyframe extraction, the amount of data that needs to be processed and transmitted is reduced, improving the real-time performance of online training monitoring in online training scenarios. For example, skipping frames when the structural similarity is greater than 0.8 can reduce the amount of data by at least 50%. Furthermore, the video stream is captured through the student terminal's camera; the resolution / frame rate of the video stream can be 1080P / 30fps to ensure video clarity and smoothness.

[0046] The acquisition of audio information from the student terminal in step 110 includes: acquiring the audio stream from the student terminal, distinguishing between student speech and environmental noise in the audio stream, and extracting acoustic features such as Mel Frequency Cepstral Coefficient (MFCC), Voice Activity Detection (VAD), and emotional tendency of the student speech.

[0047] Audio streams are acquired using a microphone array, with a sampling rate of, for example, 16 kHz. Multichannel noise suppression (RNNoise algorithm) and source separation (Conv-TasNet algorithm) are applied to distinguish student speech from ambient noise.

[0048] Step 1: Multi-channel noise suppression (RNNoise algorithm) to initially filter environmental noise. The RNNoise algorithm is a noise suppression technology specifically designed for speech scenarios. It combines sound data collected from multiple channels (such as multiple microphones) to initially filter environmental noise: (1) Noise feature learning: The algorithm first analyzes the characteristics of the speech signal (such as the frequency of human speech being concentrated in 300-3400Hz and having obvious periodic waveforms), and at the same time captures the commonalities of environmental noise through multi-channel data (such as air conditioner noise, keyboard typing, conversations at a distance, etc., these noises are usually dispersed in frequency and have no fixed period). (2) Real-time noise detection and suppression: When the student speaks, the algorithm compares the sound collected from multiple channels. For "non-periodic sounds that exist in all channels" (such as the continuous hum of an air conditioner), it is determined to be environmental noise and suppressed by attenuating its frequency energy; for "sounds that exist in at least one channel and conform to speech characteristics" (such as the student's voice), its clarity is preserved and enhanced to reduce the masking of speech by noise. (3) Dynamically adapt to noise changes: If the environmental noise changes suddenly (such as the sound of a door closing suddenly), the algorithm will quickly update the noise model to avoid mistakenly preserving sudden noise as speech or mistakenly filtering speech as noise.

[0049] Step 2: Sound source separation (Conv-TasNet algorithm) to accurately extract student speech. After the initial denoising by RNNoise, there may still be some noise with frequencies close to the speech (such as whispering conversations between students next to each other). At this time, the Conv-TasNet algorithm will further separate the speech of the target student. (1) Sound signal decomposition: The algorithm decomposes the mixed sound (including student speech, residual noise, and other interference sounds) into multiple "time-frequency feature segments". Each segment contains sound energy information at a specific frequency and time point. (2) Feature differentiation between speech and noise: The difference between "student speech" and "other sounds" is learned through a deep learning model (convolutional neural network). Student speech usually has coherent semantic features (such as continuous syllables and intonation changes). In multi-channel data, the microphone closest to the speaker collects stronger speech energy. Residual noise or interference sounds (such as the sound of turning pages or whispering by others) lack semantic coherence and are more evenly distributed or weaker in energy in multi-channel data. (3) Target speech reconstruction: Based on the feature discrimination results, the algorithm selects the time-frequency segments that belong to the target student's speech, recombines them into a clear speech signal, and marks other segments as noise and removes them.

[0050] The Prosody-LSTM model is used to identify emotional tendencies such as pleasure, confusion, and anxiety. By extracting prosodic features from speech (reflecting changes in rhythm, pitch, and intensity during speech), and then using LSTM (Long Short Term Memory) network to capture the temporal correlations of these prosodic features, emotion classification is ultimately achieved.

[0051] The step 110 involves collecting interactive information from the student's terminal, including: monitoring student behavior information, such as mouse movement trajectory, keyboard events, and interface operation sequences; and generating a student behavior summary vector based on the student's behavior information, including: operation entropy value, number of actions per unit time, and abnormal behavior matching degree.

[0052] For example, monitor students' mouse movement trajectory (sampling rate 100Hz), keyboard events (such as keystroke interval, backspace rate), and interface operation sequences (such as page dwell time, button click hotspots). Based on the students' behavioral information, generate behavioral summary vectors, including, for example, operation entropy values ​​(reflecting purposefulness), number of actions per unit time, and abnormal pattern matching degree (such as frequent window switching via Alt+Tab).

[0053] The text information collected in step 110 from the trainee's terminal includes: real-time crawling of online training chat room questions, correct and incorrect records of the answer system, and interactive text in bullet comments; based on the crawled information, the BERT-mini model is used to perform semantic analysis on question difficulty classification, sentiment polarity detection, and knowledge point relevance.

[0054] The BERT-mini model converts crawled information into high-dimensional semantic vectors (embedding vectors); using semantic vectors, it performs three parallel analysis tasks: question difficulty classification (as a regression or classification task), sentiment polarity detection (as a classification task), and knowledge point association calculation (as a similarity calculation task); it outputs the analysis results of the analysis tasks, such as the [CLS] token (classification label) representation.

[0055] The feature extraction model used in step 120 is a multimodal knowledge distillation model, which is an asymmetric multi-branch network. Each branch network processes one modality of data, and finally the distilled high-dimensional vectors are fused together.

[0056] The following describes the feature extraction process of image branching networks (also known as visual branching networks).

[0057] Step 120 involves extracting features from the image information to obtain an image feature vector, which includes processing the image information sequentially through a convolutional layer, a multi-level residual block, global average pooling, and a fully connected layer to obtain the image feature vector.

[0058] Image branching networks, such as ResNet-18, extract facial feature vectors and limb pose scores, with the input being the student's facial features and limb poses. They operate as a hierarchical, automated "feature refiner," a process that is not instantaneous but gradually implemented through multiple "processing layers" (convolutional layers, residual blocks, pooling layers, and fully connected layers).

[0059] (1) Initial convolution: The image is processed through a simple convolutional layer (usually in conjunction with a pooling layer). This layer mainly detects the lowest-level features in the image, such as edges, corners, color blocks, and brightness variations. This is crucial for subsequent determination of facial contours, the shape of eyes and mouth, etc.

[0060] (2) Hierarchical feature abstraction (core part): The core of ResNet-18 is a layer composed of 4 levels of residual blocks.

[0061] Layer 1 residual blocks: begin combining low-level features to form more complex textures and components. For example, they may detect eyelid lines, the circle of the iris, the curve of the corner of the mouth, etc.

[0062] Layer 2 residual blocks: receive features from Layer 1 and combine them further. They can already roughly identify component-level concepts such as "this is an eye" and "this is a nose".

[0063] Residual blocks in layers 3 and 4: These are the deepest layers of the network with the largest receptive field. They are responsible for integrating global information to form highly abstract, high-level semantic features. Here, the network no longer just sees "eyes" and "mouth," but can understand complex concepts such as "eyes looking to the upper left" (direction of attention), "frowning eyebrows" (micro-expressions), and "the relative position of the body and the chair back" (posture).

[0064] At this point, the forward pass of ResNet-18 is complete, and the original image has been transformed into a set of complex, high-dimensional "feature maps". These feature maps are powerful representations in themselves, but they are high-dimensional and spatially dependent, making them unsuitable for direct computation.

[0065] (3) Feature aggregation and dimensionality reduction.

[0066] To obtain the final fixed-length vector, the network uses Global Average Pooling (GAP). GAP averages all values ​​from each feature map, compressing them into a single number. For example, if layer 4 outputs 512 feature maps, GAP results in a 512-dimensional vector. This 512-dimensional vector is the global feature embedding extracted by ResNet-18 for this image. It highly condenses all semantic information in the image and forms the basis for subsequent tasks. Fully connected layers can reduce the feature vector dimension to, for example, 128 dimensions.

[0067] The following describes the feature extraction process of the audio branch network (also known as the auditory branch network).

[0068] Step 120 involves extracting audio features from the audio information to obtain an audio feature vector, which includes processing the audio information sequentially through an LSTM model and a fully connected layer to obtain the audio feature vector.

[0069] The audio branch network takes as input acoustic features such as Mel-frequency cepstral coefficients, speech activity, and emotional tendency of the trainee's speech. An LSTM is used to process the temporal acoustic features, which are then projected onto the feature vector dimension (e.g., 128-dimensional) through a fully connected layer. Specific implementation steps include:

[0070] (1) LSTM reads the acoustic features of each frame from left to right.

[0071] (2) Update internal state: For each time step, LSTM combines the new input (current acoustic features) and the hidden state of the previous time step (containing historical information of all previous frames) to calculate the output of the current time step and the updated state.

[0072] (3) Generate sequence-aware features: The output (hidden state) of each time step is a richer feature representation that has taken into account all previous context information.

[0073] (4) Output: The output of the last time step aggregates the global information of the entire frame sequence. It can represent the overall semantic summary of an audio segment. This last time step vector is input into a fully connected layer.

[0074] The following describes the feature extraction process of interactive branch networks.

[0075] Step 120 involves extracting features from the interaction information to obtain the interaction feature vector, which includes processing the interaction information sequentially through the input layer, hidden layer, output layer, and normalization layer of a deep feature encoder to obtain the interaction feature vector.

[0076] The interactive branch network is a deep feature encoder that takes as input behavioral summary vectors such as operation entropy, action frequency, and abnormal behavior matching degree. Through a series of nonlinear transformations, the input behavioral summary vectors (which may have low dimensionality and varying feature scales) are mapped into a highly abstract, dense, and well-distributed feature vector dimension (128-dimensional) latent space.

[0077] The deep feature encoder has three advantages: (1) Preserving key information: compressing and extracting the most discriminative information from the input data. (2) Eliminating redundancy: eliminating redundancy and noise in the original features through the hierarchical structure of the network. (3) Enhancing linear separability: making similar behavioral patterns closer and different behavioral patterns farther apart in the 128-dimensional space, which is convenient for subsequent task processing.

[0078] The deep feature encoder consists of the following layers (1)-(4).

[0079] (1) Input layer, which goes through the following layers and processes in sequence.

[0080] Input: Behavior summary vector x_input, assuming its dimension is d (e.g., d=10 or 20, depending on the number of features extracted).

[0081] Linear(d, 256): A linear layer that projects the input into a higher-dimensional space (256 dimensions) to enhance the model's expressive power and allow it to learn more complex combinations of features.

[0082] BatchNorm(d,256): Batch normalization layer. It alleviates the problem of inconsistent dimensions of input features (operational entropy, frequency, etc.), stabilizes the training process, and accelerates convergence.

[0083] ReLU(): Activation function that introduces non-linearity.

[0084] (2) Hidden layer

[0085] Two hidden layers were designed to progressively compress the dimension from 256 to 128, aiming to gradually condense information to the target dimension and perform non-linear transformations. The two hidden layers are:

[0086] Linear(256, 192) -> BatchNorm -> ReLU -> Dropout

[0087] Linear(192, 128) -> BatchNorm -> ReLU -> Dropout

[0088] Each hidden layer sequentially goes through a linear layer, a batch normalization layer, a ReLU activation function, and Dropout. The ReLU activation function enables the model to fit complex data distributions. Dropout randomly discards a subset of neurons (e.g., at a ratio of 0.2), a regularization technique that effectively prevents overfitting and forces the network to learn more robust features.

[0089] (3) Output layer, for example, implemented through a linear layer.

[0090] The linear layer, Linear(128, 128), outputs a 128-dimensional original vector. No activation function (such as ReLU or Sigmoid) is used in the output layer, resulting in a feature vector that varies continuously over the real number range; negative values ​​may also contain important information.

[0091] (4) Feature normalization layer

[0092] The feature normalization layer can be represented as F.normalize(x, p=2, dim=1), that is, when p=2 and dim=1, L2 normalization is performed on x along the first dimension. x: the input feature tensor, p=2: specifies the "norm type", p=2 indicates the use of L2 norm (Euclidean norm), dim=1: specifies the normalization dimension, dim=1 indicates normalization is performed on the first feature dimension of the tensor.

[0093] This is the final crucial step in outputting the 128-dimensional feature vector, which compresses the magnitude of the output vector to 1. Advantage: It maps all feature vectors onto a unit hypersphere. This makes subsequent calculations (such as cosine similarity) extremely efficient and straightforward (the vector dot product is the cosine value), making it highly suitable for similarity-based tasks (such as retrieval, face recognition, clustering, etc.).

[0094] The following describes the feature extraction process for text branching networks.

[0095] Step 120 involves extracting features from the text information to obtain a text feature vector, which includes processing the text information sequentially through global average pooling, a linear layer, and a normalization layer to obtain the text feature vector.

[0096] The input to the text feature extraction model is the [CLS] token representation output by BERT-mini (typically 256 or 512 dimensions). The input text is processed by BERT-mini to obtain the context vector representation of each token (sequence output).

[0097] Global average pooling in text feature extraction models pools the variable-length sequence output ([Seq_Len, Hidden_Dim]) into a fixed-length vector ([Hidden_Dim]), thus aggregating the information of the entire sequence. Hidden_Dim is typically 256 for BERT-mini. Then, a simple linear layer (nn.Linear(256, 128)) projects the 256-dimensional pooled vector onto the target dimension of 128. The output of the feature projection layer is the original 128-dimensional feature vector. L2 normalization is then applied to the 128-dimensional vector to make it a unit vector, thus fixing all vectors onto a unit sphere. This makes subsequent cosine similarity calculations equivalent to inner product calculations, which is highly efficient.

[0098] The final output of the text feature extraction model is a 128-dimensional floating-point vector, where each element has a value between -1 and 1 (due to L2 normalization, but the vector itself can point in any direction in space), with a magnitude of 1. For example, the vector format is: [-0.02, 0.12, 0.45, 0.07, 0.23......0.18] (128 dimensions in total).

[0099] Step 130 involves performing cross-modal self-attention processing on the image feature vector, audio feature vector, interaction feature vector, and text feature vector to obtain a fused feature vector. This includes concatenating the image feature vector, audio feature vector, interaction feature vector, and text feature vector to obtain a feature matrix. A learnable query weight matrix, key weight matrix, and value weight matrix are used to perform linear transformations on the feature matrix to obtain a query matrix, key matrix, and value matrix, respectively. A dot product operation is performed using the transpose of the query matrix and the key matrix, and the result is normalized to obtain a weight matrix, which is used to characterize the correlation strength (correlation) between modalities and capture cross-modal dependencies. Finally, the value matrix is ​​weighted and summed using the weight matrix to obtain a fused feature vector, which is used to characterize the fusion information of cross-modal features.

[0100] Therefore, based on the correlation strength between modalities, feature vectors of heterogeneous modalities are mapped to a unified semantic space, realizing a unified representation of heterogeneous data and solving the problem of synchronous use of multi-source data.

[0101] The self-attention model takes a four-way (four modalities) 128-dimensional feature vector as input and outputs a fused feature vector, represented as follows:

[0102]

[0103] Where Q, K, and V are the feature matrix obtained by concatenating image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors, the query matrix, key matrix, and value matrix obtained by linear transformation, and T represents the transpose. express function.

[0104] The fused feature vector is a dense, interpretable, embedded representation whose dimensions encode, for example, the following high-level semantic information: cognitive state: attention concentration, cognitive load, confusion / insight state; emotional state: engagement / interest level, anxiety / stress level, pleasure level; behavioral patterns: activity enthusiasm, task relevance, abnormal behavior index; knowledge mastery: understanding of current knowledge points, adaptability to problem difficulty; meta-information: modal reliability score (e.g., low confidence of image modal data when the camera is obstructed), etc., but not limited to the examples given.

[0105] Step 140, predicting the learner's learning state based on the fused feature vector, includes at least one of the following: inputting the fused feature vector into the attention prediction head to obtain the learner's attention prediction result; inputting the fused feature vector into the emotional state prediction head to obtain the learner's emotional state prediction result; inputting the fused feature vector into the comprehension prediction head to obtain the learner's comprehension prediction result; inputting the fused feature vector into the abnormal behavior prediction head to obtain the learner's abnormal behavior prediction result; and inputting the fused feature vector into the learning effect prediction head to obtain the learner's learning effect prediction result.

[0106] The attention prediction head, built based on a regression task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and finally multiply the result by a score cap (e.g., 100), outputting an attention score ranging from 0 to the score cap. The output is a continuous value, such as an attention score of 0-100%.

[0107] The sentiment state prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, apply the softmax activation function on dimension 1, and output a probability distribution representing multiple sentiment state types. For example, it outputs the probability distributions for three sentiment state types: [positive, neutral, negative].

[0108] The comprehension prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and output a comprehension probability in the range of 0 to 1. The output is a probability value, for example, 0 represents confusion and 1 represents comprehension.

[0109] The abnormal behavior prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and output an abnormal probability in the range of 0 to 1. The output is a probability value, for example, 0 represents normal and 1 represents abnormal.

[0110] The learning performance prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, apply the softmax activation function on dimension 1, and output a probability distribution representing multiple learning performance types. For example, it outputs the probability distribution of four learning performance types: [Excellent, Good, Average, Poor].

[0111] A comprehensive assessment of student status was achieved through multi-task learning. The use of shared feature extraction and a multi-task head design ensured both performance and improved computational efficiency.

[0112] In some embodiments, the student terminal is configured to collect image information, audio information, interaction information, and text information from the student terminal side; based on the feature extraction model of each modality, feature extraction is performed on the image information, audio information, interaction information, and text information respectively to obtain image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors, and these are uploaded to the training server; the training server is configured to perform cross-modal self-attention processing on the image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain fused feature vectors; and the student's learning status is predicted based on the fused feature vectors.

[0113] Therefore, the terminal does not need to upload the original image / audio / interaction / text information, but only the image / audio / interaction / text feature vector. On the one hand, it protects user privacy data, and on the other hand, it saves transmission bandwidth and enables real-time monitoring of online training. In particular, it can meet the needs of real-time monitoring for large-scale online training.

[0114] Edge preprocessing is deployed on the terminal side. Lightweight containers (Docker instances with less than 50MB of memory) deployed on the terminal side perform the following three functions: data compression: video encoding, audio compression, and behavior log serialization; feature dimensionality reduction: PCA algorithm reduces the original feature vector from 1024 dimensions to 128 dimensions; local caching: temporarily stores data when the network is interrupted (up to 30 minutes), and resumes transmission after recovery; and outputs standardized data packets, including: session identifier, timestamp (high-precision synchronous clock), image feature vector, audio feature vector, interaction feature vector, and text feature vector.

[0115] In some embodiments, the feature extraction models for each student terminal are trained using a federated learning method.

[0116] Step 1: Initialize the global model

[0117] The initial feature extraction model is provided by the training server of the online training system (the "central node" in federated learning). Taking the text modality feature extraction model as an example, it is based on a simplified feature extractor of BERT-mini, which includes the basic network structure and randomly initialized parameters.

[0118] The training server distributes this initial model to all participating learner terminals ("client nodes" in federated learning). Each terminal receives the model and saves a copy locally as the starting point for training.

[0119] Step 2: Local model training (terminal side)

[0120] Data preparation: Each student terminal uses local private data (such as text information collected by the terminal) as training data. This data is always stored locally on the terminal and will not be uploaded to the server or other terminals.

[0121] Model fine-tuning: The terminal loads a local copy of the model and trains it using local data (such as updating model parameters through backpropagation to optimize feature extraction accuracy, for example, making the model extract relevant features from local speech more accurately).

[0122] Generate parameter updates: After training is completed, the terminal does not upload the original data, but only calculates the "parameter difference" between the local model and the initial model (i.e., the updated parameters minus the initial parameters), or directly exports the trained local model parameters.

[0123] Step 3: Parameter Aggregation (Server Side)

[0124] Collecting parameter updates: Each student terminal encrypts the locally calculated model parameters (or parameter differences) and sends them to the training server.

[0125] Global parameter optimization: The training server aggregates the parameters uploaded by all terminals (e.g., weighted average, assigning weights according to the amount of data from each terminal; the larger the amount of data from a terminal, the higher the proportion of its parameters in the global model) to generate a new "global model parameter".

[0126] Model update and validation: The training server uses a small amount of public validation data (such as general feature data prepared in advance by the system) to test the performance of the aggregated global model (such as feature extraction accuracy). If the preset indicators are not met, the next round of training will begin; if the indicators are met, training can be paused.

[0127] Step 4: Model Distribution and Iterative Training

[0128] Distribute the new global model: The server distributes the aggregated new global model parameters to all student terminals.

[0129] Repeated training process: Each terminal overwrites its local old model with the newly received global model, and trains again using local data (the training starting point at this time is the optimized global model, not the randomly initialized model), generates new parameters, updates and uploads them to the server.

[0130] Termination condition: When the global model performance (such as cross-terminal feature extraction consistency and accuracy) reaches the preset target, or the training rounds reach the upper limit, the iteration stops, and the final global model is the feature extraction model shared by all terminals.

[0131] The feature extraction models for each student terminal are trained through federated learning. This allows all terminals to collaboratively optimize the model without uploading local raw data, protecting data privacy while improving the model's adaptability to various types of student data. The server uses encryption or differential methods to collect local model parameter updates from each student terminal, reducing the risk of raw data leakage.

[0132] In some embodiments, gating attention can be used before or after step 130 to adjust the weights of feature vectors for different modalities. Through the gating mechanism, the importance of features of each modality (such as text, speech, image, interaction, etc.) is dynamically determined, and weights are assigned according to importance, so that the model pays more attention to key modal information.

[0133] The gating mechanism generates a "gating vector" through a small neural network (usually a fully connected layer) to represent the weight coefficients of each modality. For example, the gating vector gate = [g1, g2, g3, g4], where g1, g2, g3, and g4 represent the weights of the image / audio / interactive / text modalities, respectively. The feature vectors of the corresponding modalities are scaled using the weight coefficients in the gating vector to highlight the contributions of important modalities.

[0134] The Dynamic Gated Attention (DGCA) mechanism automatically adjusts the weights of different modalities. For example, when the camera is occluded, visual dependence is reduced, while the analysis weight of other data such as audio, interaction, and text is increased; in noisy environments, audio dependence is reduced, while the analysis weight of other data such as video, interaction, and text is increased. Thus, recognition accuracy can still be maintained even in complex environments.

[0135] In some embodiments, when a student's learning status meets preset prompting conditions, a prompting message about the learning status is sent to the student's terminal.

[0136] For example, if a student's concentration level is less than 30% for 10 minutes, a notification will be automatically sent to the student's device so that the student can adjust their learning status in time.

[0137] In some embodiments, group training risks can also be identified based on multiple fusion feature vectors corresponding to multiple trainees.

[0138] Multiple fused feature vectors from various trainees are input into the group risk prediction head to obtain the group training risk. The group risk prediction head, built based on a multi-classification task, processes the input through a fully connected layer and applies the softmax activation function on dimension 1, outputting probability distributions representing various types of group training risks. Group training risks include, but are not limited to, large-scale network service anomalies, group misunderstandings of knowledge points, and cheating.

[0139] For example, if student A is detected frequently switching screens and answering questions similarly to student B, and there is no voice communication, it is determined that students A and B may be a cheating group.

[0140] In the event of network service anomalies during group training, a network anomaly alarm message will be sent to the management terminal so that the administrator can handle the network failure in a timely manner. Alternatively, in the event of misunderstandings of knowledge points during group training, a prompt message for adjusting teaching content will be sent to the teacher's terminal so that the teacher can adjust the teaching content or methods in a timely manner. For example, if the group error rate is >60%, a prompt message will be sent to the teacher's terminal, and the teaching content will be automatically adjusted. Or, in the event of cheating during group training, a cheating warning message will be sent to both the teacher's terminal and the student's terminal so that the student can stop the inappropriate behavior in time and the teacher can supervise the student in a timely manner.

[0141] Therefore, during online training, risks associated with group training can be identified in a timely manner and reported to relevant parties for intervention or handling, thereby improving the effectiveness of online training.

[0142] This provides a complete post-processing workflow that transforms raw prediction results into meaningful, insightful, and actionable tips.

[0143] In some embodiments, for large-scale online training scenarios, an elastic distributed architecture and resource scheduling mechanism are adopted: a container scheduler (K8s Operator) based on reinforcement learning dynamically adjusts the number of computing containers (Pods) according to traffic prediction, and adopts Kafka streaming data pipeline + Kubernetes automatic scaling to elastically support a large number of concurrent students and maintain stable performance during large-scale online training.

[0144] (1) Traffic prediction stage: 30 minutes before the start of training, the reinforcement learning operator predicts future traffic by using historical data and real-time online users, such as "the number of concurrent users will reach 50,000 in 10 minutes, and the amount of live streaming and chat data will surge".

[0145] (2) Resource expansion in advance: Based on the predicted traffic, expansion is extracted. For example, the Operator sends instructions to K8s to expand the "live streaming Pod" from 20 to 50, and the "Kafka chat-consumer-Pod" is expanded to 25 in advance. At the same time, these Pods are scheduled to nodes with sufficient resources.

[0146] (3) High-concurrency data flow: Based on the expanded resources, high-concurrency data flow is achieved. For example, during peak training periods, 50,000 students send bullet comments and ask questions at the same time. The data is written to the topic_chat of Kafka in real time, and 25 chat-consumer-pods consume the data in parallel and quickly distribute it to the interactive system. Students can see the chat content without delay.

[0147] (4) Dynamic fine-tuning and rollback: For example, when the training enters the Q&A session, the amount of answer data increases. K8s HPA (Horizontal Pod Autoscaler) detects that the lag in topic_answer exceeds 8,000 and automatically expands the answer-consumer-Pod from 8 to 16. After the Q&A session ends, the lag decreases and HPA reduces the number of Pods to 6.

[0148] (5) After training: The reinforcement learning operator predicts that traffic will drop rapidly and automatically reduces the number of all business Pods. For example, the number of live streaming Pods is reduced to 5 and the number of consumer Pods is reduced to 3, freeing up more than 90% of server resources.

[0149] The disclosed solution can be widely applied to scenarios such as online education, remote examinations, and corporate training, promoting the transformation of online inspection technology from "manual spot checks" to "intelligent full-scale monitoring". The disclosed online training inspection solution is: (1) More accurate: Multimodal fusion improves the recognition accuracy and reduces the false alarm rate. (2) More real-time: End-to-end latency is <1 second, which can support tens of thousands of students concurrently. (3) More intelligent: Automatically handles large batches of alarms, reducing manual intervention. (4) More economical: Bandwidth costs and manpower requirements are greatly reduced. (5) More secure: Meets data protection requirements and reduces the risk of privacy leakage.

[0150] The online training and inspection scheme of this disclosure has the following advantages:

[0151] First, it achieves comprehensive and accurate multimodal data perception and fusion. By integrating and collecting real-time video footage (facial expressions, postures), audio information (speech, ambient sounds), operational behavior data (mouse, keyboard, interface interaction), and text interaction data (chat, answering questions, comments), it constructs a comprehensive ability to perceive student status, providing a rich data foundation for precise inspections. It effectively integrates these heterogeneous data and uncovers their inherent correlations.

[0152] Second, improve the efficiency and real-time performance of large-scale concurrent processing. Design an efficient distributed computing architecture and streaming data processing mechanism, optimize the collection, transmission, storage and analysis process of multimodal data, significantly reduce resource consumption, and ensure that the inspection system can still maintain high throughput and low latency operation in high-concurrency scenarios with a large number of students (such as tens of thousands or even hundreds of thousands) online at the same time, so as to achieve near real-time inspection feedback.

[0153] Third, enhance intelligent analysis and proactive early warning capabilities. Utilize machine learning (especially deep learning) models to conduct in-depth analysis of the fused multimodal data, accurately identifying learners' learning status (focus, confusion, distraction), interaction level, and potential difficulties.

[0154] Fourth, establish an intelligent anomaly detection and risk warning model. This model can automatically identify abnormal behavior in individual trainees (such as prolonged inactivity, persistent confused expressions, and frequent screen switching) and group risks (such as widespread lag, a surge in question error rates, and concentrated questioning in the discussion area), and proactively push warning information to administrators or instructors in real time. It provides data-driven insights into training effectiveness, automatically generating visual reports that showcase overall trainee participation, key knowledge points / difficulties, and bottlenecks in the training process, assisting in optimizing training content and format.

[0155] Fifth, it significantly improves the efficiency and objectivity of inspections. Through automated and intelligent multimodal sensing and analysis, it significantly reduces or even replaces inefficient manual inspections, allowing limited training and management resources to focus on key issues and intervention measures. At the same time, data-driven analysis results have higher objectivity and consistency, enhancing the fairness and credibility of inspections.

[0156] Sixth, optimize the training experience and effectiveness: The ultimate goal is to improve the learning experience and overall effectiveness of training by promptly identifying problems, making precise interventions, and continuously optimizing, thereby ensuring the quality and achievement of objectives in large-scale online training.

[0157] Figure 2 A schematic diagram illustrating an online training and inspection system according to some embodiments of this disclosure is shown. For example... Figure 2 As shown, the online training and inspection system 200 of this embodiment includes the following modules or units.

[0158] The information acquisition module 210 is configured to acquire image information, audio information, interactive information, and text information from the student's terminal.

[0159] The feature extraction module 220 is configured to extract features from image information, audio information, interactive information, and text information respectively, and obtain image feature vector, audio feature vector, interactive feature vector, and text feature vector.

[0160] The feature fusion module 230 is configured to perform cross-modal self-attention processing on image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain fused feature vectors.

[0161] The prediction module 240 is configured to predict the learning status of trainees based on the fused feature vector.

[0162] The information acquisition module 210 includes:

[0163] Image acquisition unit 211 is configured to: acquire the video stream from the student terminal side, extract keyframe images from the video stream; when the structural similarity between any first keyframe image and second keyframe image is greater than a preset value, acquire the first keyframe image and skip the acquisition of the second keyframe image; or,

[0164] Audio acquisition unit 212 is configured to: acquire the audio stream from the student terminal, distinguish between student speech and environmental noise in the audio stream; extract Mel-frequency cepstral coefficients, speech activity, and emotional tendency of the student speech; or,

[0165] The interactive data acquisition unit 213 is configured to: monitor student behavior information, including mouse movement trajectory, keyboard events, and interface operation sequences; and generate a student behavior summary vector based on the student's behavior information, including: operation entropy value, number of actions per unit time, and abnormal behavior matching degree; or...

[0166] The text acquisition unit 214 is configured to: crawl the chat room questions, answer system correct and incorrect records, and bullet screen interactive text of online training in real time; and use the BERT-mini model to perform semantic analysis on question difficulty classification, sentiment polarity detection, and knowledge point relevance based on the crawled information.

[0167] The feature extraction module 220 includes:

[0168] Image feature extraction unit 221 is configured to: process image information sequentially through convolutional layers, multi-level residual blocks, global average pooling, and fully connected layers to obtain image feature vectors; or,

[0169] The audio feature extraction unit 222 is configured to process audio information sequentially through an LSTM model and a fully connected layer to obtain an audio feature vector; or...

[0170] The interaction feature extraction unit 223 is configured to process the interaction information sequentially through the input layer, hidden layer, output layer, and normalization layer of the deep feature encoder to obtain the interaction feature vector; or,

[0171] The text feature extraction unit 224 is configured to process the text information sequentially through global average pooling, a linear layer, and a normalization layer to obtain the text feature vector.

[0172] Feature fusion module 230 is configured as follows:

[0173] The feature matrix is ​​obtained by concatenating the image feature vector, audio feature vector, interaction feature vector, and text feature vector.

[0174] By using the learnable query weight matrix, key weight matrix, and value weight matrix, linear transformations are performed on the feature matrix to obtain the query matrix, key matrix, and value matrix, respectively.

[0175] The dot product operation is performed using the transpose of the query matrix and the key matrix, and the result of the dot product operation is normalized to obtain the weight matrix, which is used to characterize the correlation strength between modes.

[0176] The value matrix is ​​weighted and summed using the weight matrix to obtain the fused feature vector, which is used to represent the fused information of cross-modal features.

[0177] Prediction module 240 is configured to predict the learning status of learners, including at least one of the following:

[0178] The fused feature vector is input into the attention prediction head to obtain the student's attention prediction result;

[0179] The fused feature vector is input into the sentiment state prediction head to obtain the sentiment state prediction result of the trainee;

[0180] The fused feature vector is input into the comprehension prediction head to obtain the prediction result of the student's comprehension level;

[0181] The fused feature vector is input into the abnormal behavior prediction head to obtain the abnormal behavior prediction results of the trainees;

[0182] The fused feature vector is input into the learning effect prediction head to obtain the learning effect prediction result of the learner.

[0183] The attention prediction head, built based on a regression task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and finally multiply the processing result by a score cap to output an attention score in the range of 0 to the score cap;

[0184] The sentiment state prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, apply the softmax activation function to calculate on dimension 1, and output a probability distribution representing multiple sentiment state types;

[0185] The understanding prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and output the understanding probability in the range of 0 to 1;

[0186] The abnormal behavior prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and output the abnormal probability in the range of 0 to 1;

[0187] The learning effect prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, apply the softmax activation function to calculate on dimension 1, and output a probability distribution representing multiple learning effect types.

[0188] Among them, the information acquisition module 210 and the feature extraction module 220 can be deployed on the terminal side, such as the student terminal, and the feature fusion module 230 and the prediction module 240 can be deployed on the server side, such as the training server.

[0189] The student terminal is configured to collect image information, audio information, interactive information, and text information from the student terminal side; extract features from the image information, audio information, interactive information, and text information respectively to obtain image feature vectors, audio feature vectors, interactive feature vectors, and text feature vectors, and upload them to the training server;

[0190] The training server is configured to perform cross-modal self-attention processing on image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain a fused feature vector; based on the fused feature vector, the learning status of trainees is predicted.

[0191] Federated learning module 250 is configured to train the feature extraction model for each student terminal using the federated learning method.

[0192] The gating weight module 260 is configured to use gating attention to adjust the weights of feature vectors of different modalities.

[0193] Group risk module 270 is configured to identify group training risks based on multiple fusion feature vectors corresponding to multiple trainees.

[0194] Processing module 280 is configured to send a network anomaly alarm message to the management terminal in the event of a network service anomaly during group training; or,

[0195] In cases where misunderstandings of knowledge points arise during group training, a prompt message to adjust the teaching content is sent to the teacher's terminal; or,

[0196] In the event of cheating during group training, a cheating warning message will be sent to both the teacher's and student's terminals; or...

[0197] When a student's learning status meets the preset prompt conditions, a prompt message about the learning status is sent to the student's terminal.

[0198] Figure 3 A schematic diagram illustrating an online training and inspection system according to some embodiments of this disclosure is shown. For example... Figure 3 As shown, the online training inspection system 300 of this embodiment includes: a memory 310 and a processor 320 coupled to the memory 310. The processor 320 is configured to execute the online training inspection method in any of the foregoing embodiments based on instructions stored in the memory 310.

[0199] The online training and inspection system 300 may also include input / output interfaces 330, network interfaces 340, and storage interfaces 350. These interfaces 330, 340, 350, as well as the memory 310 and processor 320, can be connected, for example, via a bus 360.

[0200] The memory 310 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, boot loader, and other programs.

[0201] The processor 320 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, or transistors, or other discrete hardware components.

[0202] The input / output interface 330 provides a connection interface for input / output devices such as monitors, mice, keyboards, and touchscreens. The network interface 340 provides a connection interface for various networked devices. The storage interface 350 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 360 can use any bus architecture from various bus structures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0203] The online training inspection system 300 can be deployed in a distributed manner on user terminals (student terminals or teacher terminals) and training servers. For details, please refer to the deployment method of the online training inspection system 200.

[0204] It should be noted that the collection, use, storage, sharing, and transfer of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations, and require notification to the user and obtaining the user's consent or authorization. Where applicable, user personal information has undergone de-identification and / or anonymization and / or encryption technical processing. For example, the invention does not store original data such as facial / voice data, but only retains de-identified features, which meets the requirements of personal information protection.

[0205] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more (non-transitory) computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, cloud storage, etc.) containing computer program code. A computer program product should be understood as a software product that primarily implements its solution through a computer program.

[0206] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0207] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

Claims

1. An online training inspection method, comprising: Collect image information, audio information, interactive information, and text information from the student's terminal. The interactive information includes the student's mouse movement trajectory, keyboard events, or interface operation sequence. The text information includes online training chat room questions, correct and incorrect records of the answer system, or bullet screen interactive text. Feature extraction is performed on image information, audio information, interactive information, and text information respectively to obtain image feature vector, audio feature vector, interactive feature vector, and text feature vector; Cross-modal self-attention processing is performed on image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain fused feature vectors; By utilizing a dynamic gating attention mechanism, the fused feature vector is dynamically modulated by adjusting the weights of feature vectors from different modalities to obtain the final fused feature vector. The group risk prediction head inputs multiple final fused feature vectors corresponding to multiple trainees and outputs probability distributions representing multiple types of group training risks to identify group training risks. The group risk prediction head is constructed based on a multi-classification task. The group training risks include large-scale network service anomalies, group misunderstanding of knowledge points, and cheating behavior. In the event of a large-scale network service anomaly during group training, a network anomaly alarm message is sent to the management terminal. Alternatively, in cases where there is a risk of misunderstanding of knowledge points during group training, a prompt message to adjust the teaching content can be sent to the teacher's terminal; or, in cases where cheating occurs during group training, a cheating warning message can be sent to both the teacher's terminal and the student's terminal.

2. The online training inspection method according to claim 1, wherein: The acquisition of image information from the student's terminal includes: acquiring the video stream from the student's terminal and extracting keyframe images from the video stream; when the structural similarity between any first keyframe image and a second keyframe image is greater than a preset value, acquiring the first keyframe image and skipping the acquisition of the second keyframe image; or... The acquisition of audio information from the student's terminal includes: acquiring the audio stream from the student's terminal, distinguishing between student speech and environmental noise in the audio stream; extracting Mel-frequency cepstral coefficients, speech activity, and emotional tendency from the student's speech; or, Collecting interactive information from the student's terminal includes: monitoring student behavior information, including mouse movement trajectories, keyboard events, and interface operation sequences; generating a student behavior summary vector based on the student's behavior information, including: operation entropy value, number of actions per unit time, and abnormal behavior matching degree; or, The text information collected from the trainees' terminals includes: real-time crawling of online training chat room questions, correct and incorrect records of the answer system, and interactive text in bullet comments; based on the crawled information, the BERT-mini model is used to perform semantic analysis on question difficulty classification, sentiment polarity detection, and knowledge point relevance.

3. The online training inspection method according to claim 1, wherein, Feature extraction is performed on image information, audio information, interactive information, and text information respectively to obtain image feature vectors, audio feature vectors, interactive feature vectors, and text feature vectors, including: Image information is processed sequentially through convolutional layers, multi-level residual blocks, global average pooling, and fully connected layers to obtain image feature vectors; or... The audio information is processed sequentially through an LSTM model and a fully connected layer to obtain the audio feature vector; or... The interaction information is processed sequentially through the input layer, hidden layer, output layer, and normalization layer of a deep feature encoder to obtain the interaction feature vector; or, The text information is processed sequentially through global average pooling, a linear layer, and a normalization layer to obtain the text feature vector.

4. The online training inspection method according to claim 1 further includes: Based on the final fused feature vector, the student's learning status is predicted.

5. The online training inspection method according to claim 4, wherein, Based on the fused feature vector, the predicted learning status of the learner includes at least one of the following: The fused feature vector is input into the attention prediction head to obtain the student's attention prediction result; The fused feature vector is input into the sentiment state prediction head to obtain the sentiment state prediction result of the trainee; The fused feature vector is input into the comprehension prediction head to obtain the prediction result of the student's comprehension level; The fused feature vector is input into the abnormal behavior prediction head to obtain the abnormal behavior prediction results of the trainees; The fused feature vector is input into the learning effect prediction head to obtain the learning effect prediction result of the learner.

6. The online training inspection method according to claim 5, wherein, The attention prediction head, built based on a regression task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and finally multiply the processing result by a score cap to output an attention score in the range of 0 to the score cap; The sentiment state prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, apply the softmax activation function to calculate on dimension 1, and output a probability distribution representing multiple sentiment state types; The understanding prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and output the understanding probability in the range of 0 to 1; The abnormal behavior prediction head, built based on a binary classification task, is configured to: process the fused feature vector first through a fully connected layer, then through a sigmoid activation function, and output an abnormal probability in the range of 0 to 1; The learning effect prediction head, built based on a multi-classification task, is configured to: process the fused feature vector through a fully connected layer, apply the softmax activation function to calculate on dimension 1, and output a probability distribution representing multiple learning effect types.

7. The online training inspection method according to claim 4, wherein: The student terminal is configured to collect image information, audio information, interactive information, and text information from the student terminal side; extract features from the image information, audio information, interactive information, and text information respectively to obtain image feature vectors, audio feature vectors, interactive feature vectors, and text feature vectors, and upload them to the training server; The training server is configured to perform cross-modal self-attention processing on image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain a fused feature vector; based on the fused feature vector, the learning status of trainees is predicted.

8. The online training inspection method according to claim 7, wherein, The feature extraction models for each student terminal are trained using a federated learning method.

9. The online training inspection method according to claim 1, wherein cross-modal self-attention processing is performed on image feature vectors, audio feature vectors, interaction feature vectors, and text feature vectors to obtain a fused feature vector, including: The feature matrix is ​​obtained by concatenating the image feature vector, audio feature vector, interaction feature vector, and text feature vector. Using learnable query weight matrix, key weight matrix, and value weight matrix, linear transformations are performed on the feature matrix to obtain the query matrix, key matrix, and value matrix, respectively. The dot product operation is performed on the transpose of the query matrix and the key matrix, and the result of the dot product operation is normalized to obtain the weight matrix, which is used to characterize the correlation strength between modes. The fused feature vector is obtained by weighting the value matrix with the weight matrix and summing the weights.

10. The online training inspection method according to claim 4 further includes: When a student's learning status meets the preset prompt conditions, a prompt message about the learning status is sent to the student's terminal.

11. An online training inspection system, comprising: One or more modules of the online training inspection method according to any one of claims 1-10.

12. An online training inspection system, comprising: Memory; And a processor coupled to the memory, the processor being configured to execute the online training inspection method of any one of claims 1-10 based on instructions stored in the memory.

13. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the online training inspection method according to any one of claims 1-10.

14. A computer program product comprising computer instructions that, when executed by a processor, implement the online training inspection method according to any one of claims 1-10.

Images