Multi-modal learning state evaluation method for online education

Abstract

The invention discloses an online education-oriented multi-modal learning state evaluation method. The method comprises the following steps of S1, acquiring multi-modal data; s2, preprocessing the multi-modal data; s3, multi-modal feature extraction is carried out; s4, dynamic weighted multi-modal feature fusion is carried out; s5, learning state classification evaluation; s6, evaluation result feedback and model optimization are carried out; according to the method, a multi-modal data acquisition mode is adopted, four types of modal data including behavior, physiology, environment and content interaction are integrated, compared with an existing single-modal evaluation method, the information dimension is more comprehensive, the limitation of single-modal data can be effectively avoided, and the comprehensiveness of an evaluation result is improved; the dynamic weight fusion model based on the attention mechanism is constructed, the weight can be dynamically adjusted according to the correlation between the modal data and the learning state evaluation target, and compared with a fixed weight fusion method, the fusion precision is higher, and the accuracy of the evaluation result is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of online education technology and multimodal data processing technology, specifically a method for assessing the state of multimodal learning in online education. Background Technology

[0002] Online education, with its advantages of flexibility in time and space, has become an important part of the education field. Learning status directly determines learning outcomes, and accurately assessing learners' learning status is a core prerequisite for achieving personalized teaching and improving the quality of online education. Existing methods for assessing online education learning status mostly rely on single-modal data, such as behavioral data like learning time and quiz accuracy, or physiological data like facial expressions.

[0003] Single-modal data suffers from limitations such as limited information dimensions and weak resistance to interference: behavioral data is easily affected by objective factors such as network environment and equipment failure, failing to accurately reflect the learner's subjective learning state; physiological data is easily affected by individual learner differences and environmental interference, resulting in insufficient stability of assessment results. Furthermore, existing methods generally lack deep integration mechanisms for multimodal data, making it difficult to achieve comprehensive and accurate assessment of learning status, and they have not established dynamically adaptable assessment models, failing to meet the personalized assessment needs of different learning scenarios and learners.

[0004] Therefore, in view of the shortcomings of existing technologies, there is an urgent need for a method that can integrate multimodal data and achieve accurate and personalized learning status assessment in order to solve the problem of inaccurate and incomplete learning status assessment in online education.

[0005] Based on this, a multimodal learning status assessment method for online education is designed. Summary of the Invention

[0006] In view of the above situation and to overcome the shortcomings of the prior art, the present invention provides a multimodal learning status assessment method for online education, which effectively solves the problems raised in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multimodal learning status assessment method for online education, comprising the following steps: Step S1: Multimodal data acquisition A multimodal data acquisition system was built to simultaneously collect four types of modal data during the online learning process: behavioral modal data, physiological modal data, environmental modal data, and content interaction modal data. Behavioral modality data is collected in real time through the log module of the online education platform. The collected content includes learning duration, video playback operation, homework submission time, answer interval duration, and number of interactive comments. The data format is uniformly converted to JSON format. Physiological modality data is collected through sensors and wearable devices integrated into the learner's terminal device. The collected data includes facial expression data, eye movement data, and physiological signal data. The acquisition frequency is 30Hz, and the data format is converted to CSV format. Environmental modal data is collected every 5 minutes via the terminal device's positioning module and environmental sensors. The collected data includes the learned geographical location, ambient light intensity, and ambient noise decibel value. The data format is converted to XML format. Content interaction modal data is collected in real time through the content interaction module of the online education platform. The collected content includes the number of times course content is clicked, the number of times key content is marked, the number of times questions are submitted, and the frequency of knowledge point review. The data format is uniformly converted to JSON format. Step S2: Multimodal data preprocessing The four types of modal data collected in step S1 are subjected to data cleaning, data standardization and data alignment processes respectively; Data cleaning employs a combination of outlier detection and missing value imputation. Outlier detection utilizes the quartile method, 3σ criterion, DBSCAN algorithm, and quartile method for different modalities of data, and identified outliers are directly deleted. Missing value imputation utilizes interpolation of adjacent time points, imputation using the average value of the same time period, linear regression, and zero-value imputation for different modalities of data. Data standardization employs the min-max standardization method, mapping various modal data to the [0,1] interval; Data alignment is based on timestamps, and all modal data are aligned to a 1-minute time granularity. Data collected more than 1 minute is processed using the mean aggregation method, and data collected less than 1 minute is supplemented using the linear interpolation method. Step S3: Multimodal Feature Extraction For the four types of modal data after preprocessing in step S2, the corresponding feature vectors are extracted respectively to construct a multimodal feature set; Behavioral modality feature extraction adopts a combination of statistical feature extraction and behavioral pattern feature extraction. The statistical features include the mean learning time, the variance of video playback speed, the standard deviation of the answer interval, and the frequency of interactive comments. The behavioral pattern features are extracted through a Hidden Markov Model (HMM), and the concatenation results in a 40-dimensional behavioral modality feature vector. Physiological modality features were extracted by using a convolutional neural network (CNN), statistical methods, and wavelet transform to extract facial expression features, eye movement features, and physiological signal features, which were then concatenated to obtain a 336-dimensional physiological modality feature vector. Environmental modal feature extraction employs a combination of statistical and trend features to obtain an 8-dimensional environmental modal feature vector; Content interaction modality feature extraction employs the TF-IDF algorithm combined with statistical feature extraction to obtain a 64-dimensional content interaction modality feature vector; Step S4: Dynamically weighted multimodal feature fusion A dynamic weighted fusion model based on the attention mechanism was constructed, and the weight coefficients of the behavioral modality, physiological modality, environmental modality, and content interaction modality were initialized to 0.3, 0.4, 0.1, and 0.2, respectively. The dynamic weights of each modality feature are calculated based on the attention mechanism. The feature vectors of the four modalities are input into the attention network and mapped to the same feature space through a fully connected layer. The correlation coefficient between each modality feature and the learning state evaluation target is calculated, and the modality weights are adjusted according to the correlation coefficient. A weighted summation method is used to perform feature fusion, resulting in a global fused feature vector; Step S5: Learning Status Classification Assessment A learning state evaluation model based on an improved random forest is constructed, which classifies learning states into four categories: focused state, general state, distracted state, and fatigued state. The improved random forest model uses decision trees as base classifiers and introduces the AdaBoost algorithm to integrate the base classifiers, adjusting their weights according to the classification accuracy of each base classifier. Input the global fusion feature vector into the improved random forest model, and output the learner's current learning state category and confidence level. The confidence level threshold is set to 0.7. If the confidence level is lower than 0.7, repeat steps S3 to S5. Step S6: Evaluation Result Feedback and Model Optimization The evaluation results are fed back to the teachers' and learners' ends of the online education platform in real time. The feedback includes the learning status category, the duration of the status, the key modal factors affecting the status, and optimization suggestions. Feedback data on the evaluation results are collected regularly. Combined with newly added multimodal data, incremental learning methods are used to update the parameters of the improved random forest model and the dynamic weighted fusion model. The update cycle is set to 7 days.

[0008] Preferably, the video playback operation in step S1 includes playing, pausing, speeding up, and dragging the progress bar; Facial expression data was collected through the terminal camera, and the coordinates of 68 facial feature points were extracted using a deep learning-based facial feature point extraction algorithm. Eye movement data is collected through the terminal camera combined with eye-tracking algorithms to obtain blink frequency, fixation duration, and fixation point coordinates; Physiological signal data include heart rate and skin conductivity.

[0009] Preferably, the min-max standardization formula in step S2 is x'=(x-min(x)) / (max(x)-min(x)), where x is the original data, min(x) is the minimum value of the data in this class, max(x) is the maximum value of the data in this class, and x' is the standardized data.

[0010] Preferably, the facial expression features in step S3 are extracted by a CNN model. The coordinates of 68 facial feature points are input into the CNN model, and a 256-dimensional facial expression feature vector is output. Eye movement features include mean blink frequency, variance of fixation duration, and fixation point distribution entropy, resulting in a 16-dimensional eye movement feature vector. Physiological signal features were obtained by extracting the mean and variance of wavelet coefficients at each of the three-level wavelet decomposition, resulting in a 64-dimensional physiological signal feature vector.

[0011] Preferably, the dynamic weight update formula in step four is ω_i = ω_i0 × (r_i / Σr_j), where ω_i is the dynamic weight of the i-th modality, ω_i0 is the initial weight of the i-th modality, r_i is the correlation coefficient between the i-th modality feature and the evaluation target, and j∈{1,2,3,4}; the feature fusion formula is F=Σ(ω_i × F_i), where F is the global fusion feature vector, and F_i is the i-th modality feature vector.

[0012] Preferably, in step S5, the number of decision trees in the improved random forest model is set to 100, the depth of each decision tree is set to 10, the training sample set is constructed using the bootstrap sampling method, and the information gain ratio is used to select the splitting feature when splitting at each decision tree node.

[0013] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention adopts a multimodal data acquisition method, integrating four types of modal data: behavioral, physiological, environmental, and content interaction. Compared with existing single-modal assessment methods, it provides more comprehensive information dimensions, effectively avoids the limitations of single-modal data, and improves the comprehensiveness of assessment results. 2. Construct a dynamic weighted fusion model based on attention mechanism, which can dynamically adjust the weights according to the correlation between each modality data and the learning state evaluation target. Compared with the fixed weight fusion method, the fusion accuracy is higher and the accuracy of the evaluation results is enhanced. 3. An improved random forest model is used for learning state classification, and the AdaBoost algorithm is combined with the base classifier to improve the classification accuracy and generalization ability of the model, which can accurately identify different types of learning states. 4. An evaluation result feedback and model optimization mechanism has been added. Through incremental learning, the model parameters are continuously updated, which can adapt to different learning scenarios and the personalized needs of different learners, thereby improving the adaptability and sustainability of the method. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0015] In the attached diagram: Figure 1 This is a schematic diagram of a multimodal learning status assessment method for online education according to the present invention; Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0017] Depend on Figure 1 This invention relates to a method for assessing the state of multimodal learning in online education, comprising the following steps: Step S1: Multimodal data acquisition A multimodal data acquisition system was built to simultaneously collect four types of modal data of learners during the online learning process: behavioral modal data, physiological modal data, environmental modal data, and content interaction modal data. Behavioral modality data is collected through the log module of the online education platform backend. The collected content includes learning duration, video playback operations (including play, pause, speed up, and drag the progress bar), homework submission time, answer interval duration, and number of interactive comments. The collection frequency is real-time, and the data format is uniformly converted to JSON format. Physiological modality data was collected through sensors integrated into the learner's terminal device. The collected data included facial expression data, eye movement data, and physiological signal data. Facial expression data was collected through the terminal camera, using a deep learning-based facial feature point extraction algorithm to extract the coordinates of 68 facial feature points. Eye movement data was collected through the terminal camera combined with an eye-tracking algorithm to obtain blink frequency, fixation duration, and fixation point coordinates. Physiological signal data was collected through a wearable device, including heart rate and skin conductivity, at a sampling frequency of 30Hz, and the data format was converted to CSV format. Environmental modal data is collected through the terminal device's positioning module and environmental sensors. The collected data includes the learned geographical location, ambient light intensity, and ambient noise decibel value. The data is collected every 5 minutes and the data format is converted to XML format. Content interaction modal data is collected through the content interaction module of the online education platform. The collected content includes the number of times course content is clicked, the number of times key content is marked, the number of times questions are submitted, and the frequency of knowledge point review. The collection frequency is real-time, and the data format is uniformly converted to JSON format. Step S2: Multimodal data preprocessing The four types of modal data collected in step S1 are subjected to data cleaning, data standardization and data alignment processes respectively; Data cleaning employs a combination of outlier detection and missing value imputation. Outlier detection targets behavioral modality data, physiological modality data, environmental modality data, and content interaction modality data, using the quartile method, 3σ criterion, DBSCAN algorithm, and quartile method respectively for outlier identification, with identified outliers being directly deleted. Missing value imputation uses corresponding imputation methods for different modalities: missing values ​​in behavioral modality data are imputed using interpolation of data from adjacent time points; missing values ​​in physiological modality data are imputed using the average value of data within the same time period; missing values ​​in environmental modality data are imputed using linear regression; and missing values ​​in content interaction modality data are imputed with 0 values. Data standardization adopts the min-max standardization method, which maps various modal data to the interval [0,1]. The standardization formula is x'=(x - min(x)) / (max(x) - min(x)), where x is the original data, min(x) is the minimum value of the data class, max(x) is the maximum value of the data class, and x' is the standardized data. Data alignment is based on timestamps, aligning all modal data to the same time granularity, which is set to 1 minute. For physiological modal data collected more than 1 minute, the mean aggregation method is used; for environmental modal data collected less than 1 minute, the linear interpolation method is used to supplement the data, ensuring that all types of modal data correspond one-to-one at the same time node. Step S3: Multimodal Feature Extraction For the four types of modal data after preprocessing in step S2, the corresponding feature vectors are extracted respectively to construct a multimodal feature set; Behavioral modality feature extraction: A combination of statistical feature extraction and behavioral pattern feature extraction is adopted. Statistical features include the mean learning time, the variance of video playback speed, the standard deviation of the answer interval, and the frequency of interactive comments. Behavioral pattern features are extracted through a Hidden Markov Model (HMM). Video playback operation, homework submission behavior, and answer behavior are used as observation sequences. The HMM model is trained to obtain a 32-dimensional behavioral pattern feature vector. Finally, the statistical features and behavioral pattern features are concatenated to obtain a 40-dimensional behavioral modality feature vector. Physiological modality feature extraction: Facial expression features were extracted using a convolutional neural network (CNN). The coordinates of 68 facial feature points were input into the CNN model, and a 256-dimensional facial expression feature vector was output. Eye movement features were extracted using statistical methods, including the mean blink frequency, the variance of fixation duration, and the fixation point distribution entropy, resulting in a 16-dimensional eye movement feature vector. Physiological signal features were extracted using wavelet transform. The heart rate and skin conductivity data were decomposed into three layers of wavelet transform, and the mean and variance of the wavelet coefficients at each layer were extracted to obtain a 64-dimensional physiological signal feature vector. The facial expression features, eye movement features, and physiological signal features were then concatenated to obtain a 336-dimensional physiological modality feature vector. Environmental modal feature extraction: A combination of statistical and trend features is used. The statistical features include the mean ambient light intensity, the variance of ambient noise in decibels, and the stability of geographical location (measured by the number of times geographical location changes within the collection period). The trend features are extracted by linear fitting to obtain the slope of ambient light intensity change and the slope of ambient noise in decibels change, resulting in an 8-dimensional environmental modal feature vector. Content interaction modality feature extraction: The TF-IDF algorithm combined with statistical feature extraction is used to map course content clicks, key markup, question submission, and knowledge point review behavior to the course knowledge point dimension. The interaction weight of each knowledge point is calculated by the TF-IDF algorithm, and combined with the mean number of interactions and the standard deviation of the interaction interval, a 64-dimensional content interaction modality feature vector is obtained. Step S4: Dynamically weighted multimodal feature fusion A dynamic weighted fusion model based on an attention mechanism is constructed to fuse the four modal feature vectors extracted in step three to obtain a global fusion feature vector. Initialize the weight coefficients of each modality feature. The initial weights for behavioral modality, physiological modality, environmental modality, and content interaction modality are set to 0.3, 0.4, 0.1, and 0.2, respectively. The dynamic weights of each modality feature are calculated based on the attention mechanism. The feature vectors of four modalities are input into the attention network, and the modal features are mapped to the same feature space through a fully connected layer. The correlation coefficient between each modal feature and the evaluation target of the learning state is calculated. The modal weights are adjusted according to the correlation coefficient. The higher the correlation coefficient, the larger the corresponding modal weight. The dynamic weight update formula is ω_i=ω_i0×(r_i / Σr_j), where ω_i is the dynamic weight of the i-th modality, ω_i0 is the initial weight of the i-th modality, r_i is the correlation coefficient between the i-th modal feature and the evaluation target, and j∈{1,2,3,4}. Feature fusion is performed using a weighted summation method, with the fusion formula being F = Σ(ω_i×F_i), where F is the global fused feature vector and F_i is the feature vector of the i-th modality. Step S5: Learning Status Classification Assessment A learning state evaluation model based on an improved random forest is constructed, and a global fusion feature vector is used to classify and evaluate the learner's learning state. Learning states are divided into four categories: focused state, general state, distracted state, and fatigued state, with each category corresponding to a specific characteristic threshold range. Improved construction of random forest model: Decision trees are used as base classifiers, with 100 decision trees and a depth of 10 for each decision tree. Bootstrap sampling method is used to construct training sample set. When splitting at each decision tree node, information gain ratio is used to select splitting features. AdaBoost algorithm is introduced to integrate the base classifiers. The weights of each base classifier are adjusted according to their classification accuracy. The higher the classification accuracy, the greater the weight of the base classifier. The global fusion feature vector obtained in step S4 is input into the improved random forest model. The model outputs the learner's current learning state category and the confidence score of that category. The confidence score threshold is set to 0.7. If the confidence score is lower than 0.7, steps S3 to S5 are repeated to update the feature extraction and fusion results based on the new data until the confidence score is higher than 0.7. Step S6: Evaluation Result Feedback and Model Optimization The learning status assessment results obtained in step S5 are fed back to the teacher and learner ends of the online education platform in real time. The feedback includes the learning status category, the duration of the status, the key modal factors affecting the status, and optimization suggestions. After receiving the assessment results, teachers can adjust their teaching pace and push personalized learning resources based on the results; after receiving the assessment results, learners can adjust their learning behaviors and improve their learning environment based on the optimization suggestions. A model optimization mechanism was established, and feedback data on the evaluation results were collected regularly, including teachers' suggestions for correcting the evaluation results and learners' acceptance of the evaluation results. Combined with newly added multimodal data, incremental learning methods were used to update the parameters of the improved random forest model and the dynamic weighted fusion model. The update cycle was set to 7 days to ensure that the evaluation accuracy of the model continued to improve.

[0018] Example 1 A multimodal learning status assessment method for online education, with the following specific implementation steps: Step S1: Multimodal data acquisition A multimodal data acquisition system was built, which includes an online education platform backend log module, a terminal camera, an eye-tracking module, wearable sensors, environmental sensors, a positioning module, and a content interaction module. One hundred online learners were selected as test subjects. Data collection plugins were installed on the learners' terminal devices (mobile phones and computers). Wearable devices used smart bracelets, and the data collection time was 2 hours per person. Behavioral modality data is collected in real time through the log module of the online education platform's backend. The collected data includes: learning duration (in seconds), video playback actions (play, pause, speed adjustment (0.5x-2x), progress bar dragging (accurate to the second)), assignment submission time (accurate to the minute), quiz interval duration (in seconds), and number of interactive comments (in times). The data is converted to JSON format, as shown in the example below: {"time_stamp":"2024-05-01 10:00:00", “learning_duration”:120, "video_operation":{"play":3,"pause":2,"speed":1.5,"drag":1}, "homework_submit_time":"2024-05-01 10:30:00", "answer_interval":45,"comment_count":2}; Physiological modality data were collected via a terminal camera and a smart bracelet. Facial expression data was extracted using a facial feature point extraction algorithm based on MTCNN, yielding coordinates of 68 facial feature points (unit: pixels). Eye movement data was tracked using OpenCV combined with the Dlib library to obtain blink frequency (unit: times / minute), fixation duration (unit: seconds), and fixation point coordinates (unit: pixels). Heart rate (unit: beats / minute) and skin conductivity (unit: μS) were collected via a smart bracelet at a frequency of 30Hz, and the data were converted to CSV format. Environmental modal data is collected through a terminal positioning module (GPS) and environmental sensors. It learns the geographical location (latitude and longitude), ambient light intensity (unit: lux), and ambient noise decibel value (unit: dB). The data is collected every 5 minutes and converted into XML format. Content interaction modal data is collected in real time through the content interaction module of the online education platform, including the number of times course content is clicked (unit: times), the number of times key content is marked (unit: times), the number of times questions are submitted (unit: times), and the frequency of knowledge point review (unit: times / hour). The data is converted into JSON format. Step S2: Multimodal data preprocessing Data cleaning: The quartile method is used to detect outliers in behavioral and content interaction modal data. For example, learning durations exceeding 120 minutes or falling below 1 minute are considered outliers and deleted directly. The 3σ criterion is used to detect outliers in physiological modal data. For example, heart rates exceeding 150 beats / minute or falling below 40 beats / minute are considered outliers and deleted directly. The DBSCAN algorithm is used to detect outliers in environmental modal data. Light intensity exceeding 1000 lux or falling below 50 lux, and noise levels exceeding 80 dB are considered outliers and deleted directly. Missing value imputation: Missing values ​​in behavioral modal data are imputed using interpolation between adjacent time points. For example, if the learning duration at a certain moment is missing, the average of the learning durations at the previous and next moments is used. Missing values ​​in physiological modal data are imputed using the average of the same time period (every 30 minutes is a time period). Missing values ​​in environmental modal data are imputed using linear regression. A linear regression model is built based on the collected light intensity and noise level data to predict missing values. Missing values ​​in content interaction modal data are filled with 0. Data standardization: The min-max standardization method is used to map all modal data to the [0,1] interval. For example, the original heart rate data range is 60-100 beats / minute. If a learner's heart rate is 80 beats / minute, the standardized value is (80-60) / (100-60)=0.5. Data alignment: Using a 1-minute time granularity, various modal data are uniformly aligned based on timestamps. Physiological modal data are collected at a frequency of 30Hz, with 1800 data points collected per minute. Mean aggregation is used to obtain data such as average heart rate and average skin conductivity per minute. Environmental modal data are collected every 5 minutes, and linear interpolation is used to supplement the light intensity and noise decibel values ​​per minute. Step S3: Multimodal Feature Extraction Behavioral modality feature extraction: The average learning time within 2 hours, the variance of video playback speed, the standard deviation of the interval between answering questions, and the frequency of interactive comments were statistically analyzed to obtain 8-dimensional statistical features; Video playback operations, homework submission behavior, and answering behavior were used as observation sequences to train an HMM model (with 5 hidden states and 8 observation states) to obtain 32-dimensional behavioral pattern features, which were then concatenated to obtain a 40-dimensional behavioral modality feature vector. Physiological modality feature extraction: The coordinates of 68 facial feature points are input into a CNN model (network structure: input layer-convolutional layer-pooling layer-fully connected layer-output layer, input layer dimension is 68×2, output layer dimension is 256), outputting a 256-dimensional facial expression feature vector; the mean blink frequency, variance of fixation duration, and fixation point distribution entropy are statistically analyzed over 2 hours to obtain a 16-dimensional eye movement feature vector; heart rate and skin conductivity data are subjected to 3-layer wavelet decomposition (wavelet basis is db4), and the mean and variance of wavelet coefficients of each layer are extracted to obtain a 64-dimensional physiological signal feature vector, which is then concatenated to obtain a 336-dimensional physiological modality feature vector; Environmental modal feature extraction: The mean ambient light intensity, variance of ambient noise decibels, and number of geographical location changes within 2 hours are statistically analyzed (a change of more than 100 meters is considered a change) to obtain 3D statistical features; the slope of ambient light intensity change and the slope of ambient noise decibel change are extracted by linear fitting to obtain 2D trend features, and the 8D environmental modal feature vector is obtained after concatenation. Content interaction modality feature extraction: The course knowledge points are divided into 100 dimensions, and the interaction weight of each knowledge point is calculated by the TF-IDF algorithm to obtain 60-dimensional weight features; the mean number of interactions and the standard deviation of the interaction interval within 2 hours are statistically analyzed to obtain 4-dimensional statistical features, which are then concatenated to obtain a 64-dimensional content interaction modality feature vector. Step S4: Dynamically weighted multimodal feature fusion The initial weights for the behavioral modality, physiological modality, environmental modality, and content interaction modality are 0.3, 0.4, 0.1, and 0.2, respectively. Four modality feature vectors are input into an attention network. Each modality feature is mapped to a 256-dimensional feature space through two fully connected layers (512 neurons in the first layer and 256 neurons in the second layer). The correlation coefficient between each modality feature and the learning state evaluation target is calculated. Assuming the correlation coefficient for behavioral modality is 0.3, for physiological modality it is 0.45, for environmental modality it is 0.05, and for content interaction modality it is 0.2, then the dynamic weight update is: Behavioral modality ω1 = 0.3 × (0.5 × ... =0.27, physiological modality ω2=0.4×(0.45 / (0.3+0.45+0.05+0.2))=0.36, environmental modality ω3=0.1×(0.05 / (0.3+0.45+0.05+0.2))=0.005, content interaction modality ω4=0.2×(0.2 / (0.3+0.45+0.05+0.2))=0.04; The features are fused using a weighted summation method: F = 0.27 × F1 + 0.36 × F2 + 0.005 × F3 + 0.04 × F4, where F1 is a 40-dimensional behavioral modality feature vector, F2 is a 336-dimensional physiological modality feature vector, F3 is an 8-dimensional environmental modality feature vector, and F4 is a 64-dimensional content interaction modality feature vector, resulting in a 256-dimensional global fused feature vector F. Step S5: Learning Status Classification Assessment Learning states are categorized into four types: focused, general, distracted, and fatigued. The thresholds for focused states are: physiological modality features ≥40% in the global fusion feature vector; interactive comment frequency ≥0.3 in behavioral modality features; and average knowledge point interaction weight ≥0.5 in content interaction modality features. The thresholds for general states are: physiological modality features 25%-40%; interactive comment frequency 0.1-0.3 in behavioral modality features; and average knowledge point interaction weight 0.3-0.5 in content interaction modality features. The thresholds for distracted states are: physiological modality features ≤25%; video dragging frequency ≥3 times / hour in behavioral modality features; and average knowledge point interaction weight ≤0.3 in content interaction modality features. The thresholds for fatigued states are: central rate variation coefficient ≥0.2 in physiological modality features; and blinking frequency ≤5 times / minute. An improved random forest model is constructed: the number of decision trees is set to 100, the depth of each decision tree is set to 10, and the bootstrap sampling method is used to extract 100 sample subsets from the training sample set. Each sample subset is used to train a decision tree. When splitting a decision tree node, the information gain ratio is used to select the splitting feature. The AdaBoost algorithm is introduced. Initially, the weights of each base classifier are equal. The weights are adjusted according to the classification accuracy. The higher the classification accuracy, the greater the weight. Finally, the classification result is obtained through weighted voting. Input the global fusion feature vector F into the improved random forest model. If the output result indicates a focused state with a confidence level of 0.85 (higher than 0.7), then it is determined that the learner is currently in a focused state. If the output confidence level is 0.65 (lower than 0.7), then add 30 minutes of multimodal data and repeat steps S3 to S5 until the confidence level is higher than 0.7. Step S6: Evaluation Result Feedback and Model Optimization The assessment results (focus state, duration 45 minutes, key influencing factors: high facial expression focus in physiological modality, frequent interaction in behavioral modality) are fed back to the teacher and learner in real time. The teacher pushes more challenging personalized learning resources based on the assessment results, and the learner maintains the current learning environment based on the optimization suggestions. Feedback data is collected regularly, including teachers' opinions on the assessment results (if teachers believe the assessment results are incorrect, they should mark the correct status and reasons), learners' acceptance of the assessment results (using a 5-point rating system), and combined with the newly added multimodal data of 100 test subjects (2 hours of data per person per week), the decision tree parameters of the random forest model and the attention network weights of the dynamic weighted fusion model are updated and improved using incremental learning methods, with an update cycle of 7 days. Tests have shown that this method has a higher accuracy rate in classifying learning states compared to existing single-modal assessment methods, enabling precise assessment of the learning states of online learners.

[0019] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0020] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for assessing the state of multimodal learning in online education, comprising the following steps: Step S1: Multimodal data acquisition A multimodal data acquisition system was built to simultaneously collect four types of modal data during the online learning process: behavioral modal data, physiological modal data, environmental modal data, and content interaction modal data. Behavioral modality data is collected in real time through the log module of the online education platform. The collected content includes learning duration, video playback operation, homework submission time, answer interval duration, and number of interactive comments. The data format is uniformly converted to JSON format. Physiological modality data is collected through sensors and wearable devices integrated into the learner's terminal device. The collected data includes facial expression data, eye movement data, and physiological signal data. The acquisition frequency is 30Hz, and the data format is converted to CSV format. Environmental modal data is collected every 5 minutes via the terminal device's positioning module and environmental sensors. The collected data includes the learned geographical location, ambient light intensity, and ambient noise decibel value. The data format is converted to XML format. Content interaction modal data is collected in real time through the content interaction module of the online education platform. The collected content includes the number of times course content is clicked, the number of times key content is marked, the number of times questions are submitted, and the frequency of knowledge point review. The data format is uniformly converted to JSON format. Step S2: Multimodal data preprocessing The four types of modal data collected in step S1 are subjected to data cleaning, data standardization and data alignment processes respectively; Data cleaning employs a combination of outlier detection and missing value imputation. Outlier detection utilizes the quartile method, 3σ criterion, DBSCAN algorithm, and quartile method for different modalities of data, and identified outliers are directly deleted. Missing value imputation utilizes interpolation of adjacent time points, imputation using the average value of the same time period, linear regression, and zero-value imputation for different modalities of data. Data standardization employs the min-max standardization method, mapping various modal data to the [0,1] interval; Data alignment is based on timestamps, and all modal data are aligned to a 1-minute time granularity. Data collected more than 1 minute is processed using the mean aggregation method, and data collected less than 1 minute is supplemented using the linear interpolation method. Step S3: Multimodal Feature Extraction For the four types of modal data after preprocessing in step S2, the corresponding feature vectors are extracted respectively to construct a multimodal feature set; Behavioral modality feature extraction adopts a combination of statistical feature extraction and behavioral pattern feature extraction. The statistical features include the mean learning time, the variance of video playback speed, the standard deviation of the answer interval, and the frequency of interactive comments. The behavioral pattern features are extracted through a Hidden Markov Model (HMM), and the concatenation results in a 40-dimensional behavioral modality feature vector. Physiological modality features were extracted by using a convolutional neural network (CNN), statistical methods, and wavelet transform to extract facial expression features, eye movement features, and physiological signal features, which were then concatenated to obtain a 336-dimensional physiological modality feature vector. Environmental modal feature extraction employs a combination of statistical and trend features to obtain an 8-dimensional environmental modal feature vector; Content interaction modality feature extraction employs the TF-IDF algorithm combined with statistical feature extraction to obtain a 64-dimensional content interaction modality feature vector; Step S4: Dynamically weighted multimodal feature fusion A dynamic weighted fusion model based on the attention mechanism was constructed, and the weight coefficients of the behavioral modality, physiological modality, environmental modality, and content interaction modality were initialized to 0.3, 0.4, 0.1, and 0.2, respectively. The dynamic weights of each modality feature are calculated based on the attention mechanism. The feature vectors of the four modalities are input into the attention network and mapped to the same feature space through a fully connected layer. The correlation coefficient between each modality feature and the learning state evaluation target is calculated, and the modality weights are adjusted according to the correlation coefficient. A weighted summation method is used to perform feature fusion, resulting in a global fused feature vector; Step S5: Learning Status Classification Assessment A learning state evaluation model based on an improved random forest is constructed, which classifies learning states into four categories: focused state, general state, distracted state, and fatigued state. The improved random forest model uses decision trees as base classifiers and introduces the AdaBoost algorithm to integrate the base classifiers, adjusting their weights according to the classification accuracy of each base classifier. Input the global fusion feature vector into the improved random forest model, and output the learner's current learning state category and confidence level. The confidence level threshold is set to 0.

7. If the confidence level is lower than 0.7, repeat steps S3 to S5. Step S6: Evaluation Result Feedback and Model Optimization The evaluation results are fed back to the teachers' and learners' ends of the online education platform in real time. The feedback includes the learning status category, the duration of the status, the key modal factors affecting the status, and optimization suggestions. Feedback data on the evaluation results are collected regularly. Combined with newly added multimodal data, incremental learning methods are used to update the parameters of the improved random forest model and the dynamic weighted fusion model. The update cycle is set to 7 days.

2. The method for assessing the state of multimodal learning in online education according to claim 1, characterized in that: The video playback operations described in step S1 include playing, pausing, speeding up, and dragging the progress bar; Facial expression data was collected through the terminal camera, and the coordinates of 68 facial feature points were extracted using a deep learning-based facial feature point extraction algorithm. Eye movement data is collected through the terminal camera combined with eye-tracking algorithms to obtain blink frequency, fixation duration, and fixation point coordinates; Physiological signal data include heart rate and skin conductivity.

3. The method for assessing the state of multimodal learning in online education according to claim 1, characterized in that: The min-max standardization formula mentioned in step S2 is x'=(x-min(x)) / (max(x)-min(x)), where x is the original data, min(x) is the minimum value of the data in this class, max(x) is the maximum value of the data in this class, and x' is the standardized data.

4. The method for assessing the state of multimodal learning in online education according to claim 1, characterized in that: The facial expression features described in step S3 are extracted using a CNN model. The coordinates of 68 facial feature points are input into the CNN model, and a 256-dimensional facial expression feature vector is output. Eye movement features include mean blink frequency, variance of fixation duration, and fixation point distribution entropy, resulting in a 16-dimensional eye movement feature vector. Physiological signal features were obtained by extracting the mean and variance of wavelet coefficients at each of the three-level wavelet decomposition, resulting in a 64-dimensional physiological signal feature vector.

5. The method for assessing the state of multimodal learning in online education according to claim 1, characterized in that: The dynamic weight update formula in step four is ω_i = ω_i0 × (r_i / Σr_j), where ω_i is the dynamic weight of the i-th modality, ω_i0 is the initial weight of the i-th modality, r_i is the correlation coefficient between the i-th modality feature and the evaluation target, and j∈{1,2,3,4}; the feature fusion formula is F=Σ(ω_i × F_i), where F is the global fusion feature vector, and F_i is the i-th modality feature vector.

6. The method for assessing the state of multimodal learning in online education according to claim 1, characterized in that: In step S5, the number of decision trees in the improved random forest model is set to 100, the depth of each decision tree is set to 10, the bootstrap sampling method is used to construct the training sample set, and the information gain ratio is used to select the splitting feature when splitting at each decision tree node.

Images