Intelligent learning behavior monitoring and abnormity early warning method and device

Abstract

The invention provides an intelligent learning behavior monitoring and abnormity early warning method and device, and relates to the technical field of education analysis, and the device architecture comprises a multi-modal data collection module, a behavior analysis module, a dynamic abnormity grading early warning module and a processing measure module. The multi-modal data acquisition unit integrates learning platform operation records, facial expression data and physiological data as well as environment and situation information collected by the situation awareness data acquisition unit, so that omnibearing and multi-dimensional data acquisition is realized; the long-time monitoring and personalized model construction unit uses a plurality of data mining algorithms and LSTM in deep learning to combine with an attention mechanism, accurately mines student behavior rules according to long-term data and constructs a personalized model, and can flexibly and accurately judge abnormal behaviors according to student individual conditions and learning situations. Graded early warning is provided for related personnel in time, deterioration of learning problems is effectively prevented, and timeliness and effectiveness of education intervention are improved.

Description

Technical Field

[0001] The present invention relates to the field of educational analysis technology, and in particular to an intelligent learning behavior monitoring and abnormal early warning method and device. Background Art

[0002] According to a student teaching and learning behavior analysis and evaluation method, system and storage medium disclosed in China with the publication number "CN118246638A", the steps are as follows: S1. Attendance rate detection; S2. Attendance rate analysis; S3. Student interest detection; S4 Student interest analysis; S5. Learning progress monitoring; S6. Learning efficiency analysis; S7 Learning behavior analysis. The present invention obtains the attendance rate of students in each monitoring period according to the attendance data of each student in each monitoring period, obtains the learning interest coefficient of students in each monitoring period according to the learning interest data of each student in class in each monitoring period, obtains the learning efficiency of students in each monitoring period according to the learning progress data of each student in each monitoring period, and then comprehensively analyzes to obtain the learning behavior evaluation index of students in each monitoring period, which can help schools and teachers better optimize teaching management, discover student problems in time, take effective measures, and improve teaching quality and student learning motivation.

[0003] According to the Chinese publication number "CN104463746B", a method for monitoring student learning behavior is disclosed, which is used to monitor the learning behavior of students in the classroom, including the number of times each student is in a standing position, a normal sitting position for listening to a class, a sitting position in a writing state, and a sitting position lying on the desk in the classroom; and the learning behavior is automatically analyzed according to the number of times the above postures appear to obtain an evaluation result, and then the evaluation result of each student is sent to the parents through a wireless network, so that the school and parents can communicate better and improve parents' understanding of the students' performance during class. The present invention also provides a home-school communication system using the method for monitoring student learning behavior, which has a simple structure, is easy to implement, and has low cost. It only needs to be simply modified on the original monitoring equipment and can be put into use, and can realize real-time monitoring of the learning behavior of students in the classroom.

[0004] The above patent documents and prior art have the following technical problems when used:

[0005] Problem 1: In the above documents and existing technologies, existing technologies often rely on only a single or limited data source to monitor students' learning behavior. A single platform parameter record makes it difficult to determine whether students operate the gear ring because of difficulty in understanding knowledge or because of interference from the external environment or their own physiological discomfort, thus affecting the subsequent accurate analysis and effective intervention of learning behavior;

[0006] Problem 2: In the above documents and in the prior art, the monitoring of learning behavior is mostly directly analyzed according to the learning time, which fails to generate a targeted learning behavior model based on the students' learning habits and fails to take into account the individual differences of students, resulting in serious homogeneity of learning and lack of targeting;

[0007] Problem 3: Traditional warning mechanisms for abnormal learning behaviors mostly use fixed thresholds, which do not fully consider the individual learning progress of students, differences in course difficulty, and dynamic changes in behavior data. This makes the warning system prone to false positives or false negatives, and cannot identify truly abnormal behaviors in a timely and accurate manner.

[0008] Problem four: When faced with abnormal student learning behaviors, existing technologies often lack cross-disciplinary comprehensive considerations and personalized customization in their treatment strategies. They usually only provide general suggestions based on single-disciplinary knowledge or general experience, and fail to deeply combine multidisciplinary theories such as psychology, education, and neuroscience to analyze the root causes of the problem and develop targeted solutions. As a result, the treatment measures are ineffective, making it difficult to fundamentally solve students’ learning difficulties and effectively improve students’ learning motivation and learning outcomes. Summary of the invention

[0009] Technical issues solved

[0010] In view of the shortcomings of the prior art, the present invention provides an intelligent learning behavior monitoring and abnormal warning method and device, which solves the following problems:

[0011] 1. The single and one-sided nature of data collection leads to inaccurate analysis of learning behavior;

[0012] 2. The analysis of targeted learning behaviors cannot meet the personalized and targeted requirements of students’ individual differences;

[0013] 3. The problem of lack of flexibility and accuracy in abnormal warning;

[0014] 4. There is a lack of systematic and targeted strategies for dealing with user learning behaviors.

[0015] Technical Solution

[0016] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent learning behavior monitoring and abnormal warning method and device, the device architecture includes a multimodal data acquisition module, a behavior analysis module, a dynamic abnormal classification warning module and a processing measure module, wherein:

[0017] The multimodal data acquisition module is used to collect multimodal data and situational awareness data in the student's learning process, including a multimodal data acquisition unit and a situational awareness data acquisition unit. The multimodal data acquisition unit uses a variety of data acquisition and processing algorithms to obtain students' operation records, facial expression data and physiological data on the learning platform. The situational awareness data acquisition unit is used to collect environmental factor data and situational information and associate them with other data;

[0018] The behavior analysis module is used to deeply analyze the collected data to build an accurate behavior analysis model, including a behavior learning trajectory and micro-feature extraction unit and a long-term monitoring and personalized model construction unit. The behavior learning trajectory and micro-feature extraction unit is used to extract short-term and micro-level behavior features from multimodal data and construct a learning trajectory. The long-term monitoring and personalized model construction unit is used to mine student behavior patterns based on long-term data and build a personalized model;

[0019] The dynamic abnormality graded warning module is used to flexibly set and implement a dynamic abnormality graded warning mechanism, including a dynamic threshold setting unit and a graded warning strategy unit. The dynamic threshold setting unit is used to dynamically adjust the abnormality judgment threshold of multimodal data according to the student's learning progress and course situation. The graded warning strategy unit is used to divide the grades according to the severity and impact of the abnormality and transmit different levels of warning information to relevant personnel;

[0020] The processing measure module is used to generate processing strategies based on interdisciplinary knowledge and display them in various ways, including a processing strategy generation unit based on interdisciplinary knowledge and a multimodal display unit of processing methods. The processing strategy generation unit based on interdisciplinary knowledge is used to generate processing suggestions based on psychology, education and neuroscience knowledge, and the multimodal display unit of processing methods is used to convey processing suggestions to students through various forms such as voice, vision, hearing and situational simulation.

[0021] Preferably, the learning platform operation record acquisition algorithm in the multimodal data acquisition unit specifically extracts operation records including click type, detailed path of browsing page, specific steps and time of answering questions from multiple related tables of the learning platform database through customized SQL query statements, and cleans these data at the same time to remove outliers caused by system failures or network problems, and then efficiently stores them in the form of a hash table according to student ID and time order, so as to comprehensively and accurately obtain students' operation records on the learning platform.

[0022] Preferably, when the facial expression data in the multimodal data acquisition unit is collected, the face detection algorithm in the Dlib library is first used to locate the face area of ​​the image stream obtained by the camera, and then the multi-classification model based on the convolutional neural network CNN is used to classify the expression features of the face area, so as to obtain accurate facial expression data, and the physiological data is processed by a fast Fourier transform algorithm based on peak detection, bandpass filtering and spectrum analysis. The physiological data collection and processing algorithm in the multimodal data acquisition unit, for heart rate data, first uses a bandpass filtering algorithm to remove high-frequency and low-frequency noise caused by factors such as motion artifacts, and then accurately identifies the heart rate peak through an adaptive peak detection algorithm, and then calculates the heart rate value. For brain wave data, a wavelet transform algorithm is used for filtering and feature extraction, and the brain wave signal is decomposed into different frequency bands, and the energy characteristics of each frequency band are calculated. At the same time, a sliding window averaging algorithm is used to reduce signal fluctuations, so as to obtain stable and meaningful physiological data.

[0023] Preferably, the situational awareness data acquisition unit uses environmental sensors to collect data such as noise and light intensity in the learning environment. The noise data is calculated by performing analog-to-digital conversion on the analog signal collected by the sensor to calculate the average noise intensity. The light intensity data is converted into illuminance values ​​based on the light sensor readings. At the same time, the course schedule information is queried from the database of the course management system, and the course type and time information are associated with the corresponding data. For noise data, a network composed of multiple distributed noise sensors is used, each sensor collects environmental noise signals at a certain sampling frequency, and then the data of multiple sensors are fused through a distributed data fusion algorithm to reduce measurement errors and finally calculate the average noise intensity. For light intensity data, a light sensor with an automatic calibration function is used. The sensor readings are compared for calibration under known light intensities, and the light intensity values ​​are converted into standard illuminance units (lux). At the same time, the spatial layout information of the sensor is combined to accurately collect environmental factor data.

[0024] Preferably, when fusing multimodal data in the behavior learning trajectory and micro-feature extraction unit, an interpolation-based time synchronization algorithm is used to solve the timestamp alignment problem of different types of data, so as to ensure the consistency of different modal data in the time dimension; during the fusion process, an adaptive weighted average algorithm is used to dynamically adjust the weight according to the importance of each modal data in reflecting the learning behavior. The weight is obtained by analyzing a large amount of annotated data and training with a machine learning algorithm, and is used to more accurately extract short-term and micro-level behavioral features from multimodal data and construct a learning trajectory.

[0025] Preferably, the data mining algorithm in the long-term monitoring and personalized model building unit, in cluster analysis, adopts a combination of density-based spatial clustering algorithm (DBSCAN) and hierarchical clustering algorithm to cluster students' long-term learning behavior data, which can effectively identify different types of learning behavior patterns. In association rule mining, the FP-Growth algorithm is used to mine frequent item sets and association rules to discover hidden relationships between different behavioral characteristics. When building a personalized model, the long short-term memory network (LSTM) in deep learning is used. Its ability to process time series data can better capture the changes in student behavior over time. At the same time, combined with the attention mechanism, the model can focus on behavioral characteristics that are more critical to judging students' learning status, so as to more accurately mine student behavior patterns based on long-term data and build personalized models.

[0026] Preferably, a polynomial regression model is used in the threshold adjustment in the dynamic threshold setting unit, with students' learning progress (such as course completion ratio, knowledge point mastery), course difficulty (assessed through the knowledge point hierarchy and exercise difficulty in the course outline), and statistical characteristics of recent behavioral data (such as operation stability, average positivity of facial expressions), etc. as independent variables, and the threshold adjustment amount is used as the dependent variable for training. When using fuzzy logic, fuzzy sets and fuzzy rules are defined. For example, learning progress is divided into slow, medium, and fast fuzzy sets, and course difficulty is divided into low, medium, and high fuzzy sets. Fuzzy rules are determined based on expert experience and experimental data to more reasonably dynamically adjust the abnormal judgment threshold of multimodal data according to students' learning progress and course conditions.

[0027] Preferably, when evaluating the abnormal level in the hierarchical warning strategy unit, the exponentially weighted moving average algorithm is used to calculate the degree of continuous impact based on the duration of the abnormal behavior in the time dimension. For the severity of the abnormality, the degree of abnormality of the multimodal data is comprehensively considered, and the multimodal data is subjected to dimensionality reduction processing by the principal component analysis (PCA) algorithm to calculate the comprehensive abnormality score. According to the potential impact of the abnormal behavior on the learning effect, combined with the impact assessment scale formulated by educational experts, a quantitative assessment is performed through a fuzzy comprehensive evaluation algorithm. Finally, according to these quantitative results and the preset grade division threshold, the decision tree algorithm is used to divide the abnormal level, and different levels of warning information are delivered to relevant personnel in an asynchronous communication manner through integrated message push services, mail servers, SMS gateways and other communication interfaces.

[0028] Preferably, when generating processing suggestions based on psychological knowledge, the processing strategy generation unit based on interdisciplinary knowledge adopts a Bayesian network reasoning algorithm, takes students' behavioral performance and psychological state as nodes, determines the conditional probability relationship between nodes by constructing a psychological model and learning a large amount of psychological experimental data, and infers possible psychological problems and corresponding processing strategies. When based on educational knowledge, based on the educational knowledge map, a graph search algorithm is used to find educational theories and practical cases related to current students' learning problems, and generate teaching method adjustment suggestions. When based on neuroscience knowledge, a neural network simulation model is constructed to simulate the brain's neural activity in the learning process, and processing suggestions for physiological state adjustment are generated according to the model output and neuroscience research results. The voice interaction algorithm in the multimodal display unit of the processing method adopts natural Language generation technology generates natural and fluent voice text that is suitable for students' age and language habits based on the content of the processing suggestions, and then converts the text into speech through a high-quality speech synthesis engine (such as iFlytek's personalized speech synthesis technology); the visual display algorithm uses enhanced data visualization technology, such as interactive visual charts and 3D model displays, to enable students to understand the processing suggestions more clearly; the auditory assistance algorithm uses an intelligent audio generation algorithm to generate a personalized audio environment (such as white noise with different rhythms, background music related to the learning content) according to different processing scenarios; the scenario simulation algorithm uses advanced virtual reality (VR) and augmented reality (AR) development platforms (such as Unity3D combined with SteamVR) to create a highly realistic and interactive scenario simulation environment, which is used to better convey the processing suggestions to students in various forms.

[0029] Preferably, the operation method of the device is as follows:

[0030] Sp1: The multimodal data acquisition module operates. The multimodal data acquisition unit uses the corresponding algorithm to obtain students' operation records, facial expression data and physiological data on the learning platform. The situational awareness data acquisition unit collects environmental factor data and situational information and associates them with other data;

[0031] Sp2: The behavior analysis module processes the collected data. The behavior learning trajectory and micro-feature extraction unit extracts short-term and micro-level behavior features from multimodal data and constructs learning trajectories. The long-term monitoring and personalized model construction unit mines students' behavior patterns based on long-term data and constructs personalized models.

[0032] Sp3: The dynamic abnormality graded warning module is started. The dynamic threshold setting unit dynamically adjusts the abnormality judgment threshold of multimodal data according to the student's learning progress and course situation. The graded warning strategy unit divides the grades according to the severity and impact of the abnormality and transmits different levels of warning information to relevant personnel;

[0033] Sp4: The processing measures module works. The processing strategy generation unit based on interdisciplinary knowledge generates processing suggestions based on the knowledge of psychology, education and neuroscience. The multimodal display unit of processing methods conveys processing suggestions to students through various forms such as voice, vision, hearing and situational simulation.

[0034] Beneficial Effects

[0035] The present invention provides an intelligent learning behavior monitoring and abnormal warning method and device. It has the following beneficial effects:

[0036] 1. The present invention integrates the learning platform operation records, facial expression data and physiological data through a multimodal data acquisition unit, as well as the environment and situational information collected by a situational awareness data acquisition unit, thereby realizing all-round and multi-dimensional data acquisition. When collecting facial expression data, the advanced Dlib library face detection algorithm is combined with the CNN expression classification model to accurately capture students' instantaneous emotional changes. Physiological data collection uses bandpass filtering, adaptive peak detection, wavelet transform and other algorithms to process heart rate and brain wave data, which can deeply understand the impact of students' physiological state on learning behavior and realize accurate characterization and analysis of users' learning behavior.

[0037] 2. The present invention adopts the behavioral learning trajectory and micro-feature extraction unit, adopts the interpolation-based time synchronization algorithm and the adaptive weighted average algorithm to ensure the consistency of multimodal data in the time dimension and reasonably weighted fusion, effectively mining the short-term and micro-level behavioral characteristics. The long-term monitoring and personalized model construction unit uses a variety of data mining algorithms and LSTM in deep learning combined with the attention mechanism to accurately mine the student behavior patterns based on long-term data and construct a personalized model, so as to extremely accurately characterize the student's learning behavior, and provide a highly reliable data basis for subsequent early warning and intervention.

[0038] 3. The present invention adopts a dynamic threshold setting unit that adopts a polynomial regression model combined with fuzzy logic to comprehensively consider various factors such as students' learning progress, course difficulty and recent behavioral data, such as course completion ratio, knowledge point mastery, operation stability, expression positivity, etc., to dynamically adjust the abnormal judgment threshold of multimodal data, thereby overcoming the limitations of traditional fixed thresholds. The graded warning strategy unit calculates the influence of the duration of abnormal behavior through an exponentially weighted moving average algorithm, calculates the comprehensive abnormality score after reducing the dimension of multimodal data using the principal component analysis algorithm, and quantifies the potential impact of abnormalities on learning effects by combining the impact assessment scale formulated by education experts with the fuzzy comprehensive evaluation algorithm. Finally, the decision tree algorithm is used to divide the abnormality level and asynchronously transmit the warning information through multiple communication interfaces. This method can flexibly and accurately judge abnormal behaviors according to individual students' situations and learning situations, and provide graded warnings to relevant personnel in a timely manner, so that teachers and parents can intervene in advance, effectively prevent the deterioration of learning problems, and improve the timeliness and effectiveness of educational intervention.

[0039] 4. The present invention adopts a processing strategy generation unit based on interdisciplinary knowledge, and adopts a Bayesian network reasoning algorithm when based on psychological knowledge. It constructs a psychological model based on students' behavioral performance and psychological state and determines the node relationship in combination with a large amount of psychological experimental data, so as to accurately infer psychological problems and give targeted processing strategies, which is helpful to solve the root causes of abnormal learning behavior from a psychological level. It uses a graph search algorithm based on educational knowledge graphs based on pedagogy knowledge to find relevant theories and cases, and can provide appropriate teaching method adjustment suggestions according to students' specific learning problems and optimize the teaching process. It can improve students' learning status from a physiological perspective by constructing a neural network simulation model based on neuroscience knowledge and combining research results to generate physiological state adjustment suggestions. For example, for students whose learning efficiency decreases due to stress, they can be given psychological relaxation guidance, adjust the difficulty and progress in teaching, and provide physiological suggestions for relieving fatigue, so as to solve students' learning problems in an all-round and personalized way, improve students' learning experience and learning effects, and realize the integration of multiple disciplines and personalized adaptability to solve learning problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a diagram of the device architecture of the present invention;

[0041] Figure 2 A diagram showing the operation method of the device of the present invention;

[0042] Figure 3 It is a data trend diagram of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment one:

[0045] like Figure 1-3 As shown, an intelligent learning behavior monitoring and abnormal warning method and device, the device architecture includes a multimodal data acquisition module, a behavior analysis module, a dynamic abnormal classification warning module and a processing measure module, wherein:

[0046] The multimodal data acquisition module is used to collect multimodal data and situational awareness data in the learning process of students, including a multimodal data acquisition unit and a situational awareness data acquisition unit. The multimodal data acquisition unit uses a variety of data acquisition and processing algorithms to obtain students' operation records, facial expression data and physiological data on the learning platform. The situational awareness data acquisition unit is used to collect environmental factor data and situational information and associate them with other data. The multimodal data acquisition unit integrates the learning platform operation records, facial expression data and physiological data, as well as the environmental and situational information collected by the situational awareness data acquisition unit, to achieve all-round and multi-dimensional data acquisition. For example, when collecting facial expression data, the advanced Dlib library face detection algorithm is combined with the CNN expression classification model to accurately capture students' instantaneous emotional changes. Physiological data collection uses bandpass filtering, adaptive peak detection, wavelet transform and other algorithms to process heart rate and brain wave data, which can deeply understand the impact of students' physiological state on learning behavior.

[0047] The behavior analysis module is used to deeply analyze the collected data to build an accurate behavior analysis model, including a behavior learning trajectory and micro-feature extraction unit and a long-term monitoring and personalized model construction unit. The behavior learning trajectory and micro-feature extraction unit is used to extract short-term and micro-level behavior features from multimodal data and construct a learning trajectory. The long-term monitoring and personalized model construction unit is used to mine student behavior patterns based on long-term data and build a personalized model. The behavior learning trajectory and micro-feature extraction unit uses an interpolation-based time synchronization algorithm and an adaptive weighted average algorithm to ensure the consistency of multimodal data in the time dimension and reasonably weighted fusion, effectively mining short-term and micro-level behavior features. The long-term monitoring and personalized model construction unit uses a variety of data mining algorithms and LSTM in deep learning combined with the attention mechanism to accurately mine student behavior patterns based on long-term data and build a personalized model, so that it can extremely accurately characterize students' learning behaviors, providing a highly reliable data foundation for subsequent early warning and intervention;

[0048] The dynamic abnormality classification warning module is used to flexibly set up and implement the dynamic abnormality classification warning mechanism, including a dynamic threshold setting unit and a classification warning strategy unit. The dynamic threshold setting unit is used to dynamically adjust the abnormal judgment threshold of multimodal data according to the student's learning progress and course situation. The classification warning strategy unit is used to classify the levels according to the severity and impact of the abnormality and transmit different levels of warning information to relevant personnel. The dynamic threshold setting unit in the dynamic abnormality classification warning module adopts a polynomial regression model combined with fuzzy logic to comprehensively consider multiple factors such as student learning progress, course difficulty and recent behavioral data, such as course completion ratio, knowledge point mastery, operation stability, expression positivity, etc., to dynamically adjust the multimodal data. The abnormal judgment threshold overcomes the limitations of the traditional fixed threshold. The graded warning strategy unit calculates the impact of the duration of abnormal behavior through the exponentially weighted moving average algorithm, and calculates the comprehensive abnormality score after reducing the dimension of multimodal data using the principal component analysis algorithm. It combines the impact assessment scale developed by educational experts and the fuzzy comprehensive evaluation algorithm to quantify the potential impact of abnormalities on learning outcomes. Finally, it uses the decision tree algorithm to divide the abnormality level and asynchronously transmits warning information through multiple communication interfaces. This method can flexibly and accurately judge abnormal behaviors according to individual students' situations and learning situations, and provide graded warnings to relevant personnel in a timely manner, so that teachers and parents can intervene in advance, effectively prevent the deterioration of learning problems, and improve the timeliness and effectiveness of educational intervention.

[0049] The treatment measures module is used to generate treatment strategies based on interdisciplinary knowledge and display them in various ways, including a treatment strategy generation unit based on interdisciplinary knowledge and a multimodal display unit of treatment methods. The treatment strategy generation unit based on interdisciplinary knowledge is used to generate treatment suggestions based on psychology, education and neuroscience knowledge. The multimodal display unit of treatment methods is used to convey treatment suggestions to students through voice, vision, hearing and scenario simulation. When based on psychological knowledge, the Bayesian network reasoning algorithm is used to build a psychological model based on students' behavioral performance and psychological state, and determine the node relationship in combination with a large amount of psychological experimental data, so as to accurately infer psychological problems and provide targeted treatment. Strategies help to solve the root causes of abnormal learning behaviors from a psychological level. Based on pedagogical knowledge, graph search algorithms are used to find relevant theories and cases based on educational knowledge graphs. Appropriate suggestions for adjusting teaching methods can be provided according to students' specific learning problems to optimize the teaching process. Based on neuroscience knowledge, a neural network simulation model is constructed and combined with research results to generate physiological state adjustment suggestions, which can improve students' learning status from a physiological perspective. For example, for students whose learning efficiency decreases due to stress, they can be given psychological relaxation guidance, the difficulty and progress can be adjusted in teaching, and suggestions for relieving fatigue can be provided physiologically. This can solve students' learning problems in an all-round and personalized manner and improve their learning experience and learning outcomes. Specific embodiment 2:

[0051] like Figure 1-3 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0052] The device operates as follows:

[0053] Sp1: The multimodal data acquisition module operates. The multimodal data acquisition unit uses the corresponding algorithm to obtain students' operation records, facial expression data and physiological data on the learning platform. The situational awareness data acquisition unit collects environmental factor data and situational information and associates them with other data;

[0054] Sp2: The behavior analysis module processes the collected data. The behavior learning trajectory and micro-feature extraction unit extracts short-term and micro-level behavior features from multimodal data and constructs learning trajectories. The long-term monitoring and personalized model construction unit mines students' behavior patterns based on long-term data and constructs personalized models.

[0055] Sp3: The dynamic abnormality graded warning module is started. The dynamic threshold setting unit dynamically adjusts the abnormality judgment threshold of multimodal data according to the student's learning progress and course situation. The graded warning strategy unit divides the grades according to the severity and impact of the abnormality and transmits different levels of warning information to relevant personnel;

[0056] Sp4: The processing measures module works. The processing strategy generation unit based on interdisciplinary knowledge generates processing suggestions based on the knowledge of psychology, education and neuroscience. The multimodal display unit of processing methods conveys processing suggestions to students through various forms such as voice, vision, hearing and situational simulation. Specific embodiment three:

[0058] like Figure 1-3 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0059] The learning platform operation record acquisition algorithm in the multimodal data acquisition unit specifically extracts operation records including click type, detailed path of browsing page, specific steps and time of answering questions from multiple related tables of the learning platform database through customized SQL query statements, and cleans these data at the same time to remove abnormal values ​​caused by system failure or network problems, and then stores them efficiently in the form of hash tables according to student ID and time sequence, so as to comprehensively and accurately obtain students' operation records on the learning platform. When collecting facial expression data in the multimodal data acquisition unit, the face detection algorithm in the Dlib library is first used to locate the face area of ​​the image stream obtained by the camera, and then the multi-classification model based on the convolutional neural network CNN is used. The facial expression features of the human face area are classified to obtain accurate facial expression data, and the physiological data are processed by a fast Fourier transform algorithm based on peak detection, bandpass filtering and spectrum analysis. The physiological data acquisition and processing algorithm in the multimodal data acquisition unit, for heart rate data, first uses a bandpass filtering algorithm to remove high-frequency and low-frequency noise caused by motion artifacts and other factors, and then uses an adaptive peak detection algorithm to accurately identify the heart rate peak, and then calculates the heart rate value. For brain wave data, the wavelet transform algorithm is used for filtering and feature extraction, and the brain wave signal is decomposed into different frequency bands. The energy characteristics of each frequency band are calculated, and the sliding window averaging algorithm is used to reduce signal fluctuations to obtain stable and meaningful physiological data.

[0060] Environmental sensors are used in the situational awareness data collection unit to collect data such as noise and light intensity in the learning environment. The noise data is calculated by performing analog-to-digital conversion on the analog signal collected by the sensor to calculate the average noise intensity. The light intensity data is converted into illuminance values ​​based on the light sensor readings. At the same time, the course schedule information is queried from the database of the course management system, and the course type and time information are associated with the corresponding data. For noise data, a network composed of multiple distributed noise sensors is used. Each sensor collects environmental noise signals at a certain sampling frequency, and then the data of multiple sensors are fused through a distributed data fusion algorithm to reduce measurement errors and finally calculate the average noise intensity. For light intensity data, a light sensor with automatic calibration function is used. The sensor readings under known light intensities are compared for calibration, and the light intensity values ​​are converted into standard illuminance units (lux). At the same time, the spatial layout information of the sensor is combined to accurately collect environmental factor data.

[0061] When fusing multimodal data in the behavioral learning trajectory and micro-feature extraction unit, an interpolation-based time synchronization algorithm is used to align the timestamps of different types of data to ensure the consistency of different modal data in the time dimension; in the fusion process, an adaptive weighted average algorithm is used to dynamically adjust the weight according to the importance of each modal data in reflecting the learning behavior. The weight is obtained by analyzing a large amount of labeled data and training the machine learning algorithm, and is used to more accurately extract short-term and micro-level behavioral features from multimodal data and construct a learning trajectory. Based on the multimodal data of time series, the student's learning trajectory is constructed, and different types of data are integrated on the time axis to form a multidimensional learning trajectory that includes operational behavior, facial expression changes, physiological state, and environmental context. For example, a learning trajectory can be represented as a student studying a mathematics course (course type) at a certain point in time, and frequently clicking on exercises (operation) in dim light (light intensity) and a quiet environment (noise level). Records), while the expression shows confusion (facial expression), and the heart rate is slightly increased (physiological data). The data fusion algorithm is used to align and fuse data from different sources according to timestamps, and the weighted average is used to fuse data of different modes to ensure the consistency and integrity of the data. Micro features are extracted from the learning trajectory, including short-term behavioral change patterns (such as sudden changes in students' operation frequency during the explanation of a certain knowledge point), instantaneous fluctuations in expression (such as frowning when hearing a difficult problem), and slight changes in physiological signals (such as a brief increase in heart rate). These micro features can be extracted by sliding the time window. For example, a time window of 5 minutes is used to analyze the feature changes in each window. For the behavioral change pattern, the statistical features of the operation records in the time window are calculated, such as the variance and change rate of the number of operations. For the micro features of expression and physiological signals, signal processing and feature extraction algorithms are used, such as calculating the Euclidean distance change of the expression feature vector and the differential features of the physiological signal.

[0062] The data mining algorithm in the long-term monitoring and personalized model construction unit uses a combination of density-based spatial clustering algorithm (DBSCAN) and hierarchical clustering algorithm to cluster students' long-term learning behavior data in cluster analysis, which can effectively identify different types of learning behavior patterns. In association rule mining, the FP-Growth algorithm is used to mine frequent item sets and association rules to discover hidden relationships between different behavioral characteristics. When building personalized models, the long short-term memory network (LSTM) in deep learning is used. Its ability to process time series data can better capture the changes in student behavior over time. At the same time, combined with the attention mechanism, the model can focus on behavioral features that are more critical to judging students' learning status, which can be used to more accurately mine students based on long-term data. In the long-term monitoring and personalized model building unit, students' learning behavior data over a long period of time (such as one semester or one academic year) are continuously collected to establish students' long-term learning behavior patterns. Through the analysis of a large amount of data, students' behavioral patterns at different learning stages can be discovered. For example, some students have specific operating habits or physiological reaction changes before an exam. Based on the long-term analysis results, a personalized learning behavior model is built for each student. This model takes into account individual differences in students, including learning styles, habits, abilities and other factors. For example, for a student who learns quickly but is prone to carelessness, the model will focus on the relationship between his or her answering speed and error types; for a student who studies more carefully, the model will pay more attention to changes in his or her thinking time and operating frequency.

[0063] The threshold adjustment in the dynamic threshold setting unit adopts a polynomial regression model, with students' learning progress (such as the proportion of course completion, the degree of knowledge points mastered), course difficulty (assessed through the knowledge point hierarchy and exercise difficulty in the course syllabus), and statistical characteristics of recent behavioral data (such as the stability of operations, the average positive degree of facial expressions) as independent variables, and the threshold adjustment amount as the dependent variable for training. When using fuzzy logic, fuzzy sets and fuzzy rules are defined, such as dividing learning progress into slow, medium, and fast fuzzy sets, and dividing course difficulty into low, medium, and high fuzzy sets. Fuzzy rules are determined based on expert experience and experimental data to more reasonably adjust the abnormal judgment threshold of multimodal data according to students' learning progress and course conditions. For example, as students accumulate knowledge in a certain subject, their normal answering speed range may increase, and the system will automatically update the corresponding threshold. For the case of increased course difficulty, a certain degree of behavioral fluctuation range is allowed to expand, such as a decrease in operation frequency or an increase in expression confusion time is considered normal within a certain range, and the threshold of multimodal data is comprehensively considered to form a comprehensive abnormal judgment standard. Data from different modalities have different sensitivities to anomalies. For example, an abnormal increase in heart rate in physiological data may reflect a more serious problem than a slight change in facial expression. However, when the facial expression continues to show extreme confusion and the operation records are abnormal, it should also be considered an abnormality.

[0064] When evaluating the abnormal level in the graded warning strategy unit, the exponentially weighted moving average algorithm is used to calculate the degree of continuous impact based on the duration of the abnormal behavior in the time dimension. For the severity of the abnormality, the degree of abnormality of the multimodal data is comprehensively considered. After the multimodal data is reduced in dimension by the principal component analysis (PCA) algorithm, the comprehensive abnormality score is calculated. According to the potential impact of the abnormal behavior on the learning effect, combined with the impact evaluation scale formulated by education experts, a quantitative evaluation is performed through the fuzzy comprehensive evaluation algorithm. Finally, based on these quantitative results and the preset grade division threshold, the decision tree algorithm is used to divide the abnormal level. Through the integrated message push service, mail server, SMS gateway and other communication interfaces, different levels of warning information are transmitted to relevant personnel in an asynchronous communication manner. According to the severity, duration and potential impact of the abnormal behavior on the learning effect, the warning is divided into different levels. For example, a mild level warning may be a warning that students are A mild lack of concentration occurs within a short period of time (such as occasional dazed expression, but basically normal operation). A moderate-level warning means that the student has obvious learning difficulties (such as a significant decrease in answering speed and a confused expression that lasts for a long time). A high-level warning means that the student's learning behavior is seriously abnormal and may affect the entire learning process (such as long-term cessation of operation, physiological data showing high tension or fatigue, and extremely negative expression). According to different warning levels, warning information is delivered to relevant personnel (students, teachers, parents) in different ways. For mild-level warnings, a prompt message can be displayed on the student's learning interface; moderate-level warnings can notify teachers at the same time and display a detailed behavior analysis report in the teacher's teaching management system; in addition to notifying teachers and parents, high-level warnings can also provide emergency intervention suggestions, such as suggesting that parents pay attention to students' mental state and teachers provide one-on-one tutoring to students.

[0065] When generating treatment suggestions based on psychological knowledge, the treatment strategy generation unit based on interdisciplinary knowledge makes corresponding treatment suggestions for different abnormal behaviors according to the principles of psychology. For example, from the perspective of motivation theory, if it is found that students lack learning motivation and lead to abnormal behavior, an incentive mechanism can be designed, such as setting personalized reward goals, providing positive feedback, etc. From the perspective of emotion regulation theory, if students have emotional problems due to excessive stress that affect their learning, relaxation training methods (such as deep breathing exercises, meditation, etc.) can be recommended. Based on educational theory, the teaching method or learning content arrangement can be adjusted. For example, according to constructivist learning theory, if students have difficulty understanding a certain knowledge, For students who have difficulty in understanding certain points (discovered through behavioral analysis), more cases and practical application scenarios can be provided to help them build a knowledge system. Based on the teaching evaluation theory, for students with poor learning effects, the evaluation method can be optimized, such as increasing the proportion of formative evaluation, providing timely feedback on students' learning progress, and using the research results of neuroscience to provide physiological support for treatment measures. For example, if the EEG data shows that the student's brain is in a state of fatigue during the learning process, it is recommended to take a proper rest or adjust the study time schedule to match the brain's physiological rhythm. According to the principle of neuroplasticity, targeted training programs can be designed for students with learning difficulties to promote brain neural circuits. To optimize the learning path, we use the Bayesian network reasoning algorithm, take the students' behavior and psychological state as nodes, determine the conditional probability relationship between nodes by building a psychological model and learning a large amount of psychological experimental data, and infer possible psychological problems and corresponding treatment strategies. When based on educational knowledge, we use the graph search algorithm to find educational theories and practical cases related to current students' learning problems based on the educational knowledge graph, and generate suggestions for adjusting teaching methods. When based on neuroscience knowledge, we build a neural network simulation model to simulate the brain's neural activity in the learning process, and generate treatments for physiological state adjustment based on the model output and neuroscience research results. The voice interaction algorithm in the multimodal display unit of treatment suggestions and treatment methods adopts natural language generation technology to generate natural and fluent voice text that meets the age and language habits of students based on the content of treatment suggestions, and then converts the text into speech through a high-quality speech synthesis engine (such as iFlytek's personalized speech synthesis technology). The visual display algorithm uses enhanced data visualization technology, such as interactive visualization charts and 3D model displays, to enable students to understand the treatment suggestions more clearly; the auditory assistance algorithm uses an intelligent audio generation algorithm to generate personalized audio environments (such as white noise of different rhythms and background music related to the learning content) according to different treatment scenarios;The scenario simulation algorithm uses advanced virtual reality (VR) and augmented reality (AR) development platforms (such as Unity3D combined with SteamVR) to create a highly realistic and interactive scenario simulation environment, which is used to better convey processing suggestions to students in various forms. Through speech synthesis technology, processing suggestions are conveyed to students in the form of voice. For example, when it is detected that students are not paying attention, the system can use a gentle voice prompt "You seem to be a little distracted. Do you want to take a break before continuing?" This voice interaction method can give timely reminders without interfering with students' learning. Visual display, the processing measures are displayed in a visual way on the learning interface. For example, the changes in students' learning behavior can be displayed in the form of charts. Trends and suggested improvement directions, or through animation demonstrations of relaxation training steps, knowledge explanation cases, etc., visual displays can make students more intuitively understand the content of the treatment suggestions. Auditory assistance, in addition to voice interaction, uses specific audio signals to assist in the communication of treatment measures. For example, playing soothing background music to help students relax, or playing some audio that helps improve concentration when they need to concentrate (such as white noise, natural sounds, etc.). Scenario simulation, creating a scenario simulation environment for students, allowing students to practice treatment suggestions in simulated scenarios. For example, if the treatment suggestion is about improving social skills (in a collaborative learning scenario), a virtual group discussion scenario can be created for students to practice communication and collaboration skills. ; Specific embodiment four:

[0067] like Figure 1-3 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0068] In the above device architecture, the algorithm content of each module is as follows:

[0069] The algorithm formula of the multimodal data acquisition module is as follows:

[0070] Learning platform operation record acquisition algorithm: extract data from the database through customized SQL query statements, and clean and efficiently store it;

[0071] Data cleaning formula:

[0072] D clean ={d i ∣d i ∈D,P(d i )≥T}

[0073] Among them, D clean is the cleaned data set, D is the original data set, P(d i ) is the data integrity and consistency scoring function, T is the cleaning threshold;

[0074] Hash table storage formula:

[0075] H op ={(h(ID i ), R i )|i∈[1,n]}

[0076] Among them, H op Indicates the operation record stored in the hash table, h(ID i ) represents the unique identifier hash value of student i, R i Indicates the corresponding operation record set;

[0077] The operating logic is as follows:

[0078] Data extraction: Use SQL query to extract key fields of the data table;

[0079] Data cleaning: remove incomplete or outlier data and retain key behavior records;

[0080] Data storage: Use hash tables to optimize data access speed;

[0081] Through the above algorithm, the efficiency of obtaining operation records can be improved, data redundancy can be reduced, and comprehensive data support for students' learning behavior can be provided.

[0082] Facial expression data collection algorithm, the formula is as follows:

[0083] Face detection formula (Dlib library)

[0084] F loc = {r i ∣r i =(x i ,y i , w i ,h i ), i∈[1,N]}

[0085] Among them, is the detected face area, is the rectangular box connected to each person (position x i ,y i , width and height w i ,h i );

[0086] Expression classification formula CNN model:

[0087] P expr =softmax(W·X+b)

[0088] Among them, P expr is the probability distribution of classification, X is the pixel matrix of the face area, W is the weight matrix, and b is the bias term;

[0089] By locating the face area from the image stream, outputting the rectangular frame, and using the CNN model to extract and classify expression features, facial expressions can be efficiently recognized to provide accurate data for behavioral analysis.

[0090] Physiological data collection algorithm, the formula is as follows:

[0091] Heart rate detection formula:

[0092]

[0093] Among them, HR: heart rate (beats / minute), Peaks is the number of peaks in the signal, and Duration is the acquisition time (seconds);

[0094] Brain wave feature extraction formula:

[0095]

[0096] Among them, W(x)(a, b) is the wavelet coefficient of the signal at scale a and time b, ψ is the mother wavelet function, and t is time.

[0097] By performing peak detection or wavelet decomposition after bandpass filtering and denoising, key features such as heart rate and brain wave frequency band energy are extracted to provide stable and meaningful physiological data.

[0098] The algorithm formula in the behavior analysis module is as follows:

[0099] The time synchronization formula of the time synchronization and feature extraction algorithm is as follows:

[0100] T sync (i) = T i +δT n

[0101] Among them, T sync (i) is the timestamp after synchronization, T i is the original timestamp, δT i The time offset calculated for the interpolation;

[0102] The feature fusion formula is as follows:

[0103]

[0104] Among them, F is the fused feature vector, f i is the eigenvector of mode i, w i is the weight of mode i;

[0105] The interpolation algorithm is used to ensure the temporal consistency of different data modes, and the weighted average algorithm is used to adjust the weights according to the importance, so as to efficiently fuse multimodal data and improve the accuracy of behavioral feature extraction.

[0106] The personalized model construction algorithm formula is as follows:

[0107] LSTM Network:

[0108] Input Gate:

[0109] i t =σ(W xi x t +W hi h t-1 +b i )

[0110] Forget Gate:

[0111] f t =v(W xf x t +W hf h t-1 +b f )

[0112] Output Gate:

[0113] o t =σ(W xo x t +W ho h t-1 +b o )

[0114] Attention weight calculation formula:

[0115]

[0116] Among them, α t is the attention weight at time t, e t Score for attention;

[0117] Use LSTM to capture the time series rules of behavior, and combine it with the attention mechanism to enhance the focus on key features and improve the model's ability to model personalized behaviors.

[0118] The algorithm formula of the dynamic abnormal classification warning module is as follows:

[0119] Dynamic threshold setting algorithm

[0120] Dynamic threshold calculation formula (polynomial regression):

[0121] T dynamic =β0+β1x1+β2x2+…+β n x n

[0122] Among them, T dynamic is the abnormal threshold after dynamic adjustment, x1, x2, ..., xn are influencing factors (such as learning progress, course difficulty, etc.), β0 is the intercept of the regression model, β1, β2, …, β n is the weight coefficient of the regression model;

[0123] Fuzzy logic rules:

[0124]

[0125] Among them, μ C (y) is the membership degree of the resulting fuzzy set, μ A (x), μ B (z) is the membership degree of the input fuzzy set, max and min are the synthesis rules of fuzzy reasoning;

[0126] The independent variable x is generated according to the learning progress (such as completion rate), course difficulty (knowledge point level), etc. The basic threshold is dynamically adjusted according to the regression result, and combined with fuzzy rule optimization, the threshold adapts to individual changes, enhancing the accuracy and sensitivity of the system in detecting abnormal behavior.

[0127] Hierarchical warning strategy algorithm

[0128] Abnormal level calculation formula (principal component analysis PCA):

[0129] Z=W·X

[0130] Among them, Z is the comprehensive abnormal feature after dimensionality reduction, W is the principal component matrix, and X is the original multimodal abnormal feature;

[0131] Exponentially Weighted Moving Average (EWMA):

[0132] S t =αX t +(1-α)S t-1

[0133] Among them, S t is the weighted average value at time t, X t is the current abnormal behavior value, α is the smoothing coefficient;

[0134] Fuzzy comprehensive evaluation:

[0135] R=B·W

[0136] Among them, R is the comprehensive evaluation score, B is the single factor evaluation matrix, and W is the weight matrix;

[0137] PCA dimensionality reduction is used to process multimodal features, calculate comprehensive anomaly scores, comprehensive EWMA scores and fuzzy comprehensive evaluation scores, divide levels through decision trees, and push warning messages to relevant personnel according to different levels, so as to achieve flexible warning level division and help to quickly take appropriate intervention measures.

[0138] The algorithm formula of the interdisciplinary knowledge processing strategy generation algorithm is as follows:

[0139] Bayesian reasoning formula (psychology knowledge):

[0140]

[0141] Where P(H|E) is the posterior probability that hypothesis H is true given evidence E, P(E|H) is the probability that evidence E appears when hypothesis H is true, P(H) is the prior probability that hypothesis H is true, and P(E) is the probability that evidence E appears;

[0142] Education knowledge graph search formula:

[0143]

[0144] Among them, Path is the optimal path, W i is the weight of the knowledge point, d i is the distance between nodes in the path;

[0145] Based on students' behavioral and psychological data, potential psychological problems are predicted through Bayesian networks, and targeted teaching strategies are found using educational knowledge graphs. Psychological intervention and teaching adjustment suggestions are provided to help students optimize their learning status.

[0146] Multimodal display algorithm for processing methods

[0147] Voice interaction generation formula (natural language generation):

[0148]

[0149] in, is the probability of the generated sentence, W i is the i-th word;

[0150] Data visualization mapping formula:

[0151] V=f(D)

[0152] Among them, V is the visualization chart, D is the original processing suggestion data, and f is the mapping function from data to visualization elements;

[0153] Use NLP to generate natural sentences and generate sound output through the voice engine. Generate charts or 3D model displays based on the suggested content, and enhance students' acceptance and understanding of processing suggestions through multiple sensory forms. Specific embodiment five:

[0155] like Figure 1-3 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0156] In the above device architecture, the data flow between the modules is as follows:

[0157] Data input layer (collection stage)

[0158] Multimodal data acquisition module

[0159] Learning platform operation records: Use customized SQL query statements to extract operation records (such as click type, browsing path, answering steps, etc.) from the learning platform database, remove outliers, and use hash tables for efficient storage according to student ID and time order.

[0160] Facial expression data: Image stream preprocessing: Use the Dlib library to detect the face area, and use the CNN model to extract expression features and classify them.

[0161] Physiological data: Remove motion artifacts from heart rate data, adaptively detect heart rate peaks, and calculate heart rate based on peaks; EEG data: Use wavelet transform to filter and extract features, and use the sliding window averaging algorithm to smooth the signal and obtain stable features.

[0162] Context-aware data: Environmental factors: a multi-sensor network collects signals and calculates the average noise intensity through data fusion; light intensity: calibrates sensor data and converts it into standard lux units; course context: extracts course type and time information from the course management system and associates it with environmental data.

[0163] Data processing layer (analysis phase)

[0164] Behavior Analysis Module

[0165] Behavior learning trajectory and micro-feature extraction unit: Data alignment: time synchronization algorithm based on interpolation; Data fusion: using adaptive weighted average algorithm to dynamically adjust weights according to modal importance; Behavior feature extraction: extract short-term and micro-level features from multimodal data to construct behavior trajectories.

[0166] Long-term monitoring and personalized model building unit: Cluster analysis: combining DBSCAN and hierarchical clustering to identify long-term learning behavior patterns; Association rule mining: using the FP-Growth algorithm to discover hidden relationships between behavioral features; Personalized model: combining LSTM with the attention mechanism to mine long-term behavioral patterns.

[0167] Abnormal assessment layer (early warning stage)

[0168] Dynamic abnormal classification warning module

[0169] Dynamic threshold setting unit: Polynomial regression: Dynamically adjust the abnormal threshold according to learning progress, course difficulty and recent behavioral characteristics; Fuzzy logic: Based on fuzzy sets and rules, realize reasonable threshold adjustment.

[0170] Hierarchical warning strategy unit: Abnormal level assessment: calculate the abnormal score through PCA dimensionality reduction, and evaluate the severity in combination with the impact assessment scale; Warning information transmission: divide the levels based on the decision tree, and transmit the warning information through message push, email and SMS.

[0171] Decision and output layer (processing stage)

[0172] Treatment Module

[0173] Processing strategy generation unit based on interdisciplinary knowledge: Psychology: Bayesian network reasoning, generating psychological problems and suggestions; Education: educational knowledge graph search, providing teaching adjustment suggestions; Neuroscience: neural network simulation of learning process, and making suggestions for physiological state adjustment.

[0174] Multimodal display unit of processing methods: Voice interaction: natural language generation + speech synthesis technology (such as iFlytek); Visual display: interactive visualization charts and 3D model display; Auditory assistance: intelligent audio generates a personalized learning environment; Scenario simulation: virtual scenario simulation based on VR / AR technology.

[0175] Summary of data processing flow

[0176] Acquisition phase: collect learning, behavior, and environmental data from multiple sensors and platforms;

[0177] Preprocessing stage: data cleaning, filtering, feature extraction and alignment;

[0178] Analysis phase: extract short-term features and build long-term personalized models;

[0179] Abnormal assessment: dynamically adjust the threshold and classify abnormal levels;

[0180] Decision-making stage: combining interdisciplinary knowledge to generate multimodal processing strategies;

[0181] Output stage: Communicate suggestions to students and implement early warnings through multimodal presentations. Specific embodiment six:

[0183] like Figure 1-3 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0184] In the above device architecture, in actual use, the hardware structure used is as follows:

[0185] The core hardware equipment for learning platform operation record collection of multimodal data collection module:

[0186] Server: deploy the learning platform database (it is recommended to use a high-performance database server, such as Dell PowerEdge series);

[0187] Data interface: High-speed Ethernet interface, used to communicate with the server;

[0188] Storage hardware: Solid-state drives (NVMe SSDs) that support efficient querying to store processed operation record data;

[0189] Embedded control unit: Customized processors (such as the NXPi.MX series) are used to run SQL query scripts and data cleansing in real time.

[0190] Core hardware equipment for facial expression data collection:

[0191] HD camera: supports 1080p or higher resolution, frame rate not less than 30FPS (such as Logitech C920);

[0192] Processor: Dedicated graphics processing unit (GPU), such as NVIDIA Jetson Nano, for real-time face detection and expression classification;

[0193] Infrared camera (optional): used to capture expressions in low-light or nighttime environments.

[0194] Core hardware equipment for physiological data collection:

[0195] Heart rate sensor: high-precision photoelectric volume pulse wave sensor (such as MAX30102);

[0196] EEG acquisition headband: Use a multi-channel EEG acquisition device (such as MuseS or EmotivInsight), and the sampling frequency is recommended to be ≥256Hz;

[0197] Data processing unit: An embedded microcontroller (such as ESP32 or STM32) is used for signal preprocessing (filtering and peak detection).

[0198] Core hardware equipment for context-aware data collection:

[0199] Environmental Sensors:

[0200] Noise sensor: MEMS microphone array (such as INMP441), supporting multi-point distributed deployment;

[0201] Light sensor: Automatically calibrate light sensors (such as TSL2591) with a measurement accuracy of 0.1 lux;

[0202] Sensor network nodes: LoRa modules (such as SX1276) are used for low-power, long-distance data transmission;

[0203] Central control device: The core gateway device (such as Raspberry Pi or industrial-grade edge computing device) is used for data fusion.

[0204] Core hardware equipment for behavior learning trajectory and micro-feature extraction of behavior analysis module:

[0205] High-performance computing nodes: NVIDIA Jetson Xavier or AWS Snowball Edge, supporting edge AI computing;

[0206] Storage device: RAID array storage system, used to store large amounts of behavioral data;

[0207] Time synchronization hardware: GPS-based time synchronization module to ensure the consistency of multi-modal data timestamps;

[0208] Core hardware equipment for long-term monitoring and personalized model building:

[0209] High-performance servers: equipped with GPU accelerator cards (such as NVIDIA A100) for deep learning model training;

[0210] Memory and storage: Large memory (≥256GB) to support massive data processing, high-speed solid-state drive (≥1TB NVMe SSD) for model storage and reading;

[0211] Cloud computing platform: such as AWS, Azure, or local private cloud, for remote deployment and long-term operation of models.

[0212] Dynamic threshold setting of dynamic abnormality classification warning module Core hardware equipment:

[0213] Embedded edge devices: such as NVIDIA Jetson AGX Orin, which supports real-time calculation of dynamic thresholds;

[0214] Storage unit: Local non-volatile memory (NVRAM) is used to store dynamic thresholds and warning rules;

[0215] Core hardware equipment for graded warning strategy:

[0216] Edge server: Intel Xeon or AMD EPYC processor, providing large-scale data parallel processing capabilities;

[0217] Communication module:

[0218] Message push service: MQTT gateway;

[0219] SMS gateway equipment: dedicated GSM module (such as SIM800L);

[0220] Mail Server: Built-in SMTP hardware support.

[0221] The core hardware of the treatment strategy generation based on interdisciplinary knowledge of the treatment measures module:

[0222] AI inference hardware: Google Coral DevBoard or NVIDIA Jetson NX for real-time policy generation;

[0223] Knowledge graph storage: high-performance graph database servers (such as Neo4j) to store knowledge in education, psychology, and neuroscience.

[0224] Multimodal display unit core hardware equipment:

[0225] Speech synthesis: high-performance speech processing chip (such as InfineonXMC4000);

[0226] Display device: interactive touch screen, resolution ≥1920x1080, AR / VR head display device (such as MetaQuestPro) for scenario simulation.

[0227] Audio equipment: high-fidelity audio speakers (supporting 3D sound effects), noise-canceling headphones (such as Sony WH-1000XM5);

[0228] Control interface: multi-mode input devices (keyboard, touch, gesture sensor, such as LeapMotion).

[0229] System bus and interface support:

[0230] Communication protocol: Supports I2C, SPI, UART and other protocols for sensor communication, and TCP / IP or LoRaWAN between edge computing devices and servers;

[0231] Data interface: USB3.0 / 3.1 for high-speed data transmission, HDMI interface for visual content output.

[0232] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0233] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent learning behavior monitoring and abnormal warning device, characterized in that: The device architecture includes a multimodal data acquisition module, a behavior analysis module, a dynamic abnormality classification warning module and a processing measure module, wherein: The multimodal data acquisition module is used to collect multimodal data and situational awareness data in the student's learning process, including a multimodal data acquisition unit and a situational awareness data acquisition unit. The multimodal data acquisition unit uses a variety of data acquisition and processing algorithms to obtain students' operation records, facial expression data and physiological data on the learning platform. The situational awareness data acquisition unit is used to collect environmental factor data and situational information and associate them with other data; The behavior analysis module is used to deeply analyze the collected data to build an accurate behavior analysis model, including a behavior learning trajectory and micro-feature extraction unit and a long-term monitoring and personalized model construction unit. The behavior learning trajectory and micro-feature extraction unit is used to extract short-term and micro-level behavior features from multimodal data and construct a learning trajectory. The long-term monitoring and personalized model construction unit is used to mine student behavior patterns based on long-term data and build a personalized model; The dynamic abnormality graded warning module is used to flexibly set and implement a dynamic abnormality graded warning mechanism, including a dynamic threshold setting unit and a graded warning strategy unit. The dynamic threshold setting unit is used to dynamically adjust the abnormality judgment threshold of multimodal data according to the student's learning progress and course situation. The graded warning strategy unit is used to divide the grades according to the severity and impact of the abnormality and transmit different levels of warning information to relevant personnel; The processing measure module is used to generate processing strategies based on interdisciplinary knowledge and display them in various ways, including a processing strategy generation unit based on interdisciplinary knowledge and a multimodal display unit of processing methods. The processing strategy generation unit based on interdisciplinary knowledge is used to generate processing suggestions based on psychology, education and neuroscience knowledge, and the multimodal display unit of processing methods is used to convey processing suggestions to students through various forms such as voice, vision, hearing and situational simulation.

2. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: The learning platform operation record acquisition algorithm in the multimodal data acquisition unit specifically extracts operation records including click type, detailed path of browsing page, specific steps and time of answering questions from multiple related tables of the learning platform database through customized SQL query statements, and cleans these data at the same time to remove outliers caused by system failures or network problems, and then efficiently stores them in the form of a hash table according to student ID and time order, so as to comprehensively and accurately obtain students' operation records on the learning platform.

3. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: When collecting facial expression data in the multimodal data acquisition unit, the face detection algorithm in the Dlib library is first used to locate the face area of ​​the image stream obtained by the camera, and then a multi-classification model based on the convolutional neural network CNN is used to classify the expression features of the face area to obtain accurate facial expression data, and the physiological data is processed using an algorithm based on peak detection, bandpass filtering and fast Fourier transform of spectrum analysis.

4. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: The situational awareness data acquisition unit uses environmental sensors to collect data such as noise and light intensity in the learning environment. The noise data is calculated by performing analog-to-digital conversion on the analog signal collected by the sensor to calculate the average noise intensity. The light intensity data is converted into illuminance values ​​based on the light sensor readings. At the same time, the course schedule information is queried from the database of the course management system, and the course type and time information are associated with the corresponding data.

5. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: When fusing multimodal data in the behavior learning trajectory and micro-feature extraction unit, an interpolation-based time synchronization algorithm is used to align timestamps of different types of data to ensure consistency of different modal data in the time dimension; during the fusion process, an adaptive weighted average algorithm is used to dynamically adjust the weights of each modal data according to its importance in reflecting the learning behavior. The weights are obtained through analysis of a large amount of annotated data and training of a machine learning algorithm, and are used to more accurately extract short-term and micro-level behavioral features from multimodal data and construct learning trajectories.

6. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: The data mining algorithm in the long-term monitoring and personalized model construction unit, in cluster analysis, adopts a combination of density-based spatial clustering algorithm (DBSCAN) and hierarchical clustering algorithm to cluster students' long-term learning behavior data, which can effectively identify different types of learning behavior patterns. In association rule mining, the FP-Growth algorithm is used to mine frequent item sets and association rules to discover hidden relationships between different behavioral characteristics. When building a personalized model, the long short-term memory network (LSTM) in deep learning is used. Its ability to process time series data can better capture the changes in student behavior over time. At the same time, combined with the attention mechanism, the model can focus on behavioral features that are more critical to judging students' learning status, so as to more accurately mine student behavior patterns based on long-term data and build personalized models.

7. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: A polynomial regression model is used in the threshold adjustment in the dynamic threshold setting unit. The statistical characteristics of students' learning progress, course difficulty, and recent behavior data are used as independent variables, and the threshold adjustment amount is used as the dependent variable for training. When using fuzzy logic, fuzzy sets and fuzzy rules are defined to more reasonably dynamically adjust the abnormal judgment threshold of multimodal data according to students' learning progress and course conditions.

8. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: When evaluating the abnormal level in the hierarchical warning strategy unit, the exponentially weighted moving average algorithm is used to calculate the continuous impact degree according to the duration of the abnormal behavior in the time dimension. For the severity of the abnormality, the abnormal degree of the multimodal data is comprehensively considered, and the multimodal data is reduced in dimension by the principal component analysis (PCA) algorithm to calculate the comprehensive abnormality score. According to the potential impact of the abnormal behavior on the learning effect, combined with the impact assessment scale formulated by education experts, a quantitative assessment is performed through the fuzzy comprehensive evaluation algorithm. Finally, according to these quantitative results and the preset grade division threshold, the decision tree algorithm is used to divide the abnormal level, and the warning information of different levels is transmitted to the relevant personnel in an asynchronous communication manner through the integrated message push service, mail server and SMS gateway and other communication interfaces.

9. The intelligent learning behavior monitoring and abnormal warning device according to claim 1 is characterized by: When generating processing suggestions based on psychological knowledge, the processing strategy generation unit based on interdisciplinary knowledge adopts a Bayesian network reasoning algorithm, takes students' behavioral performance and psychological state as nodes, determines the conditional probability relationship between nodes by constructing a psychological model and learning a large amount of psychological experimental data, and infers possible psychological problems and corresponding processing strategies. When based on educational knowledge, based on the educational knowledge graph, a graph search algorithm is used to find educational theories and practical cases related to current students' learning problems, and generate teaching method adjustment suggestions. When based on neuroscience knowledge, a neural network simulation model is constructed to simulate the brain's neural activity in the learning process, and processing suggestions for physiological state adjustment are generated according to the model output and neuroscience research results.

10. The intelligent learning behavior monitoring and abnormal warning device according to claim 1, characterized in that: The operation method of the device is as follows: Sp1: The multimodal data acquisition module operates. The multimodal data acquisition unit uses the corresponding algorithm to obtain students' operation records, facial expression data and physiological data on the learning platform. The situational awareness data acquisition unit collects environmental factor data and situational information and associates them with other data; Sp2: The behavior analysis module processes the collected data. The behavior learning trajectory and micro-feature extraction unit extracts short-term and micro-level behavior features from multimodal data and constructs learning trajectories. The long-term monitoring and personalized model construction unit mines students' behavior patterns based on long-term data and constructs personalized models. Sp3: The dynamic abnormality graded warning module is started. The dynamic threshold setting unit dynamically adjusts the abnormality judgment threshold of multimodal data according to the student's learning progress and course situation. The graded warning strategy unit divides the grades according to the severity and impact of the abnormality and transmits different levels of warning information to relevant personnel; Sp4: The processing measures module works. The processing strategy generation unit based on interdisciplinary knowledge generates processing suggestions based on the knowledge of psychology, education and neuroscience. The multimodal display unit of processing methods conveys processing suggestions to students through various forms such as voice, vision, hearing and situational simulation.

Images