Mathematical learning difficulty real-time diagnosis method and system based on multi-source behavior data

Abstract

This invention relates to the field of educational technology and discloses a real-time diagnostic method and system for mathematical learning difficulties based on multi-source behavioral data. The method acquires students' multi-source raw data and performs real-time de-identification processing on the raw data to obtain multi-source target data. Behavioral features, cognitive features, and emotional features are extracted from the aligned data. A pre-defined knowledge graph and the behavioral, cognitive, and emotional features are input into a hybrid deep learning model to obtain a three-dimensional state vector. The hybrid deep learning model is constructed by combining a sequence coding network with a self-attention mechanism and a graph neural network. The three-dimensional state vector is then fused with the knowledge point labels of the current learning task to obtain fused features, which are input into a target diagnostic model to obtain a diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. This improves the accuracy and precision of the diagnosis.

Description

Technical Field

[0001] This invention relates to the field of educational technology, and relates to, but is not limited to, a method and system for real-time diagnosis of mathematical learning difficulties based on multi-source behavioral data. Background Technology

[0002] Difficulty in learning mathematics is one of the key factors affecting academic success among college students. With the development of educational informatization, early warning systems based on online learning logs, such as using answer sequences, can predict academic risks. However, these methods lack sufficient accuracy in diagnosing learning difficulties. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a real-time diagnostic method and system for mathematical learning difficulties based on multi-source behavioral data, which can improve the accuracy and precision of the diagnosis.

[0004] The specific technical solutions of this invention are as follows: The first aspect of this application provides a real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data, including: The system acquires students' raw data from multiple sources and performs real-time anonymization processing on the raw data to obtain multi-source target data. The multi-source target data includes at least students' video behavior data, cognitive log data, and emotional data. Spatiotemporal alignment of multi-source target data is performed, and behavioral, cognitive, and emotional features are extracted from the aligned data. A pre-defined knowledge graph and behavioral, cognitive, and emotional features are input into a hybrid deep learning model to obtain a three-dimensional state vector. The hybrid deep learning model is constructed by combining a sequence coding network with a self-attention mechanism and a graph neural network. The sequence coding network is used to capture the long-term dependencies and interactions between behavioral, cognitive, and emotional features, while the graph neural network is used to model the topological structure between knowledge points in the knowledge graph and the diffusion path of erroneous cognitions in the knowledge graph. Erroneous cognitions are determined based on cognitive features. The three-dimensional state vector is fused with the knowledge point labels of the current learning task to obtain fused features. These fused features are then input into the target diagnostic model to obtain a diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. The sequence feature processing branch processes the serialized feature portion of the fused features to capture dependencies across time steps. The image feature processing branch processes the image-based feature portion of the fused features to extract spatial visual features. The fusion decision module fuses the feature vectors output by the sequence feature processing branch and the image feature processing branch and outputs the diagnostic result.

[0005] In some embodiments, obtaining students' multi-source raw data includes: The learning video stream of students is collected by image acquisition equipment, and the facial movements, head posture, gesture trajectory and handwriting features of students are extracted from the learning video stream in real time to obtain video behavior data. Students' learning logs are collected from the learning management system and then digitized to obtain cognitive log data. Students' voice signals are collected through voice acquisition devices, and / or students' physiological signals are collected through wearable devices, and the voice signals and / or physiological signals are used as emotional data.

[0006] In some embodiments, real-time de-identification processing is performed on multi-source raw data to obtain multi-source target data, including: Dynamic blurring is applied to facial images in video behavior data, and voiceprint obfuscation or encryption is applied to speech signals in emotional data to obtain desensitized video behavior data and desensitized emotional data. Desensitized video behavioral data, desensitized emotional data, and cognitive log data were used as multi-source target data.

[0007] In some embodiments, spatiotemporal alignment is performed on multi-source target data, and behavioral features, cognitive features, and emotional features are extracted from the aligned data, including: An attention-based timestamp alignment algorithm is used to align the sampling frequencies of multi-source target data to obtain aligned data. Extract problem-solving focus index and operation hesitation time from the aligned data as behavioral features; Extract knowledge path complexity and error pattern repetition rate from the aligned data as cognitive features; Anxiety fluctuation coefficients were extracted from the aligned data as an emotional feature.

[0008] In some embodiments, the sequence feature processing branch is constructed based on a lightweight self-attention sequence encoding network, and the image feature processing branch is constructed based on a lightweight convolutional neural network.

[0009] In some embodiments, the target diagnostic model performs the following operations: Temporal modeling is performed on the serialized feature portion of the fused features through a sequence feature processing branch to obtain temporally dependent features; Spatial visual features are obtained by extracting spatial features from the image-based feature portion of the fused features through an image feature processing branch. The fusion decision module concatenates temporal-dependent features and spatial visual features to obtain a concatenated feature vector. The multilayer perceptron classification head is then used to compress the dimensions and integrate the information of the concatenated feature vector to obtain the diagnostic results.

[0010] In some embodiments, the method further includes: Based on the diagnostic results, an early warning message is sent to the teacher's monitoring terminal, and personalized prompts and intervention data are sent to the student's interactive terminal; the intervention data is obtained by matching the difficulty type attribution in the diagnostic results from a pre-set intervention database.

[0011] In some embodiments, the method further includes: Using a federated learning framework, the current global diagnostic model is obtained from the server or cloud, trained on local anonymized data to obtain a local diagnostic model, and the parameters of the local diagnostic model are uploaded to the server. A new global diagnostic model is obtained from the server and used as the target diagnostic model; the new global diagnostic model is obtained by updating the parameters of the local diagnostic model.

[0012] A second aspect of this application provides a real-time diagnostic system for mathematical learning difficulties based on multi-source behavioral data, comprising: The acquisition module is used to acquire students' raw data from multiple sources. The data processing module is used to perform real-time desensitization processing on multi-source raw data to obtain multi-source target data; among which, the multi-source target data includes at least students' video behavior data, cognitive log data, and emotional data; The spatiotemporal alignment module is used to perform spatiotemporal alignment on multi-source target data and extract behavioral features, cognitive features and emotional features from the aligned data. The data processing module is also used to input the preset knowledge graph and behavioral, cognitive, and emotional features into the hybrid deep learning model to obtain a three-dimensional state vector. The hybrid deep learning model is constructed by combining a sequence coding network with a self-attention mechanism and a graph neural network. The sequence coding network is used to capture the long-term dependencies and interactions between behavioral, cognitive, and emotional features, while the graph neural network is used to model the topological structure between knowledge points in the knowledge graph and the diffusion path of erroneous cognitions in the knowledge graph. Erroneous cognitions are determined based on cognitive features. The diagnostic module is used to fuse the 3D state vector with the knowledge point labels of the current learning task to obtain fused features. These fused features are then input into the target diagnostic model to obtain a diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. The sequence feature processing branch processes the serialized feature portion of the fused features to capture dependencies across time steps. The image feature processing branch processes the image-based feature portion of the fused features to extract spatial visual features. The fusion decision module fuses the feature vectors output by the sequence feature processing branch and the image feature processing branch and outputs the diagnostic result.

[0013] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this embodiment of the invention, multi-source raw data of students is acquired, and the multi-source raw data is de-identified in real time to obtain multi-source target data; wherein, the multi-source target data includes at least student video behavior data, cognitive log data, and emotional data; the multi-source target data is spatiotemporally aligned, and behavioral features, cognitive features, and emotional features are extracted from the aligned data; a preset knowledge graph and behavioral features, cognitive features, and emotional features are input into a hybrid deep learning model to obtain a three-dimensional state vector; wherein, the hybrid deep learning model is constructed based on a combination of a sequence coding network with a self-attention mechanism and a graph neural network; the sequence coding network is used to capture the long-term dependencies and interactions between behavioral features, cognitive features, and emotional features, and the graph neural network is used to model the topological structure between knowledge points in the knowledge graph and the diffusion path of erroneous cognitions in the knowledge graph; erroneous cognitions Based on cognitive features, the three-dimensional state vector is fused with the knowledge point labels of the current learning task to obtain fused features. These fused features are then input into the target diagnostic model to obtain a diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. The sequence feature processing branch processes the serialized feature portion of the fused features to capture dependencies across time steps. The image feature processing branch processes the image-based feature portion of the fused features to extract spatial visual features. The fusion decision module fuses the feature vectors output by the sequence feature processing branch and the image feature processing branch and outputs the diagnostic result. This approach captures changes in behavioral, cognitive, and emotional interactions across time steps and extracts spatial features of erroneous cognitive distributions within the knowledge graph topology, thereby improving the accuracy and precision of the diagnosis. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A flowchart illustrating the real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the structure of a real-time diagnostic system for mathematical learning difficulties based on multi-source behavioral data, provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0017] It should be noted that the terms "first, second, and third" used in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0018] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments of the invention pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0019] Figure 1This is a flowchart illustrating a real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data, provided by an embodiment of the present invention. The method can be executed via a control device, which may include at least one of a personal computer, laptop computer, smartphone, and tablet computer; however, this embodiment does not limit the type of control device.

[0020] like Figure 1 As shown, the real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data provided in this embodiment of the invention may include steps S101-S104.

[0021] S101. Obtain students' multi-source raw data and perform real-time anonymization processing on the multi-source raw data to obtain multi-source target data.

[0022] In some embodiments, multi-source target data includes at least students’ video behavior data, cognitive log data, and emotional data.

[0023] In some embodiments, acquiring students' multi-source raw data includes: acquiring students' learning video streams through image acquisition devices, and extracting students' facial movements, head postures, gesture trajectories, and handwriting features from the learning video streams in real time to obtain video behavior data; acquiring students' learning logs from the learning management system, and digitizing the learning logs to obtain cognitive log data; acquiring students' voice signals through voice acquisition devices, and / or acquiring students' physiological signals through wearable devices, and using the voice signals and / or physiological signals as emotional data.

[0024] In some embodiments, the image acquisition device may be a high-definition camera array deployed in the classroom. The camera array synchronously acquires video streams of students learning mathematics from multiple angles, including but not limited to close-ups of the student's face, upper body posture, hand operation area, and desktop writing area. The video stream is then analyzed in real time using computer vision algorithms to extract facial movements, head posture, gesture trajectories, and handwriting features to obtain video behavior data. Facial movements include changes in the student's eyebrow and eye area, and gaze direction. Head posture includes the head's pitch angle, yaw angle, and roll angle.

[0025] In some embodiments, the movement trajectory and spatial position changes of a student's fingers can be determined by identifying key points in the student's hand skeleton, thereby determining the student's gesture trajectory. For example, when a student attempts to draw an auxiliary line, the key points in the student's hand skeleton can be extracted from the learning video stream. Based on the positional changes of the key points in the student's hand skeleton, the movement trajectory and spatial position changes of the student's fingers can be determined, thereby determining the student's gesture trajectory when drawing the auxiliary line.

[0026] In some embodiments, handwriting images of students during the writing process can be captured in real time from the learning video stream, and the characteristics of the student's handwriting can be determined from the handwriting image fragments. For example, when a student writes "In isosceles triangle ABC, AB=AC", the handwriting image of this step can be extracted synchronously from the learning video stream to obtain the handwriting characteristics.

[0027] In some embodiments, student learning logs can be obtained from the learning management system via a data interface. These learning logs can include structured and unstructured learning logs. Structured learning logs can include answer sequences, page dwell times, and interactive click flows. Answer sequences refer to the order and duration of a student's answers to each question; page dwell time refers to the duration a student spends on each knowledge point page; and interactive click flows refer to actions such as mouse clicks, option switching, and page navigation. Unstructured learning logs include students' handwritten solutions, formula derivations, and graphical representations from paper or electronic assignments.

[0028] In some embodiments, after obtaining the learning log, the unstructured learning log can be digitized using content recognition technology (such as optical character recognition, formula recognition algorithm) to extract structured information such as problem-solving steps, derivation logic, and graphical parameters. This structured information is then combined with the structured learning log to obtain cognitive log data.

[0029] In some embodiments, the voice acquisition device may be a microphone array installed in the classroom, which can acquire students' voice signals in real time when they talk to themselves, participate in group discussions, or answer questions. The wearable device may be a smart bracelet worn by the students, and the physiological signals may include the students' heart rate, skin conductance, etc.

[0030] In some embodiments, real-time desensitization processing is performed on multi-source raw data to obtain multi-source target data, including: dynamic blurring of facial images in video behavior data and voiceprint obfuscation or encryption of speech signals in emotion data to obtain desensitized video behavior data and desensitized emotion data; the desensitized video behavior data, desensitized emotion data and cognitive log data are used as multi-source target data.

[0031] In some embodiments, an edge computing device can be deployed at the data acquisition end. The edge computing device can be an embedded artificial intelligence computing device. This application does not limit the specific model or deployment method of the edge computing device.

[0032] In some embodiments, after obtaining the multi-source raw data, the multi-source raw data is de-identified in real time using an edge computing device, and then the de-identified multi-source target data is uploaded to the cloud or server. This ensures the privacy and security of the obtained multi-source target data.

[0033] In some embodiments, dynamic blurring can employ a Gaussian blurring algorithm, which smooths the face region by setting an appropriate Gaussian kernel standard deviation parameter, thereby blurring biometric features that can identify an individual.

[0034] For example, the standard deviation of the Gaussian kernel can be set to 3.0. In this way, the blurred face image cannot be used for face recognition or identity verification, but behavioral features such as changes in the movement of the eyebrows and eyes and head posture can be extracted.

[0035] In some embodiments, voiceprint obfuscation refers to transforming or scrambling the voiceprint features in a speech signal so that the processed speech signal cannot be used for speaker identification or identity verification. For example, individual voiceprint features can be masked by changing the formant distribution of the speech signal, and voiceprint-related features such as Mel-frequency cepstral coefficients can be selectively masked. This application does not limit the voiceprint obfuscation method.

[0036] In some embodiments, encryption processing refers to encrypting acoustic features extracted from the speech signal. For example, emotional data such as fundamental frequency, jitter rate, and energy extracted from the speech signal can be encrypted. This application does not limit the encryption processing method.

[0037] S102. Perform spatiotemporal alignment on the multi-source target data, and extract behavioral features, cognitive features, and emotional features from the aligned data.

[0038] In some embodiments, spatiotemporal alignment of multi-source target data and extraction of behavioral, cognitive, and emotional features from the aligned data include: using an attention-based timestamp alignment algorithm to align the sampling frequencies of the multi-source target data to obtain aligned data; extracting problem-solving focus index and operation hesitation duration from the aligned data as behavioral features; extracting knowledge path complexity and error pattern repetition rate from the aligned data as cognitive features; and extracting anxiety fluctuation coefficient from the aligned data as emotional features.

[0039] In some embodiments, since the multi-source target data comes from different acquisition devices, the sampling frequencies of each modality of data (i.e., video behavior data, cognitive log data, and sentiment data) differ. In order to unify the video behavior data, cognitive log data, and sentiment data in the time dimension, spatiotemporal alignment processing needs to be performed.

[0040] In some embodiments, the event timestamps in the cognitive log data are used as the baseline anchor point. The correlation between each time point in the video behavior data and the sentiment data and the baseline anchor point is calculated through an attention mechanism. Based on the correlation, the video behavior data and sentiment data are interpolated or downsampled so that the video behavior data, cognitive log data and sentiment data are aligned on a unified timeline.

[0041] For example, when a student submits a question-answering step (i.e., cognitive log data) at a certain moment, the video frames and audio segments within a preset window before and after that moment can be traced back through the attention mechanism in the video behavior data and emotional data, and the correlation between each video frame and each audio segment and the cognitive time can be calculated. The video frames and audio segments with the highest correlation are selected as the behavior data and emotional data aligned with the cognitive event.

[0042] In some embodiments, a problem-solving focus index can be calculated based on facial movements and head postures in video behavior data, and an operation hesitation duration can be calculated based on the number of pauses in gesture trajectories and handwriting features. A higher problem-solving focus index indicates a higher level of concentration during problem-solving, while a longer operation hesitation duration indicates a less proficient grasp of the current problem-solving steps.

[0043] In some embodiments, the knowledge path complexity can be calculated based on the number of jumps and branch paths in the cognitive log data, and the error pattern repetition rate can be calculated based on the number of occurrences of the same error pattern in the cognitive log data. The higher the knowledge path complexity, the more difficult it is for students to master the cognitive process of the knowledge point; the higher the error pattern repetition rate, the greater the possibility that students have a wrong understanding of the corresponding knowledge point.

[0044] In some embodiments, the anxiety fluctuation coefficient can be calculated by weighting the fundamental frequency fluctuation amplitude of the speech signal, the rate of change of the physiological signal center rate, and the fluctuation amplitude of the skin conductance response in the emotional data. A higher anxiety fluctuation coefficient indicates more intense emotional fluctuations in the student at the current learning moment.

[0045] S103. Input the preset knowledge graph and behavioral, cognitive, and emotional features into the hybrid deep learning model to obtain a three-dimensional state vector. The hybrid deep learning model is constructed by combining a sequence coding network with a self-attention mechanism and a graph neural network. The sequence coding network is used to capture the long-term dependencies and interactions between behavioral, cognitive, and emotional features, while the graph neural network is used to model the topological structure between knowledge points in the knowledge graph and the diffusion path of erroneous cognitions in the knowledge graph. Erroneous cognitions are determined based on cognitive features.

[0046] In some embodiments, the hybrid deep learning model is deployed in the cloud or on a server. In the hybrid deep learning model, a sequence encoding network based on a self-attention mechanism and a graph neural network are configured in parallel. The sequence encoding network encodes behavioral, cognitive, and emotional features, capturing the nonlinear interactions between these features at different time steps. For example, it captures the impact of decreased focus on cognitive path selection when a student's anxiety fluctuation coefficient increases, thus obtaining a sequence encoding result containing temporal correlations. The graph neural network combines the adjacency relationships between knowledge points in the knowledge graph, embedding erroneous cognitions as node attributes into the knowledge graph, capturing the diffusion characteristics of erroneous cognitions along the knowledge point association paths. For example, if a student consistently makes a sign error on the knowledge point of "rearranging terms in a linear equation" and this error is not corrected, the error will spread along the "linear equation-quadratic equation" association path in the knowledge graph when solving "quadratic equations". After obtaining the sequence encoding result and diffusion characteristics, the sequence encoding result and diffusion characteristics are concatenated and fused to obtain a three-dimensional state vector.

[0047] In some embodiments, the sequence coding network based on the self-attention mechanism can be a network structure composed of multiple stacked Transformer encoder blocks. Each Transformer encoder block includes a multi-head self-attention layer and a feedforward neural network layer. The output is processed through residual connections and layer normalization, enabling the encoding of multimodal features (i.e., behavioral features, cognitive features, and emotional features) at different time steps, capturing long-range dependencies. Each multi-head self-attention layer can learn the attention weights between features in different subspaces, thereby comprehensively characterizing the interactive effects of behavioral, cognitive, and emotional features at different times. This application does not limit the specific structure of the sequence coding network based on the self-attention mechanism.

[0048] In some embodiments, the graph neural network can be a Graph Convolutional Network (GCN) formed by stacking multiple graph convolutional layers. It performs convolutional transformations on the features of each node and its neighboring nodes in the knowledge graph, aggregating neighborhood information to update the feature representation of each knowledge point node, thus capturing the features of erroneous cognition propagating along the knowledge association structure. Alternatively, it can be a Graph Attention Network (GAT), which calculates the attention weights of neighboring nodes to the central node, adaptively adjusting the contributions of different neighboring knowledge points to the features of the central node, thereby determining the differences in the influence of different knowledge points during the spread of erroneous cognition. This application does not limit the specific type of graph neural network.

[0049] S104. The three-dimensional state vector is fused with the knowledge point labels of the current learning task to obtain fused features. The fused features are then input into the target diagnostic model to obtain a diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. The sequence feature processing branch processes the serialized feature portion of the fused features to capture dependencies across time steps. The image feature processing branch processes the image-based feature portion of the fused features to extract spatial visual features. The fusion decision module fuses the feature vectors output by the sequence feature processing branch and the image feature processing branch and outputs the diagnostic result.

[0050] In some embodiments, the sequence feature processing branch is constructed based on a lightweight self-attention sequence encoding network, and the image feature processing branch is constructed based on a lightweight convolutional neural network.

[0051] In some embodiments, the target diagnostic model is deployed locally at the target institution. After a student completes a math learning task, the three-dimensional state vector obtained during the learning process can be input into the target diagnostic model for diagnosis to obtain the corresponding diagnostic results.

[0052] In some embodiments, the target diagnostic model specifically performs the following operations: performing temporal modeling on the serialized feature part of the fused features through the sequence feature processing branch to obtain temporally dependent features; extracting spatial features from the image-based feature part of the fused features through the image feature processing branch to obtain spatial visual features; concatenating the temporally dependent features and spatial visual features through the fusion decision module to obtain the concatenated feature vector; and using a multilayer perceptron classification head to perform dimensionality compression and information integration on the concatenated feature vector to obtain the diagnostic result.

[0053] In some embodiments, the sequence feature processing branch and the image feature processing branch are two parallel branches. The sequence feature processing branch can be composed of multiple stacked small Transformer encoder blocks. Compared to the sequence encoding network in the above embodiments, the sequence feature processing branch has a smaller parameter size, faster inference speed, and can adapt to lightweight deployment requirements. In some application scenarios, the sequence feature processing branch can use a lightweight Transformer (LiteTransformer) or a Group Transformer-based time series Reduction Network (GT-MRNet). This application does not limit the type and structure of the sequence feature processing branch. The image feature processing branch can be composed of multiple stacked depthwise separable convolutional layers. In practical applications, depthwise separable convolution can significantly reduce the computational cost and parameter count, ensuring model inference efficiency while extracting spatial visual features of the image, meeting the needs of real-time diagnosis. In some application scenarios, the image feature processing branch can use MobileNetV3 or ShuffleNetV2. This application does not limit the specific structure of the image feature processing branch.

[0054] In some embodiments, the fusion decision module includes a stitching layer and a multilayer perceptron (MLP) classification head. The stitching layer stitches together temporally dependent features and spatial visual features to obtain a stitched feature vector. The MLP classification head performs dimensionality compression and information integration on the stitched feature vector to obtain the corresponding difficulty probability, difficulty type probability distribution, and difficulty localization embedding vector as diagnostic results. The difficulty probability represents the confidence level that a student has a corresponding type of mathematical learning difficulty; the difficulty type probability distribution is used to determine whether the student's error is related to knowledge recognition, problem-solving habits, or emotional interference; and the difficulty localization embedding vector characterizes which related knowledge points or areas of the student's learning difficulty lie in.

[0055] It is understood that the embodiments of this application process the fusion features of different structures in parallel through sequence feature processing branch and image feature processing branch. This can capture the changes in behavioral, cognitive and emotional interactions across time steps, and extract the spatial features of the distribution of erroneous cognitions in the knowledge graph topology, thereby improving the accuracy and precision of diagnosis.

[0056] In some embodiments, the real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data provided in this application further includes: sending early warning information to the teacher monitoring terminal and sending personalized prompt information and intervention data to the student interactive terminal based on the diagnostic results; wherein, the intervention data is obtained by matching from a preset intervention database based on the attribution of difficulty type in the diagnostic results.

[0057] In some embodiments, when the diagnostic results show that a student has difficulty learning a certain knowledge point, an early warning message can be generated. The early warning message includes the location of the difficult knowledge point, the type of error, and the confidence level. The early warning message is then sent to the classroom monitoring terminal so that the teacher can identify the student's current problem and provide targeted tutoring.

[0058] In some embodiments, the control device can match corresponding intervention data from a preset intervention database based on the attribution of difficulty type in the diagnostic results and send it to the student interactive terminal. Simultaneously, it generates emotion regulation guidance prompts and sends these prompts as personalized information to the student interactive terminal. The intervention data includes explanations of knowledge points related to misconceptions and targeted variation training exercises.

[0059] In some embodiments, the real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data provided in this application further includes: obtaining a current global diagnostic model from a server or cloud through a federated learning framework, training the current global diagnostic model on local desensitized data to obtain a local diagnostic model, and uploading the parameters of the local diagnostic model to the server; obtaining a new global diagnostic model from the server and using the new global diagnostic model as the target diagnostic model; wherein the new global diagnostic model is obtained by updating the parameters of the local diagnostic model.

[0060] In some embodiments, since the data of each target institution is stored locally and cannot be directly exported, a horizontal federated learning framework can be adopted to complete the multi-center joint training of the model without sharing the original user data.

[0061] For example, using a federated learning framework, each target institution downloads the current version of the global diagnostic model (i.e., the current global diagnostic model) from the cloud or server, and trains the current version of the global diagnostic model on local anonymized data to obtain a local diagnostic model. Then, each target institution uploads the parameters of its local diagnostic model to the cloud or server. The cloud or server aggregates the parameters of the local diagnostic models corresponding to each target institution and updates the current version of the global diagnostic model to obtain a new global diagnostic model. This new global diagnostic model is then distributed to each target institution, which uses it as its target diagnostic model.

[0062] In some embodiments, the above steps can be repeated to iteratively update the target diagnostic model.

[0063] Understandably, the federated learning framework can both protect the privacy of students from different institutions and continuously optimize the diagnostic performance of the target diagnostic model by utilizing massive amounts of data from multiple institutions, ensuring that the model can output stable and accurate diagnostic results under different learning scenarios and learning situations. This forms a complete closed loop of "data collection - diagnostic analysis - intervention feedback - effect evaluation - model optimization".

[0064] Based on the same inventive concept, this application also provides a real-time diagnostic system for mathematical learning difficulties based on multi-source behavioral data, which is used to implement a real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data. The solution provided by this system is similar to the solution described in the above method; therefore, please refer to the limitations of the real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data described above, and will not be repeated here. Specifically, Figure 2 This is a schematic diagram of the structure of a real-time diagnostic system for mathematical learning difficulties based on multi-source behavioral data, as described in an embodiment of this application. Figure 2 As shown, the real-time diagnostic system for mathematical learning difficulties based on multi-source behavioral data includes: Module 201 is used to acquire students' multi-source raw data; The data processing module 202 is used to perform real-time desensitization processing on multi-source raw data to obtain multi-source target data; wherein, the multi-source target data includes at least students' video behavior data, cognitive log data and emotional data; The spatiotemporal alignment module 203 is used to perform spatiotemporal alignment on multi-source target data and extract behavioral features, cognitive features and emotional features from the aligned data. The data processing module 202 is also used to input the preset knowledge graph and behavioral features, cognitive features, and emotional features into the hybrid deep learning model to obtain a three-dimensional state vector. The hybrid deep learning model is constructed by combining a sequence coding network with a self-attention mechanism and a graph neural network. The sequence coding network is used to capture the long-term dependencies and interactions between behavioral features, cognitive features, and emotional features, while the graph neural network is used to model the topological structure between knowledge points in the knowledge graph and the diffusion path of erroneous cognitions in the knowledge graph. Erroneous cognitions are determined based on cognitive features. The diagnostic module 204 is used to fuse the three-dimensional state vector with the knowledge point labels of the current learning task to obtain fused features, and input the fused features into the target diagnostic model to obtain a diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. The sequence feature processing branch is used to process the serialized feature part in the fused features to capture the dependencies across time steps. The image feature processing branch is used to process the image-based feature part in the fused features to extract spatial visual features. The fusion decision module is used to fuse the feature vectors output by the sequence feature processing branch and the image feature processing branch and output the diagnostic result.

[0065] In some embodiments, the acquisition module 201 is further configured to acquire students' learning video streams through an image acquisition device, and extract students' facial movements, head postures, gesture trajectories, and handwriting features from the learning video streams in real time to obtain video behavior data; acquire students' learning logs from the learning management system, and digitize the learning logs to obtain cognitive log data; acquire students' voice signals through a voice acquisition device, and / or acquire students' physiological signals through a wearable device, and use the voice signals and / or physiological signals as emotional data.

[0066] In some embodiments, the data processing module 202 is further configured to perform dynamic blurring processing on the face image in the video behavior data, and perform voiceprint obfuscation or encryption processing on the voice signal in the emotion data to obtain desensitized video behavior data and desensitized emotion data; and use the desensitized video behavior data, desensitized emotion data and cognitive log data as multi-source target data.

[0067] In some embodiments, the spatiotemporal alignment module 203 uses an attention-based timestamp alignment algorithm to align the sampling frequencies of multi-source target data to obtain aligned data; it extracts problem-solving focus index and operation hesitation duration from the aligned data as behavioral features; it extracts knowledge path complexity and error pattern repetition rate from the aligned data as cognitive features; and it extracts anxiety fluctuation coefficient from the aligned data as emotional features.

[0068] In some embodiments, the sequence feature processing branch is constructed based on a lightweight self-attention sequence encoding network, and the image feature processing branch is constructed based on a lightweight convolutional neural network.

[0069] In some embodiments, the diagnostic module 204 is further configured to perform temporal modeling on the serialized feature portion of the fused features through a sequence feature processing branch to obtain temporal dependent features; Spatial visual features are obtained by extracting spatial features from the image-based feature portion of the fused features through an image feature processing branch. The fusion decision module concatenates temporal-dependent features and spatial visual features to obtain a concatenated feature vector. The multilayer perceptron classification head is then used to compress the dimensions and integrate the information of the concatenated feature vector to obtain the diagnostic results.

[0070] In some embodiments, the data processing module 202 is further configured to send early warning information to the teacher monitoring terminal and personalized prompt information and intervention data to the student interactive terminal based on the diagnostic results; wherein the intervention data is obtained by matching from a preset intervention database based on the attribution of difficulty type in the diagnostic results.

[0071] In some embodiments, the system further includes an update module, configured to: obtain the current global diagnostic model from a server or cloud via a federated learning framework; train the current global diagnostic model on local de-identified data to obtain a local diagnostic model, and upload the parameters of the local diagnostic model to the server; obtain a new global diagnostic model from the server, and use the new global diagnostic model as the target diagnostic model; wherein the new global diagnostic model is obtained by updating the parameters of the local diagnostic model.

[0072] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of the invention, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the invention. The sequence numbers of the above-described embodiments of the invention are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0073] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0074] In the several embodiments provided by this invention, it should be understood that the disclosed methods can be implemented in other ways. The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0075] The above description is merely an embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time diagnostic method for mathematical learning difficulties based on multi-source behavioral data, characterized in that, include: The system acquires students' raw data from multiple sources and performs real-time anonymization processing on the raw data to obtain multi-source target data. The multi-source target data includes at least students' video behavior data, cognitive log data, and emotional data. Spatiotemporal alignment of multi-source target data is performed, and behavioral, cognitive, and emotional features are extracted from the aligned data. A pre-defined knowledge graph and behavioral, cognitive, and emotional features are input into a hybrid deep learning model to obtain a three-dimensional state vector. The hybrid deep learning model is constructed by combining a sequence coding network with a self-attention mechanism and a graph neural network. The sequence coding network is used to capture the long-term dependencies and interactions between behavioral, cognitive, and emotional features, while the graph neural network is used to model the topological structure between knowledge points in the knowledge graph and the diffusion path of erroneous cognitions in the knowledge graph. Erroneous cognitions are determined based on cognitive features. The three-dimensional state vector is fused with the knowledge point labels of the current learning task to obtain fused features. These fused features are then input into the target diagnostic model to obtain a diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. The sequence feature processing branch processes the serialized feature portion of the fused features to capture dependencies across time steps. The image feature processing branch processes the image-based feature portion of the fused features to extract spatial visual features. The fusion decision module fuses the feature vectors output by the sequence feature processing branch and the image feature processing branch and outputs the diagnostic result.

2. The method according to claim 1, characterized in that, Obtain student raw data from multiple sources, including: The learning video stream of students is collected by image acquisition equipment, and the facial movements, head posture, gesture trajectory and handwriting features of students are extracted in real time from the learning video stream to obtain video behavior data. Students' learning logs are collected from the learning management system and then digitized to obtain cognitive log data. Students' voice signals are collected through voice acquisition devices, and / or students' physiological signals are collected through wearable devices, and the voice signals and / or physiological signals are used as emotional data.

3. The method according to claim 1 or 2, characterized in that, Real-time anonymization processing is performed on the original data from multiple sources to obtain multi-source target data, including: Dynamic blurring is applied to facial images in video behavior data, and voiceprint obfuscation or encryption is applied to speech signals in emotional data to obtain desensitized video behavior data and desensitized emotional data. Desensitized video behavioral data, desensitized emotional data, and cognitive log data were used as multi-source target data.

4. The method according to claim 1, characterized in that, Spatiotemporal alignment of multi-source target data is performed, and behavioral, cognitive, and emotional features are extracted from the aligned data, including: An attention-based timestamp alignment algorithm is used to align the sampling frequencies of multi-source target data to obtain aligned data. Extract problem-solving focus index and operation hesitation time from the aligned data as behavioral features; Extract knowledge path complexity and error pattern repetition rate from the aligned data as cognitive features; Anxiety fluctuation coefficients were extracted from the aligned data as an emotional feature.

5. The method according to claim 1, characterized in that, The sequence feature processing branch is built based on a lightweight self-attention sequence encoding network, and the image feature processing branch is built based on a lightweight convolutional neural network.

6. The method according to claim 5, characterized in that, The target diagnostic model performs the following operations: Temporal modeling is performed on the serialized feature portion of the fused features through a sequence feature processing branch to obtain temporally dependent features; Spatial visual features are obtained by extracting spatial features from the image-based feature portion of the fused features through an image feature processing branch. The fusion decision module concatenates temporal-dependent features and spatial visual features to obtain a concatenated feature vector. The multilayer perceptron classification head is then used to compress the dimensions and integrate the information of the concatenated feature vector to obtain the diagnostic results.

7. The method according to claim 1, characterized in that, The method further includes: Based on the diagnostic results, an early warning message is sent to the teacher's monitoring terminal, and personalized prompts and intervention data are sent to the student's interactive terminal; the intervention data is obtained by matching the difficulty type attribution in the diagnostic results from a pre-set intervention database.

8. The method according to claim 1, characterized in that, The method further includes: Using a federated learning framework, the current global diagnostic model is obtained from the server or cloud, trained on local anonymized data to obtain a local diagnostic model, and the parameters of the local diagnostic model are uploaded to the server. A new global diagnostic model is obtained from the server and used as the target diagnostic model; the new global diagnostic model is obtained by updating the parameters of the local diagnostic model.

9. A real-time diagnostic system for mathematical learning difficulties based on multi-source behavioral data, characterized in that, include: The acquisition module is used to acquire students' raw data from multiple sources. The data processing module is used to perform real-time desensitization processing on multi-source raw data to obtain multi-source target data; among which, the multi-source target data includes at least students' video behavior data, cognitive log data, and emotional data; The spatiotemporal alignment module is used to perform spatiotemporal alignment on multi-source target data and extract behavioral features, cognitive features and emotional features from the aligned data. The data processing module is also used to input the preset knowledge graph and behavioral, cognitive, and emotional features into the hybrid deep learning model to obtain a three-dimensional state vector. The hybrid deep learning model is constructed by combining a sequence coding network with a self-attention mechanism and a graph neural network. The sequence coding network is used to capture the long-term dependencies and interactions between behavioral, cognitive, and emotional features, while the graph neural network is used to model the topological structure between knowledge points in the knowledge graph and the diffusion path of erroneous cognitions in the knowledge graph. Erroneous cognitions are determined based on cognitive features. The diagnostic module is used to fuse the 3D state vector with the knowledge point labels of the current learning task to obtain fused features. The fused features are then input into the target diagnostic model to obtain the diagnostic result. The target diagnostic model includes a sequence feature processing branch, an image feature processing branch, and a fusion decision module. The sequence feature processing branch processes the serialized feature portion of the fused features to capture dependencies across time steps. The image feature processing branch processes the image-based feature portion of the fused features to extract spatial visual features. The fusion decision module fuses the feature vectors output by the sequence feature processing branch and the image feature processing branch and outputs the diagnostic result.

Images