Online learning cognitive abnormal student detection method and system based on multi-modal representation

Abstract

The application discloses a kind of online learning cognitive abnormal student detection method and system based on multi-modal representation, belong to abnormal detection field.The application first obtains the multi-modal data in the learning process of student, constructs student relationship diagram;Then the multi-modal data of each student is input into the multi-modal learning representation module pre-trained through auxiliary classification task, obtains the multi-modal embedding corresponding to each student;Finally, the multi-modal embedding and student relationship diagram are input into the cognitive abnormal student detection module pre-trained through contrast learning framework, obtain the cognitive abnormal score of the student corresponding to target node, the application can automatically extract student cognitive information, avoid the heavy work of artificial processing, simultaneously based on multi-view multi-strategy contrast realizes cognitive abnormal student detection, can consider on the basis of multiple influence factors the cognitive level of student on platform is detected, auxiliary promotion online learning effect.

Description

Technical Field

[0001] This invention belongs to the field of anomaly detection, specifically relating to a method and system for detecting students with cognitive abnormalities in online learning based on multimodal representation. Background Technology

[0002] Online learning is a learning method that utilizes computer networks or mobile networks for remote instruction, allowing students to interact with teachers in virtual classrooms. In recent years, with the rapid development of internet technology, online learning has gradually become a mainstream teaching model. As the scale of online learning continues to expand, a large amount of educational data has accumulated in classrooms, providing new possibilities for personalized cognitive support for students. The vast amount of multimodal educational data generated in online classrooms (such as text, audio, and video) is closely related to students' cognitive states. However, in large-scale online learning scenarios, manually processing this multimodal information is time-consuming and labor-intensive, making it difficult for teachers to comprehensively and promptly analyze the cognitive status of each student. Furthermore, students' cognitive deficiencies are often influenced by multiple factors and manifest in diverse ways; therefore, it is almost impossible for teachers to quickly and accurately identify cognitively weak students and provide targeted guidance in large-scale classrooms. Thus, how to efficiently process multimodal data and identify student cognitive anomalies has become a key challenge in online education.

[0003] Data in online classrooms exhibits graph structure characteristics. Graph structures, as a general data representation method, can be effectively applied to multimodal data modeling in online classroom scenarios. Therefore, the detection of cognitively abnormal students in online learning scenarios can be viewed as an unsupervised node-level anomaly detection task based on multimodal representation learning on a static graph. Currently, multimodal representation learning methods mainly rely on supervised or semi-supervised learning approaches, but obtaining accurate teaching data labels in actual teaching scenarios is quite challenging. Mainstream methods for unsupervised node-level anomaly detection can be divided into two categories: one is based on graph autoencoders, such as DOMINANT, which, based on the assumption that anomalous samples are more difficult to reconstruct, uses graph convolutional networks (GCNs) to construct autoencoders to reconstruct attribute and adjacency matrices to detect anomalous nodes; the other is based on contrastive learning, such as CoLA, which captures local information from network data by sampling instances and uses contrastive learning techniques to learn node representations.

[0004] Although these methods have achieved certain results, there are still significant shortcomings: (1) Most current methods only compare and learn from individual perspectives and lack consideration of all-round information. Students' cognition in online classrooms is often affected by a variety of factors; (2) The interaction information between nodes is not fully utilized in anomaly detection, while the interaction between students is crucial to cognitive changes in the classroom; (3) The influence of the correlation between nodes on the attribute graph on anomaly detection performance is not considered. There are different degrees of correlation between students in online classrooms, and these complex relationships will interfere with anomaly detection. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art and to provide a method and system for detecting students with cognitive abnormalities in online learning based on multimodal representation.

[0006] The specific technical solution adopted in this invention is as follows:

[0007] In a first aspect, the present invention provides a method for detecting students with cognitive abnormalities in online learning based on multimodal representation, comprising:

[0008] S1. Obtain multimodal data of each student's learning process in the target online course. The multimodal data includes video data recording students' facial expressions and body behaviors, audio data recording the sound of the online classroom, and text data recording the interactive discussion text of the online classroom. Construct a student relationship graph by taking all students as nodes, the interactions between students as edges, and the basic information of students registered on the platform as node features.

[0009] S2. Input the multimodal data of each student in the target online course into the multimodal learning representation module pre-trained by the auxiliary classification task to obtain the multimodal embedding corresponding to each student; in the multimodal learning representation module, firstly, for the video, audio and text modes in each multimodal data, the different modal data are encoded using modality-specific encoders to obtain their respective single modal embeddings, and then the single modal embeddings of the three modes are concatenated and input into the Transformer-based shared encoder to fuse them into a multimodal embedding;

[0010] S3. Input the multimodal embeddings and student relationship graphs corresponding to all students in the target online course into the cognitive abnormality student detection module pre-trained through the contrastive learning framework to obtain the cognitive abnormality score of the student corresponding to the target node.

[0011] The cognitively abnormal student detection module consists of a multi-view construction module, a global feature extraction module, a focused feature extraction module, a hypergraph feature extraction module, and a scoring module. The multi-view construction module combines the student relationship graph and the multimodal embedding to construct a global view, and simultaneously converts the global view into a focused view through edge filtering based on node similarity. The global feature extraction module generates a subgraph-level representation of the target node corresponding to the student to be detected within the global view. and node-level representation The focused feature extraction module is used to generate a subgraph-level representation of the target node in the focused view for the target node corresponding to the student to be detected. and node-level representation The hypergraph feature extraction module is used to generate a subgraph-level representation of the target node under the hypergraph view through dual hypergraph transformation and hypergraph neural network. In the scoring module, based on the various feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module under multiple rounds of random walks, and combined with historical information in the memory bank, multi-dimensional anomaly scoring is performed through a bilinear layer, and the final cognitive anomaly score of the target node is obtained by fusion.

[0012] As a preferred embodiment of the first aspect above, in S2, when the multimodal learning representation module is pre-trained through the auxiliary classification task, the signal conversion module additionally set during the training phase is used to perform different categories of signal conversion on the single-modal embedding of each modality in the multimodal data. Then, based on the single-modal embedding after signal conversion, the signal conversion category of each modality is predicted through a specific modality classification head, thereby optimizing the multimodal learning representation module through classification cross-entropy loss.

[0013] As a preferred embodiment of the first aspect, the signal conversion module has five types of signal conversion categories: arrangement, time distortion, noise addition, amplitude distortion, and clipping.

[0014] As a preferred embodiment of the first aspect above, the internal processes of each sub-module of the cognitively abnormal student detection module in S3 are as follows:

[0015] In the multi-view construction module, the node features of student nodes in the student relationship graph are replaced with corresponding multimodal embeddings to form a global view; the edge connections in the global view are filtered based on the feature similarity between nodes, and edges with feature similarity between nodes below the threshold are deleted, thereby converting the global view into a focused view.

[0016] In the global feature extraction module, for the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the global view. A subgraph is sampled through random walk and input into the first graph convolutional network to aggregate and generate hidden representations layer by layer. The node dimensions of all the hidden representations finally output by the first graph convolutional network are averaged to obtain the subgraph-level representation of the target node in the global view. Simultaneously, the node features of the target node in the global view are input into the first graph convolutional network, mapping them to the node-level representation of the target node in the global view. ;

[0017] In the focused feature extraction module, for the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the focused view. A subgraph is sampled through random walk and input into the second graph convolutional network to generate hidden representations layer by layer. Based on all the hidden representations of the nodes finally output by the second graph convolutional network, the similarity of each other node to the target node is calculated. Then, the similarity is mapped to weights through a multilayer perceptron, and the hidden representations of all nodes finally output by the second graph convolutional network are weighted and fused to obtain the subgraph-level representation of the target node in the focused view. Simultaneously, the node features of the target node in the focused view are input into the second graph convolutional network, mapping them to the node-level representation of the target node in the focused view. ;

[0018] In the hypergraph feature extraction module, the subgraphs collected in the global feature extraction module are first subjected to dual hypergraph transformation to obtain subgraphs under the hypergraph view, which are then input into the hypergraph neural network for encoding. The average of the hypernode dimensions of all the hypernode hidden representations finally output by the hypergraph neural network is then performed to obtain the subgraph-level representation of the target node under the hypergraph view. ;

[0019] In the scoring module, based on the various feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module, the similarity of the target node's node-level representation with other nodes in the global view is calculated using a bilinear layer, which serves as the first dimension anomaly score. The similarity of the target node's node-level representation in the current online course and historical online courses in the global view is calculated using a bilinear layer, which serves as the second dimension anomaly score. The similarity of the target node's node-level representation in the global view and the focused view is calculated using a bilinear layer, which serves as the third dimension anomaly score. The anomaly scores of the three dimensions are weighted and summed to obtain the total cognitive anomaly score of the student corresponding to the target node in the target online course.

[0020] As a preferred embodiment of the first aspect mentioned above, in the cognitive abnormality student detection module, the global feature extraction module, the focused feature extraction module, and the focused feature extraction module need to perform multiple rounds of random walks. The feature representations generated by the multiple rounds of random walks are respectively input into the scoring module to calculate the abnormality scores and the total cognitive abnormality score in the three dimensions. Then, the scores of the same category from the multiple rounds of random walks are averaged, and the abnormality scores in the three dimensions and the total cognitive abnormality score are finally output in the form of average scores.

[0021] As a preferred embodiment of the first aspect above, in S3, when training the cognitively abnormal student detection module through the contrastive learning framework, it is necessary to perform focused-global contrastive learning, historical-current contrastive learning, and individual-interaction contrastive learning, and use the weighted sum of the contrastive learning losses of each as the total loss function;

[0022] In the focused-global contrastive learning, positive and negative sample pairs are constructed for both the focused view and the global view. In the focused view, a positive sample pair is formed by combining the subgraph-level representation of the target node with the node-level representation of the target node, and a negative sample pair is formed by combining the subgraph-level representation of the target node with the node-level representation of other nodes. In the global view, a positive sample pair is formed by combining the subgraph-level representation of the target node with the node-level representation of the target node, and a negative sample pair is formed by combining the subgraph-level representation of the target node with the node-level representation of other nodes.

[0023] In the aforementioned history-current comparative learning, the node-level representation of the target node in the current target online course global view is used. The average node-level representation of the target node in the global view of all historical online courses stored in the memory is used to form a positive sample pair. The node-level representation of the target node in the current target online course in the global view is then used as the basis for this positive sample pair. The average node-level representation of all other nodes in the global view of all historical online courses stored in the memory forms a negative sample pair;

[0024] In the individual-interaction contrastive learning, positive sample pairs are formed by comparing the subgraph-level representation of the target node in the hypergraph view with the subgraph-level representation of the target node in the global view, and negative sample pairs are formed by comparing the subgraph-level representation of the target node in the hypergraph view with the subgraph-level representations of other nodes in the global view.

[0025] As a preferred embodiment of the first aspect, the multimodal learning representation module is first trained independently, and after training, the learning parameters within the module are fixed and used to generate the multimodal embedding corresponding to each student, and then the cognitive abnormality student detection module is trained independently.

[0026] Secondly, the present invention provides an online learning cognitive abnormality student detection system based on multimodal representation, comprising:

[0027] The multimodal data acquisition module is used to collect multimodal data from each student during the learning process in each online course on the online learning platform.

[0028] The detection target specification module allows management users to specify student users who need to be tested for cognitive abnormalities via a GUI interface.

[0029] The cognitive abnormality detection module is used to obtain multimodal data corresponding to the specified student user from the multimodal data acquisition module according to the specified student user in the specified module of the detection object, and then obtain the cognitive abnormality score of the specified student user according to the online learning cognitive abnormality student detection method based on multimodal representation as described in any of the above-mentioned first aspect schemes.

[0030] The memory module is used to store data generated by the other three system modules in historical online courses on the online learning platform.

[0031] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the online learning cognitive abnormality student detection method based on multimodal representation as described in any of the solutions of the first aspect above.

[0032] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0033] The memory is used to store computer programs;

[0034] The processor is configured to, when executing the computer program, implement the online learning cognitive abnormality student detection method based on multimodal representation as described in any of the first aspects above.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] 1) This invention addresses the difficulty of obtaining centralized labels in online education datasets by designing an unsupervised multimodal representation learning framework. Pseudo-labels are automatically generated through signal transformation methods (such as noise addition and amplitude distortion) for pre-training of self-supervised learning models to extract students' cognitive representations. Furthermore, considering the correlation and heterogeneity among different modalities in educational data, this invention designs an intermediate fusion method combining temporal convolution and Transformer to achieve effective fusion of multimodal data.

[0037] 2) This invention constructs online classroom data into a cognitive graph and models the cognitive anomaly detection problem as a node-level anomaly detection task on a static graph. Through multiple augmented views and multiple comparison strategies, it comprehensively analyzes the degree of cognitive anomalies among students. This invention introduces three comparative learning strategies: a focused-global comparison strategy (extracting key influencing factors), an individual-interaction comparison strategy (considering rich interactions among students), and a historical-current comparison strategy (identifying abnormal changes in cognitive development), to more accurately identify students with cognitive anomalies.

[0038] 3) This invention automatically extracts students' cognitive information through a multimodal representation learning module, avoiding the heavy workload of manual processing; the cognitive abnormality student detection module based on multi-view and multi-strategy comparison can analyze the comprehensive cognitive level of each student based on multiple influencing factors, thus helping to improve the effectiveness of online learning. Attached Figure Description

[0039] Figure 1 A schematic diagram illustrating the steps of an online learning cognitive abnormality detection method based on multimodal representation;

[0040] Figure 2 A schematic diagram of the module composition for the detection module for students with cognitive impairment;

[0041] Figure 3 A schematic diagram of the module composition of an online learning cognitive abnormality detection system based on multimodal representation;

[0042] Figure 4 This is a schematic diagram of the components of a computer electronic device;

[0043] Figure 5 This example illustrates the module structure and training framework required for detecting students with cognitive abnormalities in online learning. Detailed Implementation

[0044] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in various embodiments of the present invention can be combined accordingly without mutual conflict.

[0045] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0046] This invention provides a method for detecting students with cognitive abnormalities in online learning based on multimodal representation, which has made core improvements in two aspects.

[0047] First, traditional multimodal representation learning methods mostly employ supervised training. However, in real-world teaching environments, obtaining accurate student cognitive labels is challenging, standard protocols are not yet clearly defined, and data labeling costs are high. To address this, this invention extracts student cognitive representations through unsupervised multimodal representation learning via signal transformation. Specifically, this invention proposes an unsupervised multimodal representation learning framework, with two key aspects: signal transformation to generate pseudo-labels and a cross-modal fusion strategy.

[0048] 1) Signal conversion to generate pseudo-labels: This invention designs a variety of signal conversion methods (such as noise addition, amplitude distortion, etc.) to perform signal conversion on unlabeled multimodal educational data and automatically generate pseudo-labels for pre-training of self-supervised learning (SSL) models, thereby extracting generalized multimodal representations related to cognition and supporting downstream tasks.

[0049] 2) Cross-modal fusion strategy: Given that educational data is both correlated and heterogeneous among different modalities, this invention proposes an intermediate fusion method that combines temporal convolution and Transformer to model multimodal data with heterogeneity and cross-modal correlation, thereby achieving effective fusion.

[0050] Secondly, in online learning scenarios, traditional anomaly detection methods often struggle to capture student interactions and fail to analyze a student's cognitive state based on factors more relevant to the target student, or consider past cognitive changes. This leads to insufficient accuracy in identifying cognitive anomalies. To address this, this invention leverages the advantages of contrastive learning in unsupervised node-level anomaly detection tasks, designing a label-free online learning cognitive anomaly detection module based on multi-strategy contrast. This module comprehensively considers multiple influencing factors in online classroom scenarios, thereby more accurately identifying students with cognitive anomalies. This invention constructs online classroom data into a cognitive graph, models the cognitive anomaly detection problem as a node-level anomaly detection task on a static graph, and comprehensively analyzes the degree of cognitive anomalies through multiple augmented views and multiple contrastive strategies. Specifically, the contrastive learning strategies in this invention are divided into three types: focus-global contrastive strategy, individual-interaction contrastive strategy, and history-present contrastive strategy.

[0051] 1) Focus-Wide Comparison Strategy: This invention defines the original graph composed of multimodal educational data as a "whole-domain graph," where each student node is connected to multiple peer nodes to reflect the student's overall cognitive state in the online classroom. However, educational practice shows that students' cognitive states are often primarily influenced by a few core peers. Therefore, this invention innovatively introduces a "focus-whole-domain comparison strategy," extracting a "focus graph" from the whole-domain graph using a similarity-based sampling method. In the focus graph, only adjacent nodes with a higher correlation to the target student's cognitive state are retained, thereby extracting key factors that significantly impact student cognition without disrupting the overall topology. Through comparative analysis of the focus graph and the whole-domain graph, the system can more accurately identify cognitive anomalies, effectively improving the sensitivity and accuracy of detection.

[0052] 2) Individual-Interaction Comparison Strategy: Traditional methods mainly focus on information at the individual node level, without considering the rich interactions between students. This invention constructs a dual hypergraph to extend the message passing mechanism between nodes to the perspective of students' social interactions, thereby more comprehensively detecting cognitive abnormalities in students.

[0053] 3) History-Current Comparison Strategy: In online classrooms, each lesson is not entirely independent. This invention incorporates historical information and proposes a history-current comparison method, which compares the current moment with the cognitive information of the same student in previous lessons to identify abnormal changes in cognitive development.

[0054] The following is a detailed description of the specific implementation of the online learning cognitive abnormality student detection method based on multimodal representation of the present invention.

[0055] like Figure 1 As shown, in a preferred embodiment of the present invention, a method for detecting students with cognitive abnormalities in online learning based on multimodal representation is provided, which specifically includes steps S1 to S3. The specific implementation of steps S1 to S3 will be described in detail below.

[0056] S1. Obtain multimodal data of each student's learning process in the target online course. The multimodal data includes video data recording students' facial expressions and body behaviors, audio data recording the sound of the online classroom, and text data recording the interactive discussion text of the online classroom. Construct a student relationship graph by taking all students as nodes, the interactions between students as edges, and the basic information of students registered on the platform as node features.

[0057] It should be noted that student interaction refers to the interactive behaviors that students engage in during the course. If there is interaction between two students, then edges can be constructed to connect the nodes of these two students in the student relationship graph.

[0058] S2. Input the multimodal data of each student in the target online course into the multimodal learning representation module pre-trained by the auxiliary classification task to obtain the multimodal embedding corresponding to each student; in the multimodal learning representation module, firstly, for the video, audio and text modes in each multimodal data, the different modal data are encoded using modality-specific encoders to obtain their respective single modal embeddings, and then the single modal embeddings of the three modes are concatenated and input into the Transformer-based shared encoder to fuse them into a multimodal embedding.

[0059] In embodiments of the present invention, a modality-specific encoder refers to an independent encoder for each modality, and encoders for different modalities are not shared. The modality-specific encoder uses a temporal convolutional network (TCN) as its backbone network, followed by a multilayer perceptron and layer normalization to obtain the encoded output. The raw data of the video, audio, and text modalities in the multimodal data needs to undergo necessary preprocessing before being input into the modality-specific encoder to meet its requirements. Data preprocessing is prior art and will not be elaborated here.

[0060] S3. Input the multimodal embeddings and student relationship graphs corresponding to all students in the target online course into the cognitive abnormality student detection module pre-trained through a contrastive learning framework to obtain the cognitive abnormality score of the student corresponding to the target node.

[0061] like Figure 2 As shown, the cognitive abnormality student detection module of the present invention consists of a multi-view construction module, a global feature extraction module, a focused feature extraction module, a hypergraph feature extraction module, and a scoring module.

[0062] 1) A multi-view construction module is used to construct a global view by combining the student relationship graph and the multimodal embedding, and at the same time, the global view is converted into a focused view by edge filtering operation based on node similarity.

[0063] In an embodiment of the present invention, the internal process of the multi-view construction module is as follows:

[0064] The node features of student nodes in the student relationship graph are replaced with corresponding multimodal embeddings to form a global view. The edge connections in the global view are filtered based on the feature similarity between nodes, and edges with feature similarity between nodes below a threshold are deleted, thereby converting the global view into a focused view.

[0065] 2) The global feature extraction module is used to generate a subgraph-level representation of the target node in the global view for the target node corresponding to the student to be detected. and node-level representation .

[0066] In an embodiment of the present invention, the internal process of the global feature extraction module is as follows:

[0067] For the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the global view. A subgraph is sampled through random walk and input into the first graph convolutional network to generate hidden representations layer by layer. The hidden representations of all nodes in the final output of the first graph convolutional network are averaged by node dimension to obtain the subgraph-level representation of the target node in the global view. Simultaneously, the node features of the target node in the global view are input into the first graph convolutional network, mapping them to the node-level representation of the target node in the global view. .

[0068] 3) Focused feature extraction module, used to generate a subgraph-level representation of the target node in the focused view for the target node corresponding to the student to be detected. and node-level representation .

[0069] In an embodiment of the present invention, the internal process of the focused feature extraction module is as follows:

[0070] For the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the focused view. A subgraph is sampled through random walk and input into the second graph convolutional network to generate hidden representations layer by layer. Based on the hidden representations of all nodes finally output by the second graph convolutional network, the similarity of each other node to the target node is calculated. Then, the similarity is mapped to weights through a multilayer perceptron, and the hidden representations of all nodes finally output by the second graph convolutional network are weighted and fused to obtain the subgraph-level representation of the target node in the focused view. Simultaneously, the node features of the target node in the focused view are input into the second graph convolutional network, mapping them to the node-level representation of the target node in the focused view. ;

[0071] 4) Hypergraph feature extraction module, used to generate subgraph-level representations of target nodes under the hypergraph view through dual hypergraph transformation and hypergraph neural network. .

[0072] In an embodiment of the present invention, the internal process of the hypergraph feature extraction module is as follows:

[0073] First, the subgraphs collected in the global feature extraction module are subjected to dual hypergraph transformation to obtain subgraphs under the hypergraph view, which are then input into the hypergraph neural network for encoding. Finally, the hypergraph dimension of all the hypernode hidden representations output by the hypergraph neural network is averaged to obtain the subgraph-level representation of the target node under the hypergraph view. ;

[0074] 5) The scoring module is used to perform multi-dimensional anomaly scoring through a bilinear layer based on the various feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module under multiple rounds of random walks, combined with historical information in the memory bank, and finally to obtain the total cognitive anomaly score of the target node.

[0075] In an embodiment of the present invention, the internal process of the scoring module is as follows:

[0076] Based on the various feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module, the similarity of the target node's node-level representation with other nodes (which can be any node other than the target node, or the node with the closest number) in the global view is calculated using a bilinear layer, and this is used as the first dimension anomaly score. The similarity of the target node's node-level representation in the current online course and historical online courses in the global view is calculated using a bilinear layer, and this is used as the second dimension anomaly score. The similarity of the target node's node-level representation in the global view and the focused view is calculated using a bilinear layer, and this is used as the third dimension anomaly score. The anomaly scores of the three dimensions are weighted and summed to obtain the total cognitive anomaly score of the student corresponding to the target node in the target online course.

[0077] It should be noted that, in the above-mentioned cognitive abnormality student detection module, considering the randomness of random walk sampling, it can be set that the global feature extraction module, the focused feature extraction module, and the focused feature extraction module need to perform multiple rounds of random walks during a single detection process. The feature representations generated by the multiple rounds of random walks are respectively input into the scoring module to calculate the abnormality scores of the three dimensions and the total cognitive abnormality score. Then, the scores of the same category from the multiple rounds of random walks are averaged, and the three-dimensional abnormality scores and the total cognitive abnormality score are finally output in the form of average scores.

[0078] It should be noted that, in the embodiments of the present invention, both the multimodal learning representation module and the cognitively abnormal student detection module need to be pre-trained, but they are not trained jointly. Instead, the multimodal learning representation module is trained independently first, and after training, the learning parameters within the module are fixed and used to generate the multimodal embedding corresponding to each student. Then, the cognitively abnormal student detection module is trained independently.

[0079] Furthermore, in the embodiments of the present invention, when pre-training the multimodal learning representation module, an auxiliary classification task for identifying signal transition categories needs to be designed. Specifically, the multimodal learning representation module is pre-trained using this auxiliary classification task by: utilizing a signal transition module additionally set during the training phase to perform different categories of signal transitions on the single-modal embeddings of each modality in the multimodal data; and then predicting the signal transition category of each modality based on the signal-transformed single-modal embeddings using a specific modality classification head, thereby optimizing the multimodal learning representation module through classification cross-entropy loss.

[0080] In the signal conversion module of this invention, the signal conversion category can be designed according to actual needs. In the embodiments of this invention, the signal conversion categories are divided into five types: arrangement, time warp, noise addition, amplitude warp, and clipping. Therefore, the final specific modality classification head outputs these five category labels.

[0081] Furthermore, the cognitively abnormal student detection module is trained using a contrastive learning framework. In embodiments of this invention, when training the cognitively abnormal student detection module using the contrastive learning framework, focused-global contrastive learning, historical-current contrastive learning, and individual-interaction contrastive learning are performed. The weighted sum of the contrastive learning losses for each type is used as the total loss function. The loss terms for different contrastive learning methods can be designed according to their respective comparison objects. The construction of positive and negative sample pairs is a crucial step in contrastive learning. In embodiments of this invention, the selection of positive and negative sample pairs for each of the three contrastive learning types is as follows:

[0082] 1) In the focused-global contrastive learning, positive and negative sample pairs are constructed for both the focused view and the global view. In the focused view, positive sample pairs are formed by the subgraph-level representation of the target node and the node-level representation of the target node, and negative sample pairs are formed by the subgraph-level representation of the target node and the node-level representation of other nodes. In the global view, positive sample pairs are formed by the subgraph-level representation of the target node and the node-level representation of the target node, and negative sample pairs are formed by the subgraph-level representation of the target node and the node-level representation of other nodes.

[0083] 2) In the history-current comparative learning, the node-level representation of the target node in the current target online course global view is used. The average node-level representation of the target node in the global view of all historical online courses stored in the memory is used to form a positive sample pair. The node-level representation of the target node in the current target online course in the global view is then used as the basis for this positive sample pair. The average node-level representation of all other nodes in the global view of all historical online courses stored in the memory forms a negative sample pair;

[0084] 3) In individual-interactive contrastive learning, positive sample pairs are formed by comparing the subgraph-level representation of the target node in the hypergraph view with the subgraph-level representation of the target node in the global view, and negative sample pairs are formed by comparing the subgraph-level representation of the target node in the hypergraph view with the subgraph-level representation of other nodes in the global view.

[0085] Similarly, based on the same inventive concept, such as Figure 3 As shown, another preferred embodiment of the present invention also provides an online learning cognitive abnormality student detection system based on multimodal representation, which includes:

[0086] The multimodal data acquisition module is used to collect multimodal data from each student during the learning process in each online course on the online learning platform.

[0087] The detection target specification module allows management users to specify student users who need to be tested for cognitive abnormalities via a GUI interface.

[0088] The cognitive abnormality detection module is used to obtain multimodal data corresponding to the specified student user from the multimodal data acquisition module according to the specified student user in the specified module of the detection object, and then obtain the cognitive abnormality score of the specified student user according to the online learning cognitive abnormality student detection method based on multimodal representation as described in the above embodiment.

[0089] The memory module is used to store data generated by the other three system modules in historical online courses on the online learning platform.

[0090] It should be noted that the aforementioned multimodal data acquisition module can be built into the backend server of the online learning platform to record multimodal data of each student's learning process in real time. The module for specifying the detection target can be located on the front-end interface of the online learning platform, providing the specified function through buttons or other means. Of course, the module can also have default settings for the detection target, such as all students being detected by default. If the administrator does not modify the default settings, the detection will be performed according to the default settings, with each student on the platform as the detection target, thus obtaining the cognitive abnormality scores of all students. By analyzing the relative levels of these cognitive abnormality scores, students requiring focused attention can be further filtered to support the online teaching process. Additionally, a threshold for cognitive abnormality scores can be set; if a student's cognitive abnormality score exceeds the threshold, a corresponding reminder can be issued. The reminder method can be implemented through a fixed module or display logic in the user interface, or through system messages, pop-ups, emails, SMS messages, etc.

[0091] It should also be noted that the online learning cognitive abnormality student detection methods based on multimodal representations described in S1-S3 of the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, such as... Figure 4 As shown, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the online learning cognitive abnormality student detection method based on multimodal representation provided in the above embodiments, which includes a memory and a processor;

[0092] The memory is used to store computer programs;

[0093] The processor is configured to implement the online learning cognitive abnormality student detection method based on multimodal representation in the above embodiments when executing the computer program.

[0094] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0095] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the online learning cognitive abnormality student detection method based on multimodal representation provided in the above embodiment. The storage medium stores a computer program, which, when executed by a processor, can realize the online learning cognitive abnormality student detection method based on multimodal representation in the above embodiment.

[0096] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0097] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0098] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0099] The following specific embodiment will demonstrate the implementation method and technical effect of the online learning cognitive abnormality student detection method based on multimodal representation described in S1~S3 above.

[0100] Example

[0101] This embodiment specifically demonstrates the implementation of the online learning cognitive abnormality student detection method based on multimodal representation described in S1~S3 above. Its specific module structure and training framework are as follows: Figure 5 As shown below, the overall implementation process will be explained in detail.

[0102] 1. Preprocessing of online classroom data

[0103] 1.1 Acquisition and Preprocessing of Multimodal Educational Data

[0104] This embodiment collects multimodal educational data from multiple online classes within a single class on the same day from multiple sources. Data from the last class is used as test data, while the rest are used as training data. For clarity, the following description uses three modalities of data. First, facial expressions and behavioral data of course participants are collected using the recording system of the online learning devices used by students. Second, audio information from both the teacher and students during the class is simultaneously collected using recording equipment. Finally, text information generated during the class is collected using computer equipment. After collecting the initial multimodal educational data, the data is standardized to ensure consistency of subsequent module inputs. Ultimately, three initial inputs are generated for the multimodal cognitive representation learning module: Where v, w, and t represent the video modality, audio modality, and text modality, respectively. Participants in the same online class need to have their respective multimodal data collected.

[0105] 1.2 Acquisition and Preprocessing of Basic Student Information and Interactive Perception Data in the Classroom

[0106] Before the online class begins, the system collects basic student information (such as age and gender). This information serves as the initial characteristics of the students, compensating for the difficulty in collecting deeper student information in online learning scenarios with a large number of students. Furthermore, this embodiment also collects real-time data on student interactions during the online classroom teaching process. After the above information is collected and processed, the student data in the classroom can be initially constructed into graph data. In this context, each student is considered a node. Interactions between students are considered side-by-side. The student's basic information is considered as the initial node information. This graph-structured data representation facilitates the application of graph neural network (GNN) methods in subsequent modules for the detection of students with cognitive abnormalities.

[0107] 2. Construction and training of an online learning cognitive abnormality detection model for students.

[0108] 2.1 Model Framework

[0109] 2.1.1 Cognitive Representation Learning Based on Multimodal Classroom Data

[0110] The cognitive representation learning module based on multimodal classroom data consists of two key elements: the encoder and the encoder. Modality-Specific Transform Classifier Encoder Including modality-specific encoders based on temporal convolution and Transformer-based shared encoder ,in, Es is responsible for modeling and analyzing the differences between multimodal data, enabling the exchange of information between modalities. Ultimately, the student's multimodal cognitive features obtained from the encoder are used as a modality-specific transform classifier. The input is used to identify the transformation type of each modality for training the cognitive representation learning module. Furthermore, a cognitive classification head is concatenated for the cognitive recognition task. This method maps students' multimodal cognitive features to cognitive vectors with specific meanings, helping teachers understand students' cognitive status. The implementation of these key components will be described in the following paragraphs.

[0111] First, a modality-specific encoder is defined. Considering the heterogeneity of multimodal signals, a separate encoder is first employed for each modality, using a temporal convolutional network (TCN) as the backbone to capture relevant information within low-level modalities. The key components of this network are two dilated causal convolutional layers. Furthermore, the receptive field is increased through dilated convolution, and the time step is adjusted accordingly. Dilated convolution on can be defined as:

[0112]

[0113] In the formula, Modal numbering, The coefficient of thermal expansion is 1 / 3. It is the size of The filters are applied. Each convolutional layer is followed by a weight normalization layer for the convolutional filters, an activation layer for regularization, and a dropout layer. Then, residual connections are created between the input and output of the block, where 1 × 1 convolutions are introduced to eliminate channel number mismatches between the input and output. Finally, a modality-specific projection head (implemented using an MLP) and layer normalization are applied to map low-level features to a higher-dimensional embedding space. The final output of the modality-specific encoder Ep is:

[0114]

[0115] in Representing modes The characteristics are represented.

[0116] A shared encoder was then defined. This was accomplished using a Transformer, where each modality identifies components of other modalities that are highly correlated with itself through an attention mechanism to achieve better signal transformation classification. For example, when a student answers a question, it is necessary to recognize not only the cognition contained in the audio information but also facial expressions, body language, etc., from the video information. To achieve this, low-level features of each modality are first stacked to form a multimodal embedding. Then, by scaling the dot product attention, the correlation between modes is calculated:

[0117]

[0118] Where Q, K, and V represent the query, key, and value of the attention, respectively, and T represents the matrix transpose. For the shared encoder Es, Q, K, and V are derived from multimodal features. The linear mapping is derived from this, and the final output of the attention layer is:

[0119]

[0120] in It is a learnable matrix. The final output... It contains students' multimodal cognitive information, which corresponds to students' cognitive representations, and to some extent provides a basis for the subsequent detection module for students with cognitive abnormalities.

[0121] Finally, by using a classification head specific to a particular modality, the classification results for each modality are obtained. These results are then used to train the cognitive representation learning module based on multimodal classroom data and to obtain the class probabilities of labeled samples, thus assisting teachers in providing more targeted guidance to students with cognitive deficiencies. The formulaic expression of the classification process is as follows:

[0122]

[0123] in The output of a specific classification head representing different modalities It is a signal conversion type label.

[0124] 2.1.2 View Construction Based on Hypergraph and Similarity Focus Enhancement

[0125] The multimodal cognitive representation of each student is obtained through cognitive representation learning based on multimodal classroom data. The graph constructed above based on students' basic information can then be used as a reference. Further updated to include cognitive maps for students in online classrooms. ,in It includes multimodal cognitive representations for all students. However, analyzing students' cognitive abilities solely based on their own experiences and current environment has certain limitations. Therefore, this embodiment designs various graph augmentation techniques to construct comparative views of students' cognitive maps. These views are crucial for self-supervised learning tasks, as they can help the model uncover deeper cognitive semantic information for each student.

[0126] Interactions between students can influence the cognition of both students involved. Inspired by this, this embodiment uses the original cognitive map... Perform dual hypergraph transformations to construct a hypergraph view of the cognitive graph. A hypergraph is a graph where edges can connect multiple nodes. Dual hypergraph transformations can migrate the message passing mechanism between nodes in a traditional graph to edges. Specifically, this involves the nodes of a cognitive graph. It was transformed into a hyperedge And on the side It was transformed into a supernode This method allows for the capture of cognitive information contained in interactions between students.

[0127] In addition, this embodiment will use the original cognitive map It is also defined as a global graph, and a focal graph is constructed based on similarity sampling. This method can improve the robustness of the embedding. Unlike previous K-nearest neighbor-based sampling methods, this embodiment takes into account the varying interaction densities between students and others due to differences in student personalities. The method used in this embodiment first calculates the similarity matrix of the original attribute network. ,in Represents a node The similarity score can generally be set as the inner product of features, and then when the similarity score of any pair of nodes in the graph is greater than a specified threshold... We retain the edges between the node pairs when they are not present, and delete them otherwise. The formulaic expression for obtaining the focused view structure is as follows:

[0128]

[0129] in Let be the adjacency matrix of the global graph.

[0130] To comprehensively assess students' cognitive status, in addition to considering the consistency of students' cognition with the class as a whole, this embodiment also considers the cognitive information of students' core social circles. The system first constructs a corresponding focal graph based on the global graph, and then uses a random walk strategy to sample the subgraphs, obtaining the subgraph of each node from the student global graph and focal graph of the online classroom, and further constructing the sampled global graph subgraphs into the aforementioned hypergraph view.

[0131] Therefore, the processing flow executed in step 2.1.2 above can constitute the multi-view construction module in the cognitive abnormality student detection module.

[0132] 2.1.3 Multi-granularity feature extraction under multi-contrast strategy

[0133] To capture student cognitive information at multiple levels, this embodiment constructs a multi-strategy contrastive network based on multiple augmented views. It mainly comprises three parts of contrastive learning: focus-global contrast, individual-interaction contrast, and history-current contrast. For subsequent contrastive learning to proceed, features of the augmented views under the multiple contrastive strategies need to be extracted.

[0134] The focus-global contrast method comprises two views: a global graph and a focus graph. This module employs a node-subgraph contrast method on both views to calculate discrimination scores. The main difference between the two views lies in the different readout strategies used. Furthermore, to prevent the target node information in the subgraph from influencing discrimination, the node attributes corresponding to the target node of the student to be detected in the subgraph are anonymized. The overall framework of the two views is roughly similar. First, in the global view, the target node is used as the starting sampling point. A subgraph is sampled through a random walk and mapped to a low-dimensional embedding via a graph convolutional network.

[0135]

[0136] in For all nodes in the subgraph Layered hidden representation, with a total of L layers. It is the normalization of the adjacency matrix. It is a normalization of the angle matrix. Represents the learnable parameter matrix, The activation function is used. The average readout function is applied to average the readouts of the latent representations of all nodes within the subgraph:

[0137]

[0138] In the formula: P is the total number of nodes in the subgraph. This represents a subgraph-level representation within the global view.

[0139] Adopting the above Compute the same graph convolutional network to map the target nodes corresponding to the students to be detected to the same dimension:

[0140]

[0141] in For the target node's first Layer hidden representation, initial hidden representation That is, the multimodal cognitive representation of the target node. .make The target node is ultimately embedded in the subgraph of the global view.

[0142] For the sampled subgraph of the focused view, the target node is used as the starting sampling point under the focused view. A subgraph is sampled through random walk and mapped to a low-dimensional embedding through a graph convolutional network. The graph convolutional network uses a different graph convolutional network with different parameters than that in the global view to obtain the representation on the focused view subgraph.

[0143]

[0144] The meanings of the parameters in the formula are basically the same as those in the global graph.

[0145] To minimize the impact of noise on graph-level embeddings, we abandon average readout and propose a simple yet effective similarity-based attention mechanism in the focused view, which adaptively selects useful embeddings for subgraphs. Intuitively, the more similar a node in a subgraph is to the target node, the more weight it receives during pooling, and vice versa. To allow the module to adaptively utilize similarity information, a multilayer perceptron mapping is applied to the similarity vector to obtain the weight vector. The formula is expressed as:

[0146]

[0147] in is a learnable parameter, and b is the bias. Let the similarity vector be the similarity vector between the target node and the other nodes, based on similarity... get As an attention mechanism factor, it is applied to the process of obtaining the focused view subgraph-level embedding:

[0148]

[0149] Similarly, the same approach is adopted. Compute the same graph convolutional network to map the target node to the same dimension on the focused view:

[0150]

[0151] in For the target node's first Layer hidden representation, initial hidden representation That is, the multimodal cognitive representation of the target node. .make The target node of the focused view is ultimately embedded.

[0152] Individual-interaction contrast is defined between the global view and the hypergraph view. This helps to fully consider the interactive information in multimodal classroom data, resulting in more representative student cognitive representations. The subgraph-level representation of the global view has already been obtained above; the subgraph-level representation of the hypergraph view is obtained through a hypergraph neural network and a readout function.

[0153] For the subgraph obtained by random walk sampling of the target node under the global view, it is transformed into a subgraph under the hypergraph view through dual hypergraph transformation. The subgraph information is first encoded by a hypergraph neural network:

[0154]

[0155] in and These are diagonal matrix representations of the degrees of the supernodes and superedges, respectively, representing the number of superedges to which a supernode belongs and the number of nodes contained in a superedge. The association matrix of a hypergraph view is used to represent the structural information of the hypergraph. is a learnable parameter matrix.

[0156] Obtain the output of the subgraph under the hypergraph view in the last layer of the hypergraph neural network. The final hidden representations of all supernodes in the hypergraph neural network are averaged by the supernode dimension. This means that the final hidden representations of all supernodes are read out by averaging, thus obtaining the subgraph-level representation of the target node under the hypergraph view. The input to a hypergraph neural network is a representation of the original edges. For simplicity, it defaults to encoding the specific time when the interaction occurs and the duration of the interaction.

[0157] Unlike traditional graph contrastive learning methods, individual-interaction contrastive learning uses target nodes in the hypergraph view. Subgraph-level representation and target node in global view The subgraph-level representation constitutes positive sample pairs, with the target node in the hypergraph view. Subgraph-level representation and other nodes in the global view The subgraph-level representation constitutes the negative sample pair.

[0158] Finally, based on the widely accepted assumption that students' cognition does not change significantly in a short period of time, this embodiment designs a historical node-current node comparison. Specifically, it compares the target node representation obtained by processing the student's cognitive global view through a graph convolutional network at a previous time (corresponding to a specific online course) with the target node representation obtained by processing the student's cognitive global view through a graph convolutional network at the current time. This is more conducive to the task of detecting students with cognitive abnormalities. After each course, the node-level representation of the global view learned in this course is... Stored in a memory bank, and retrieving different moments from the historical moments stored in the memory bank. average Used for comparing historical nodes with the current node.

[0159] The multi-policy contrastive network based on multiple augmented views ultimately yields multi-granularity representations of all target nodes under multiple augmented views. (The last part, "target node," appears to be incomplete and lacks context. It's left as is.) For example, we obtain the subgraph-level representations on the global view, focused view, and hypergraph view respectively. Node-level representation on the subgraph of the global view and the focused view and historical node-level representation Where dense, spars, and hyper represent the global view, focused view, and hypergraph view, respectively. The above results are used for subsequent model training and anomaly score calculation.

[0160] Therefore, the three granularity feature extraction processes executed in step 2.1.3 above can constitute the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module in the cognitively abnormal student detection module.

[0161] 2.3 Cognitive Representation Learning and Anomaly Detection Model Training and Anomaly Score Calculation

[0162] 2.3.1 Training of the Unsupervised Multimodal Representation Learning Module

[0163] This embodiment innovatively proposes generating a dataset with pseudo-labels through signal transformation. This is used for pre-training of the unsupervised multimodal representation learning module. This embodiment applies five signal transformation methods to multimodal data: permutation, time warp, noise addition, amplitude warp, and cropping. These transformation methods are applied to video, audio, and text modalities, and the generated transformed signal data is input into the multimodal representation learning module along with the original multimodal signal data. The above signal transformation process can be described as follows.

[0164] 1) Gaussian noise addition: The input signal is Gaussian white noise interference is The noise is taken from a zero-mean normal distribution, and the variance of this normal distribution is expressed by the formula... The calculation yielded, where It is the average power of the signal, and the final output is ;

[0165] 2) Amplitude distortion: The amplitude of the original signal is determined by a cubic spline interpolation function. The resulting random smooth curve changes, and the final output is .

[0166] 3) Arrangement: The original signal is divided into The process involves creating non-overlapping segments, which are then shuffled in time and eventually reassembled.

[0167] 4) Time warp: The original signal is split into... Non-overlapping segments, half of which are randomly selected by a linear interpolation function. Stretch, the other half is made of Extrusion, in which Let k be the i-th segment of the original signal, and k be the stretching factor. Finally, the two halves are reassembled.

[0168] 5) Pruning: The original signal is divided into n non-overlapping segments, one of which is randomly divided and then reassembled to the original length.

[0169] After signal transformation, the signal type is used as the pseudo-label for the corresponding data. All data with pseudo-labels are merged into a pseudo-label dataset, which is used for training the multimodal learning representation module, i.e., for training the modality-specific encoder. and shared encoders and modality-specific classification heads Training loss The calculation formula is as follows:

[0170]

[0171]

[0172] in Indicates the signal conversion category label, This represents the total loss of the unsupervised multimodal cognitive representation learning module.

[0173] 2.3.2 Training and Cognitive Abnormality Score Calculation of the Module for Detecting Students with Cognitive Abnormalities

[0174] Each student node is obtained through the cognitive anomaly detection module. After multi-view, multi-level representation, the similarity between positive and negative sample pairs is calculated, and the loss of each of the three comparison strategies is calculated.

[0175] For focused-to-global contrastive learning, this embodiment employs a node-subgraph comparison method on the focused view and the global view. In the focused view, the target node is used... Subgraph-level representation and target node The node-level representation constitutes a positive sample pair, the target node. Subgraph-level representation of other nodes The node-level representation constitutes the negative sample pair; in the global view, the target node in the global view... Subgraph-level representation and target node The node-level representation constitutes a positive sample pair, the target node. Subgraph-level representation of other nodes The node-level representation constitutes negative sample pairs;

[0176] Unlike previous methods, this embodiment uses the same discriminator on both views, which enhances the information interaction between the two views. Specifically, it uses a dual linear model (i.e., a bilinear layer). The similarity of the positive and negative sample pairs embedded in the focused view and the global view were obtained respectively:

[0177]

[0178]

[0179] in It is a shared, learnable weight matrix. and These represent the global view and the focused view, respectively.

[0180] In view For global view At that time, the similarity of positive sample pairs will be... Similarity with negative sample pairs Find the difference and use it as an outlier score.

[0181]

[0182] The goal of focused-global contrastive learning is to make the similarity score of positive sample pairs closer to 1, and vice versa. Therefore, in this embodiment, the loss function of focused-global contrastive learning is defined as the contrast loss between the subgraph-level representation and the node-level representation in the global view. Comparison loss between subgraph-level and node-level representations in the focused view sum:

[0183]

[0184]

[0185] For historical node-current node comparison learning, this embodiment will store the target student node in the memory. In history courses, the average node-level representation obtained through graph convolutional networks With the current course Compare the embedding similarity of positive sample pairs. Embedding similarity with negative sample pairs They are respectively:

[0186]

[0187]

[0188] in Discriminator in Focused-Global Contrast Learning shared.

[0189] Because of the general consistency between individual cognition, small group cognition, and group cognition in the classroom, this embodiment defines the historical node-current node contrast loss as:

[0190]

[0191] When the comparison pair is a positive sample pair Set to 1 when the comparison pair is a negative sample pair. Set to 0 and Replace with The sum of the positive and negative sample comparison losses is the final result. .

[0192] Finally, for individual-interaction comparative learning, the target nodes of student-to-student interaction will be addressed separately. A comparison is made between subgraph-level embeddings on a hypergraph view and subgraph-level embeddings on a global view targeting an individual student. Specifically, the target node in the hypergraph view... Subgraph-level representation and target node in global view The subgraph-level representation constitutes positive sample pairs, with the target node in the hypergraph view. Subgraph-level representation and other nodes in the global view The subgraph-level representation constitutes the negative sample pair.

[0193] In this embodiment, the individual-interaction contrast loss The contrast loss, defined as the subgraph-level representation in the hypergraph view and the global view, is as follows:

[0194]

[0195] Taking into account the above losses, the total loss for training the cognitively abnormal student detection module is obtained as follows:

[0196]

[0197] in , , This is a hyperparameter.

[0198] This embodiment comprehensively integrates anomaly information from the perspectives of multiple enhanced views and multiple comparison strategies, and additionally considers the similarity of cross-view comparison of the target student node-level embedding in the focused view and the global view:

[0199]

[0200] As one of the bases for calculating abnormal scores, the scoring module includes a section for students... The cognitive impairment score can be expressed as:

[0201]

[0202] Where δ is a hyperparameter.

[0203] It should be noted that in calculating the above... It is necessary to calculate first. , and The calculation of these three similarity coefficients requires not only the target node. The information also requires the use of other nodes. Information from other nodes Only one node in the diagram needs to be selected. In this embodiment, the other nodes... Information can be obtained from the entire graph and the target node. The node number is adjacent to the node.

[0204] Since the random walk strategy used when sampling subgraphs has a certain degree of randomness, this embodiment performs multiple rounds of detection under a multi-round random walk setting, and calculates the average of the above-mentioned anomaly detection scores over multiple rounds. Used to determine the final degree of cognitive abnormality in students, among which For students The final overall cognitive abnormality score. Therefore, this process of calculating the student's final degree of cognitive abnormality constitutes the scoring module in the cognitive abnormality student detection module. The cognitive abnormality student detection module used in the actual reasoning stage can be composed of the aforementioned multi-view construction module, global feature extraction module, focused feature extraction module, hypergraph feature extraction module, and scoring module.

[0205] 3. Detection of students with cognitive impairments in online learning

[0206] 1) Transformer-based multimodal cognitive representation learning

[0207] First, the unsupervised training method proposed in this embodiment is based on loss. The multimodal representation learning module is pre-trained; then, a large-scale multimodal teaching data set of students learning online is used as the input sample and fed into the trained multimodal representation learning module. Finally, the cognitive representation of each student obtained from the multimodal data of the online classroom is used for the downstream cognitively abnormal student detection task.

[0208] 2) Detection of Cognitive Abnormal Students Based on Multi-Strategy Contrast with Multiple Augmented Views

[0209] The student cognitive representations obtained from the multimodal representation learning module are used as student features. A comprehensive view of student cognition is constructed based on student interactions collected in the classroom. Subsequently, multi-strategy contrastive learning based on multiple augmented views is performed to obtain a score for the degree of cognitive abnormality of each student, as well as the manifestation of cognitive abnormality at various levels, such as the cognitive abnormality exhibited by students in their core social circles that are highly similar to their own.

[0210]

[0211] in: This indicates the degree of abnormality in the target student's cognition compared to the class's overall cognition;

[0212] This indicates the degree of abnormality exhibited by the target student based on differences in cognition displayed in the class and core social circles;

[0213] This indicates the degree of cognitive abnormality of the target student over time.

[0214] This represents the overall cognitive abnormality score of the target student obtained by weighting the above three factors.

[0215] The primary objective of this invention is to assess the degree of cognitive abnormality in each student, thereby assisting teachers in quickly and accurately identifying students with cognitive abnormalities in large-scale online learning scenarios. Therefore, applying unsupervised multimodal cognitive representation learning and multi-view, multi-strategy contrastive cognitive abnormality detection to the field of education is highly practical and helps promote students' cognitive development in online learning environments.

[0216] To verify the technical effectiveness of the proposed method, it was validated in a real-world online learning scenario. The dataset used in this embodiment is a multimodal educational dataset collected during online classes, recording multimodal data from students' learning process. During the courses using this dataset, instructors arranged appropriate discussion, Q&A, and written assignment sessions to obtain data from multiple modalities. Furthermore, to more intuitively demonstrate the ability of the proposed model to detect cognitively abnormal students, a certain number of lower-grade students were included in the class as typical examples of students to be detected with cognitive abnormalities. Students' basic personal information was collected before the course began as initial features for each student, and student interactions were implemented using online devices for real-time data collection. In the constructed student relationship graph, each student in this dataset was constructed as a node, and each interaction between students was constructed as an edge in the graph.

[0217] In this embodiment, the evaluation metric chosen is AUC, the area under the ROC curve, which is most commonly used in anomaly detection. The value ranges from 0 to 1, with a larger value indicating better model performance. Table 1 compares the method of this invention with unsupervised node-level anomaly detection methods for static graphs. For fairness, subsequent experiments were conducted on the aforementioned multimodal education dataset with an anomaly rate of 10%, and the cognitive representation obtained using the method of this invention was used as the node features of the input graph for the benchmark methods.

[0218] Table 1. Results of the method of this invention in detecting cognitively abnormal students on a multimodal educational dataset.

[0219]

[0220] Experimental results show that the method of the present invention is significantly improved compared with the benchmark method. One major reason is that the method of the present invention adopts multiple comparison strategies and effectively models the dependency relationship between different influencing factors of students' cognition. The aforementioned comparison methods include: Dominant, an anomaly detection model based on graph autoencoders, derived from existing literature: Ding K, Li J, Bhanushali R, et al. Deep anomaly detection on attributed networks[C] / / Proceedings of the 2019 SIAM international conference on datamining. Society for Industrial and Applied Mathematics, 2019: 594-602.; CoLA, an anomaly detection model based on node-level contrast, derived from existing literature: Liu Y, Li Z, Pan S, et al. Anomaly detection on attributed networks via contrastive self-supervised learning[J]. IEEE transactions on neural networks and learning systems, 2021,33(6): 2378-2392.; and SL-GAD, an anomaly detection model based on contrastive and generative methods, derived from existing literature: Zheng Y, Jin M, Liu Y, et al. Generative and contrastive self-supervised learning for graph anomaly detection[J]. IEEE Transactions on Knowledge. andData Engineering, 2021, 35(12): 12220-12233.

[0221] Furthermore, this embodiment conducted ablation studies to better examine the contributions of the different modules proposed in this invention. The experimental results are shown in Table 2, and the specific details of each ablation variant are as follows:

[0222] w / o Multimodal cognition: Based on the complete framework (S1~S3) proposed in this invention, the multimodal representation learning module is removed, and the student's basic information is used as the initial node features of the input graph data. ;

[0223] w / o Focus-Global: Based on the complete framework proposed in this invention, the focus-global comparison strategy is not adopted;

[0224] w / o Individual-interaction: Based on the complete framework proposed in this invention, the individual-interaction comparison strategy is not adopted;

[0225] w / o Historical Node-Current Node: Based on the complete framework proposed in this invention, the historical node-current node comparison strategy is not adopted.

[0226] Table 2 Ablation experiment results of different modules of the present invention on a multimodal educational dataset.

[0227]

[0228] The ablation experiments demonstrate that the multiple contrast strategies based on enhanced views proposed in this invention are highly effective. This is primarily because the contrast strategies are specifically designed for online learning scenarios, enabling the analysis of student cognition from multiple perspectives within the online classroom. Furthermore, the multimodal cognitive representation learning module significantly impacts the experimental results, illustrating its effectiveness in extracting general cognitive representations of students from multimodal educational data in unsupervised environments. This is mainly attributed to its signal transformation-based unsupervised training method and its multimodal feature extraction and fusion module.

[0229] In summary, this invention provides an efficient and accurate solution for detecting cognitively abnormal students in unsupervised online learning scenarios.

[0230] The embodiments described above are merely some preferred implementations of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A multi-modal representation based online learning cognitive abnormal student detection method, characterized in that, The method comprises the following steps: S1, obtaining multi-modal data of each student in the target online course, the multi-modal data comprising video data recording facial expressions and body behaviors of the student, audio data recording sounds of the online classroom, and text data recording discussions of the online classroom; taking all students as nodes, interactions between the students as edges, and basic information of the students registered on the platform as node features, constructing a student relationship graph; S2, inputting the multi-modal data of each student in the target online course into a multi-modal learning representation module pre-trained through an auxiliary classification task to obtain multi-modal embeddings corresponding to each student; in the multi-modal learning representation module, first, for video, audio and text data in each multi-modal data, a modal-specific encoder is used to encode different modal data to obtain respective single-modal embeddings; then, the single-modal embeddings of the three modalities are spliced and input into a shared encoder based on Transformer to be fused into multi-modal embeddings; S3, inputting the multi-modal embeddings corresponding to all students in the target online course and the student relationship graph into a cognitive abnormal student detection module pre-trained through a contrast learning framework to obtain a cognitive abnormality score of the target node corresponding to the student; The cognitive abnormal student detection module comprises a multi-view construction module, a global feature extraction module, a focused feature extraction module, a hypergraph feature extraction module and a scoring module; the multi-view construction module is used to construct a global view by combining the student relationship graph and the multi-modal embeddings, and simultaneously convert the global view into a focused view through edge filtering operation of node similarity; in the global feature extraction module, a subgraph-level representation and a node-level representation of the target node in the global view are generated for the target node corresponding to the student to be detected; the focused feature extraction module is used to generate a subgraph-level representation and a node-level representation of the target node in the focused view for the target node corresponding to the student to be detected; the hypergraph feature extraction module is used to generate a subgraph-level representation of the target node in a hypergraph view through a dual hypergraph transformation and a hypergraph neural network; in the scoring module, according to the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module and the hypergraph feature extraction module under multiple rounds of random walks, combined with historical information in a memory bank, abnormal scores are obtained through a bilinear layer in multiple dimensions, and finally a cognitive abnormality total score of the target node is obtained by fusion; The internal processes of the sub-modules of the cognitive abnormal student detection module are as follows: In the multi-view construction module, the node features of the student nodes in the student relationship graph are replaced by the corresponding multi-modal embeddings to form a global view; the edge connections in the global view are filtered based on the feature similarity between the nodes, and the edges with a feature similarity between the nodes lower than a threshold value are deleted, so as to convert the global view into a focused view. In the global feature extraction module, for the target node corresponding to the student to be detected, the target node is taken as a starting sampling point in a global view, a subgraph is sampled through random walk and input into a first graph convolution network to generate hidden representations through layer-by-layer aggregation, the hidden representations of all nodes finally output by the first graph convolution network are averaged in node dimension, and a subgraph-level representation of the target node in the global view is obtained; meanwhile, the node features of the target node in the global view are input into the first graph convolution network, and the node features are mapped into a node-level representation of the target node in the global view. In the focused feature extraction module, for the target node corresponding to the student to be detected, the target node is taken as a starting sampling point in a focused view, a subgraph is sampled through random walk and input into a second graph convolution network to generate hidden representations through layer-by-layer aggregation, the similarity of each node relative to the target node is calculated based on the hidden representations of all nodes finally output by the second graph convolution network, and then the similarity is mapped into a weight through a multi-layer perception, and then the hidden representations of all nodes finally output by the second graph convolution network are fused through weighting, to obtain a subgraph-level representation of the target node in the focused view; meanwhile, the node features of the target node in the focused view are input into the second graph convolution network, and the node features are mapped into a node-level representation of the target node in the focused view. In the hypergraph feature extraction module, the subgraph collected in the global feature extraction module is first subjected to a dual hypergraph transformation to obtain a subgraph in a hypergraph view and input into a hypergraph neural network for encoding, and the hidden representations of all hypernodes finally output by the hypergraph neural network are averaged in hypernode dimension to obtain a subgraph-level representation of the target node in the hypergraph view. In the scoring module, based on the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module and the hypergraph feature extraction module, the similarity of the node-level representations of the target node and other nodes in the global view is calculated through a bilinear layer as a first-dimension abnormal score; the similarity of the node-level representations of the target node in the current online course and historical online courses in the global view is calculated through a bilinear layer as a second-dimension abnormal score; the similarity of the node-level representations of the target node in the global view and the focused view is calculated through a bilinear layer as a third-dimension abnormal score; and the abnormal scores in the three dimensions are weighted and summed to obtain a cognitive abnormal total score of the student corresponding to the target node in the target online course. In S2, when the multi-modal learning representation module is pre-trained through an auxiliary classification task, a signal conversion module is additionally set in the training stage to perform different category signal conversions on the single-modal embeddings of each modality in the multi-modal data, and then the signal conversion categories of each modality are predicted based on the signal-converted single-modal embeddings through a specific modality classification head, so that the multi-modal learning representation module is optimized through a classification cross-entropy loss. In S3, when the cognitive abnormal student detection module is trained through a contrast learning framework, focused-global contrast learning, history-current contrast learning and individual-interaction contrast learning need to be performed, and the weighted sum of the respective contrast learning losses is taken as a total loss function. In the focus-global contrast learning, positive and negative sample pairs are constructed for the focus view and the global view respectively; In the focus view, a positive sample pair is formed by the subgraph-level representation of the target node and the node-level representation of the target node in the focus view, and a negative sample pair is formed by the subgraph-level representation of the target node and the node-level representation of other nodes; in the global view, a positive sample pair is formed by the subgraph-level representation of the target node and the node-level representation of the target node in the global view, and a negative sample pair is formed by the subgraph-level representation of the target node and the node-level representation of other nodes; In the history-current contrast learning, a positive sample pair is formed by the node-level representation of the target node in the global view of the current target online course and the average of the node-level representations of the target node in the global view of all historical online courses stored in the memory bank, and a negative sample pair is formed by the node-level representation of the target node in the global view of the current target online course and the average of the node-level representations of other nodes in the global view of all historical online courses stored in the memory bank; In the individual-interaction contrast learning, a positive sample pair is formed by the subgraph-level representation of the target node in the supergraph view and the subgraph-level representation of the target node in the global view, and a negative sample pair is formed by the subgraph-level representation of the target node in the supergraph view and the subgraph-level representation of other nodes in the global view.

2. The multimodal representation based online learning cognitive anomaly student detection method of claim 1, wherein, The signal conversion categories in the signal conversion module are divided into five categories, namely, permutation, time warping, noise addition, amplitude warping and cropping.

3. The online learning cognitive anomaly student detection method based on multi-modal representation of claim 1, wherein, In the cognitive abnormal student detection module, the global feature extraction module, the focus feature extraction module and the focus feature extraction module need to perform multiple rounds of random walk, and the feature representations generated by the multiple rounds of random walk are input into the scoring module to calculate the abnormal scores in three dimensions and the total cognitive abnormal score, and then the same type of scores of the multiple rounds of random walk are averaged to output the abnormal scores in three dimensions and the total cognitive abnormal score in the form of average scores.

4. The online learning cognitive anomaly student detection method based on multi-modal representation of claim 1, wherein, The multi-modal learning representation module is first independently trained, and after training, the learning parameters in the module are fixed and used to generate the multi-modal embedding corresponding to each student, and then the cognitive abnormal student detection module is independently trained.

5. A multi-modal representation based online learning cognitive anomaly student detection system, characterized in that, It comprises: a multi-modal data acquisition module for acquiring multi-modal data of each student during the learning process in each online course on an online learning platform; a detection object specifying module for allowing a management user to specify a student user who needs to be detected for cognitive abnormalities through a GUI interface; a cognitive abnormality detection module for obtaining the multi-modal data corresponding to the specified student user from the multi-modal data acquisition module according to the student user specified in the detection object specifying module, and then obtaining the cognitive abnormality score of the specified student user according to the multi-modal representation-based online learning cognitive abnormal student detection method of any one of claims 1-4; a memory bank module for storing data generated by the other three system modules in historical online courses on an online learning platform.

6. A computer readable storage medium characterized by, The storage medium stores a computer program, and when the computer program is executed by a processor, the multi-modal representation-based online learning cognitive abnormal student detection method of any one of claims 1-4 is realized. The storage medium stores a computer program, and when the computer program is executed by a processor, the multi-modal representation-based online learning cognitive abnormal student detection method of any one of claims 1-4 is realized.

7. A computer electronic device, comprising: The method comprises the following steps: The memory is used for storing a computer program; The processor is used for realizing the method for detecting cognitive abnormal students in online learning based on multi-modal representation according to any one of claims 1-4 when the computer program is executed.

Images