Online learning cognitive anomaly student detection method and system based on multi-modal representation

Abstract

The invention discloses an online learning cognitive anomaly student detection method and system based on multi-modal representation, and belongs to the field of anomaly detection. The method comprises the following steps: firstly, acquiring multi-modal data in a student learning process, and constructing a student relation graph; then inputting the multi-modal data of each student into a multi-modal learning representation module pre-trained by an auxiliary classification task to obtain multi-modal embedding corresponding to each student; finally, the multi-mode embedding and student relation graph is input into a cognitive anomaly student detection module which is trained through a comparative learning framework in advance, and cognitive anomaly scores of students corresponding to the target nodes are obtained.According to the method, student cognitive information can be automatically extracted, heavy work of manual processing is avoided, and the learning efficiency is improved. And meanwhile, cognitive anomaly student detection is realized based on multi-view multi-strategy comparison, the cognitive level of students on the platform can be detected on the basis of considering multiple influence factors, and the online learning effect is assisted to be improved.

Description

Technical Field

[0001] The present invention belongs to the field of anomaly detection, and specifically relates to a method and system for detecting students with cognitive anomalies in online learning based on multimodal representation. Background Art

[0002] Online learning is a learning method that uses computer Internet or mobile network for remote teaching, and students can interact with teachers in virtual classrooms. In recent years, with the rapid development of Internet technology, online learning has gradually become a mainstream teaching mode. With the continuous expansion of the scale of online learning, a large amount of educational data has accumulated in the classroom, providing new possibilities for personalized cognitive support for students. The large amount of multimodal educational data (such as text, audio, video, etc.) generated in online classrooms is closely related to the cognitive state of students. However, in large-scale online learning scenarios, manual processing of these multimodal information is time-consuming and labor-intensive, making it difficult for teachers to comprehensively and timely analyze the cognitive status of each student. In addition, students' cognitive deficiencies are often affected by many factors and have various manifestations. Therefore, it is almost impossible for teachers to quickly and accurately identify students with weak cognition in large-scale classrooms and provide targeted guidance. Therefore, how to efficiently process multimodal data and identify students' cognitive abnormalities has become a key problem in online education.

[0003] The data in online classes has graph structure characteristics. As a general data representation method, graph structure can be effectively applied to multimodal data modeling in online classroom scenarios. Therefore, cognitive abnormal student detection in online learning scenarios can be regarded as an unsupervised node-level anomaly detection task on a static graph based on multimodal representation learning. At present, multimodal representation learning methods mainly rely on supervised or semi-supervised learning methods, but in actual teaching scenarios, it is difficult to obtain real teaching data labels. The mainstream methods of unsupervised node-level anomaly detection can be divided into two categories: one is based on graph autoencoder methods, such as DOMINANT, which is based on the assumption that abnormal samples are more difficult to reconstruct and uses graph convolutional networks (GCN) to construct autoencoders to reconstruct attribute matrices and adjacency matrices to detect abnormal nodes; the other is based on contrastive learning methods, such as CoLA, which captures local information of network data by sampling instances and uses contrastive learning technology to learn node representations.

[0004] Although these methods have achieved certain results, they still have significant shortcomings: (1) Most current methods only conduct comparative learning from individual perspectives and lack consideration of all-round information. Students' cognition in online classes is often affected by multiple factors; (2) The interactive information between nodes is not fully utilized in anomaly detection, while the interaction between students in the classroom is crucial to cognitive changes; (3) The impact of the correlation between nodes on the attribute graph on the anomaly detection performance is not considered. There are different degrees of correlation between students in online classes, and these complex relationships will interfere with anomaly detection. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and provide a method and system for detecting students with cognitive abnormalities in online learning based on multimodal representation.

[0006] The specific technical solutions adopted by the present invention are as follows:

[0007] In a first aspect, the present invention provides a method for detecting students with cognitive abnormalities in online learning based on multimodal representation, which comprises:

[0008] S1. Obtain multimodal data of each student's learning process in the target online course, wherein the multimodal data includes video data recording students' facial expressions and body movements, audio data recording online classroom sounds, and text data recording online classroom interactive discussion texts; construct a student relationship graph by taking all students as nodes, interactions between students as edges, and basic information registered by students on the platform as node features;

[0009] S2. Input the multimodal data of each student in the target online course into a multimodal learning representation module pre-trained by the auxiliary classification task to obtain the multimodal embedding corresponding to each student; in the multimodal learning representation module, firstly, for each of the three modalities of video, audio and text in each multimodal data, respectively, use the modality-specific encoder to encode the data of different modalities to obtain their respective single-modal embeddings, and then splice the single-modal embeddings of the three modalities and input them into the Transformer-based shared encoder to fuse them into multimodal embeddings;

[0010] S3, input the multimodal embeddings and student relationship graphs corresponding to all students in the target online course into the cognitive abnormality student detection module pre-trained by the contrastive learning framework to obtain the cognitive abnormality score of the student corresponding to the target node;

[0011] The cognitive abnormal student detection module consists of a multi-view construction module, a global feature extraction module, a focused feature extraction module, a hypergraph feature extraction module and a scoring module; the multi-view construction module is used to combine the student relationship graph and the multimodal embedding to construct a global view, and at the same time, the global view is converted into a focused view through the edge filtering operation of node similarity; in the global feature extraction module, a subgraph-level representation of the target node under the global view is generated for the target node corresponding to the student to be detected And node-level representation The focus feature extraction module is used to generate a subgraph level representation of the target node under the focus view for the target node corresponding to the student to be detected. and node-level representation The hypergraph feature extraction module is used to generate a subgraph level representation of the target node under the hypergraph view through dual hypergraph transformation and hypergraph neural network. In the scoring module, based on the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module under multiple rounds of random walks, combined with the historical information in the memory bank, multi-dimensional anomaly scoring is performed through a bilinear layer, and the final total cognitive anomaly score of the target node is obtained by fusion.

[0012] As a preferred embodiment of the above-mentioned first aspect, in S2, when the multimodal learning representation module is pre-trained through an auxiliary classification task, the signal conversion module additionally set in the training phase is used to perform different categories of signal conversions on the unimodal embeddings of each modality in the multimodal data, and then the signal conversion categories of each modality are predicted through a specific modality classification head based on the unimodal embeddings after signal conversion, thereby optimizing the multimodal learning representation module through classification cross entropy loss.

[0013] As a preferred embodiment of the first aspect, the signal conversion categories in the signal conversion module are divided into five categories, namely, arrangement, time distortion, noise addition, amplitude distortion and clipping.

[0014] As a preferred embodiment of the first aspect, in S3, the internal processes of each submodule of the cognitive abnormal student detection module are as follows:

[0015] In the multi-view construction module, the node features of the student nodes in the student relationship graph are replaced with the corresponding multimodal embeddings to form a global view; the edge connections in the global view are screened based on the feature similarity between the nodes, and the edges whose feature similarity between the nodes is lower than the threshold are deleted, thereby converting the global view into a focused view;

[0016] In the global feature extraction module, for the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the global view, a subgraph is sampled by random walk and input into the first graph convolutional network for layer-by-layer aggregation to generate hidden representation, and the node dimension of all node hidden representations finally output by the first graph convolutional network is averaged to obtain the subgraph-level representation of the target node in the global view. ; At the same time, the node features of the target node in the global view are input into the first graph convolutional network to map it to the node-level representation of the target node in the global view ;

[0017] In the focus feature extraction module, for the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the focus view, a subgraph is sampled by random walk and input into the second graph convolution network for layer-by-layer aggregation to generate a hidden representation, the similarity of each other node relative to the target node is calculated based on the hidden representation of all nodes finally output by the second graph convolution network, and then the similarity is mapped to a weight through a multi-layer perceptron, and then the node hidden representation of all nodes finally output by the second graph convolution network is weighted fused to obtain the subgraph-level representation of the target node in the focus view. At the same time, the node features of the target node in the focused view are input into the second graph convolutional network to map it to the node-level representation of the target node in the focused view ;

[0018] In the hypergraph feature extraction module, the subgraph collected in the global feature extraction module is first transformed into a dual hypergraph to obtain the subgraph under the hypergraph view and input it into the hypergraph neural network for encoding. The supernode dimension is averaged for all supernode hidden representations finally output by the hypergraph neural network to obtain the subgraph-level representation of the target node under the hypergraph view. ;

[0019] In the scoring module, according to the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module, the node-level representation similarity between the target node and other nodes under the global view is calculated through a bilinear layer as the first dimension anomaly score; the node-level representation similarity between the target node in the current online course and the historical online course under the global view is calculated through a bilinear layer as the second dimension anomaly score; the node-level representation similarity between the target node under the global view and the focused view is calculated through a bilinear layer as the third dimension anomaly score; the anomaly scores of the three dimensions are weighted and summed to obtain the total cognitive anomaly score of the student corresponding to the target node in the target online course.

[0020] As a preferred embodiment of the above-mentioned first aspect, in the cognitive abnormality student detection module, the global feature extraction module, the focused feature extraction module and the focused feature extraction module need to perform multiple rounds of random walks, and the feature representations generated by the multiple rounds of random walks are respectively input into the scoring module to calculate the abnormal scores of the three dimensions and the total score of cognitive abnormalities, and then the scores of the same category of the multiple rounds of random walks are averaged, and the abnormal scores of the three dimensions and the total score of cognitive abnormalities are finally output in the form of average scores.

[0021] As a preferred embodiment of the first aspect, in S3, when training the cognitive abnormal student detection module through the contrastive learning framework, it is necessary to perform focus-global contrastive learning, history-current contrastive learning and individual-interaction contrastive learning, and use the weighted sum of the respective contrastive learning losses as the total loss function;

[0022] In the focus-global contrastive learning, positive and negative sample pairs are constructed for the focus view and the global view respectively; and in the focus view, the subgraph-level representation of the target node in the focus view and the node-level representation of the target node constitute the positive sample pair, and the subgraph-level representation of the target node and the node-level representation of other nodes constitute the negative sample pair; in the global view, the subgraph-level representation of the target node in the global view and the node-level representation of the target node constitute the positive sample pair, and the subgraph-level representation of the target node and the node-level representation of other nodes constitute the negative sample pair;

[0023] In the history-current comparative learning, the node-level representation of the target node in the global view of the current target online course is The positive sample pair is formed by the average value of the node-level representation of the target node in the global view of all historical online courses stored in the memory bank, and the node-level representation of the target node in the global view of the current target online course is The negative sample pairs are formed with the average of the node-level representations of other nodes in the global view of all historical online courses stored in the memory bank;

[0024] In the individual-interaction contrastive learning, the subgraph-level representation of the target node in the hypergraph view and the subgraph-level representation of the target node in the global view constitute positive sample pairs, and the subgraph-level representation of the target node in the hypergraph view and the subgraph-level representation of other nodes in the global view constitute negative sample pairs.

[0025] As a preferred embodiment of the above-mentioned first aspect, the multimodal learning representation module is first trained independently. After the training, the learning parameters in the module are fixed and used to generate the multimodal embedding corresponding to each student, and then the cognitive abnormality student detection module is independently trained.

[0026] In a second aspect, the present invention provides an online learning cognitive abnormal student detection system based on multimodal representation, which comprises:

[0027] The multimodal data collection module is used to collect multimodal data of each student’s learning process in each online course on the online learning platform;

[0028] The detection object designation module is used to allow the management user to designate the student users who need to be tested for cognitive abnormalities through the GUI interface;

[0029] A cognitive anomaly detection module, used to obtain multimodal data corresponding to the specified student user from the multimodal data acquisition module according to the student user specified in the detection object designation module, and then obtain a cognitive anomaly score for the specified student user according to the online learning cognitive anomaly student detection method based on multimodal representation as described in any one of the solutions of the first aspect above;

[0030] The memory module is used to store the data generated by the other three system modules in the history online courses on the online learning platform.

[0031] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting students with cognitive abnormalities in online learning based on multimodal representation is implemented as described in any one of the schemes of the first aspect above.

[0032] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0033] The memory is used to store computer programs;

[0034] The processor is used to implement the method for detecting students with cognitive abnormalities in online learning based on multimodal representation as described in any one of the solutions of the first aspect when executing the computer program.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) Aiming at the difficulty of obtaining labels in online education datasets, this paper designs an unsupervised multimodal representation learning framework, which automatically generates pseudo labels through signal conversion methods (such as noise addition, amplitude distortion, etc.) for pre-training of self-supervised learning models and extracting students' cognitive representations. At the same time, in view of the fact that the various modes of educational data are both related and heterogeneous, this paper designs an intermediate fusion method combining time convolution and Transformer to achieve effective fusion of multimodal data.

[0037] 2) This paper constructs online classroom data into a cognitive graph, models the cognitive anomaly detection problem as a node-level anomaly detection task on a static graph, and comprehensively analyzes the degree of cognitive anomaly of students through multiple enhanced views and multiple comparison strategies. This paper introduces three comparative learning strategies: focus-global comparison strategy (extracting key influencing factors), individual-interaction comparison strategy (considering rich interactions between students), and history-current comparison strategy (identifying abnormal changes in cognitive development) to more accurately identify students with cognitive anomalies.

[0038] 3) The present invention automatically extracts students' cognitive information through a multimodal representation learning module, avoiding the heavy work of manual processing; the cognitive abnormality student detection module based on multi-view and multi-strategy comparison can analyze the comprehensive cognitive level of each student based on considering multiple influencing factors, thereby helping to improve the effect of online learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the steps of a method for detecting students with cognitive abnormalities in online learning based on multimodal representation;

[0040] Figure 2 This is a schematic diagram of the module composition of the cognitive abnormal student detection module;

[0041] Figure 3 This is a schematic diagram of the module composition of the online learning cognitive abnormal student detection system based on multimodal representation;

[0042] Figure 4 A schematic diagram of the components of computer electronic equipment;

[0043] Figure 5 It is the module structure and training framework required for online learning of cognitive abnormality student detection in the embodiment. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0045] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0046] The present invention provides a method for detecting students with cognitive abnormalities in online learning based on multimodal representation, which makes core improvements in two aspects.

[0047] First, traditional multimodal representation learning methods mostly use supervised training. However, in real teaching environments, it is challenging to obtain accurate student cognitive labels, standard protocols have not yet been clearly defined, and the cost of data labeling is high. In view of this situation, the present invention extracts students' cognitive representations through unsupervised multimodal representation learning of signal conversion. Specifically, the present invention proposes an unsupervised multimodal representation learning framework, which has two key points: generating pseudo labels by signal conversion and cross-modal fusion strategy.

[0048] 1) Signal conversion to generate pseudo labels: The present invention designs a variety of signal conversion methods (such as noise addition, amplitude distortion, etc.) to perform signal conversion on unlabeled multimodal educational data and automatically generate pseudo labels for pre-training of self-supervised learning (SSL) models, thereby extracting generalized multimodal representations related to cognition and supporting downstream tasks.

[0049] 2) Cross-modal fusion strategy: In view of the fact that the various modalities of educational data are both related and heterogeneous, the present invention proposes an intermediate fusion method that combines temporal convolution and Transformer to model multimodal data with heterogeneity and cross-modal associations to achieve effective fusion.

[0050] Secondly, in online learning scenarios, traditional anomaly detection methods usually find it difficult to capture interactions between students, nor can they analyze the cognitive state of the target student based on factors that are more relevant to him or her, or consider his past cognitive changes, which leads to deficiencies in the accurate identification of cognitive anomalies. In response to this situation, based on the advantages of contrastive learning in unsupervised node-level anomaly detection tasks, the present invention designs a label-free online learning cognitive anomaly detection module based on multi-strategy comparison, which comprehensively considers multiple influencing factors in online classroom scenarios, thereby more accurately identifying students with cognitive anomalies. The present invention constructs online classroom data as a cognitive graph, models the cognitive anomaly detection problem as a node-level anomaly detection task on a static graph, and comprehensively analyzes the degree of cognitive anomalies of students through multiple enhanced views and multiple comparison strategies. The specific contrastive learning strategies in the present invention are divided into three types: focus-global comparison strategy, individual-interaction comparison strategy, and history-current comparison strategy,

[0051] 1) Focused-global comparison strategy: The present invention defines the original graph composed of multimodal educational data as a "global graph", in which each student node is connected to multiple peer nodes to reflect the comprehensive connection status of the student in the online classroom. However, educational practice shows that students' cognitive state is often mainly affected by a few core peers. Therefore, the present invention innovatively introduces the "focused-global comparison strategy" to extract a "focused graph" from the global graph through a similarity-based sampling method. In the focused graph, only adjacent nodes with a higher correlation with the cognitive state of the target student are retained, so as to extract the key factors that have a greater impact on students' cognition without destroying the overall topological structure. Through comparative analysis of the focused graph and the global graph, the system can more accurately identify cognitive anomalies and effectively improve the sensitivity and accuracy of detection.

[0052] 2) Individual-interaction comparison strategy: Traditional methods mainly focus on the information at the individual level of nodes, without considering the rich interactions between students. This invention extends the message passing mechanism between nodes to the social interaction perspective of students by constructing a dual hypergraph, thereby more comprehensively detecting students' cognitive abnormalities.

[0053] 3) History-Current Comparison Strategy: In online classes, each class is not completely independent. This invention incorporates historical information and proposes a history-current comparison method to compare the cognitive information of the same student at the current moment with that of previous courses to identify abnormal changes in cognitive development.

[0054] The specific implementation of the online learning cognitive abnormal student detection method based on multimodal representation of the present invention is described in detail below.

[0055] like Figure 1 As shown, in a preferred embodiment of the present invention, a method for detecting students with cognitive abnormalities in online learning based on multimodal representation is provided, which specifically includes steps S1 to S3. The specific implementation of steps S1 to S3 is described below.

[0056] S1. Obtain multimodal data of each student's learning process in the target online course, wherein the multimodal data includes video data recording students' facial expressions and body movements, audio data recording online classroom sounds, and text data recording online classroom interactive discussion texts; construct a student relationship graph by taking all students as nodes, interactions between students as edges, and basic information registered by students on the platform as node features.

[0057] It should be noted that the interaction between students refers to the interactive behavior of students in the course. If there is interaction between two students, the nodes of these two students in the student relationship graph can be connected by edges.

[0058] S2. Input the multimodal data of each student in the target online course into a multimodal learning representation module that has been pre-trained by the auxiliary classification task to obtain the multimodal embedding corresponding to each student; in the multimodal learning representation module, firstly encode the three modalities of video, audio and text in each multimodal data using modality-specific encoders to obtain their respective single-modal embeddings, and then concatenate the single-modal embeddings of the three modalities and input them into the Transformer-based shared encoder to fuse them into multimodal embeddings.

[0059] In an embodiment of the present invention, a modality-specific encoder refers to an independent encoder for each modality, and encoders between different modalities are not shared. The modality-specific encoder uses a temporal convolutional network (TCN) as the backbone network, and cascades a multi-layer perceptron and layer normalization after the backbone network to obtain the encoded output. The original data of the three modes of video, audio and text in the multimodal data needs to be pre-processed before being input into the modality-specific encoder to meet the requirements of the modality-specific encoder. Data preprocessing belongs to the prior art and will not be repeated here.

[0060] S3. Input the multimodal embeddings and student relationship graph corresponding to all students in the target online course into the cognitive abnormality student detection module pre-trained by the contrastive learning framework to obtain the cognitive abnormality score of the student corresponding to the target node.

[0061] like Figure 2 As shown, the cognitive abnormal student detection module designed by the present invention is composed of a multi-view construction module, a global feature extraction module, a focused feature extraction module, a hypergraph feature extraction module and a scoring module.

[0062] 1) A multi-view construction module is used to combine the student relationship graph and the multimodal embedding to construct a global view, and at the same time, convert the global view into a focused view through an edge filtering operation of node similarity.

[0063] In the embodiment of the present invention, the internal process of the multi-view construction module is as follows:

[0064] The node features of the student nodes in the student relationship graph are replaced with the corresponding multimodal embeddings to form a global view. The edge connections in the global view are filtered based on the feature similarity between nodes, and the edges whose feature similarity between nodes is lower than the threshold are deleted, thereby converting the global view into a focused view.

[0065] 2) Global feature extraction module, which is used to generate a subgraph-level representation of the target node under the global view for the target node corresponding to the student to be detected And node-level representation .

[0066] In the embodiment of the present invention, the internal process of the global feature extraction module is as follows:

[0067] For the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the global view. A subgraph is sampled by random walk and input into the first graph convolutional network for layer-by-layer aggregation to generate hidden representations. The node dimension of all node hidden representations finally output by the first graph convolutional network is averaged to obtain the subgraph-level representation of the target node in the global view. ; At the same time, the node features of the target node in the global view are input into the first graph convolutional network to map it to the node-level representation of the target node in the global view .

[0068] 3) Focus feature extraction module, which is used to generate a subgraph-level representation of the target node under the focus view for the target node corresponding to the student to be detected and node-level representation .

[0069] In the embodiment of the present invention, the internal process of the focus feature extraction module is as follows:

[0070] For the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the focused view, a subgraph is sampled by random walk and input into the second graph convolutional network for layer-by-layer aggregation to generate hidden representations, and the similarity of each other node relative to the target node is calculated based on the hidden representations of all nodes finally output by the second graph convolutional network. Then, the similarity is mapped to a weight through a multi-layer perceptron, and then the node hidden representations of all nodes finally output by the second graph convolutional network are weighted fused to obtain the subgraph-level representation of the target node in the focused view. At the same time, the node features of the target node in the focused view are input into the second graph convolutional network to map it to the node-level representation of the target node in the focused view ;

[0071] 4) Hypergraph feature extraction module, which is used to generate subgraph-level representation of target nodes under the hypergraph view through dual hypergraph transformation and hypergraph neural network .

[0072] In an embodiment of the present invention, the internal process of the hypergraph feature extraction module is as follows:

[0073] First, the subgraph collected in the global feature extraction module is transformed into a dual hypergraph to obtain the subgraph under the hypergraph view and input into the hypergraph neural network for encoding. The supernode dimension is averaged for all supernode hidden representations finally output by the hypergraph neural network to obtain the subgraph level representation of the target node under the hypergraph view. ;

[0074] 5) The scoring module is used to perform multi-dimensional anomaly scoring through a bilinear layer based on the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module under multiple rounds of random walks, combined with the historical information in the memory bank, and finally obtain the total cognitive anomaly score of the target node.

[0075] In the embodiment of the present invention, the internal process of the scoring module is as follows:

[0076] According to the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module respectively, the node-level representation similarity between the target node and other nodes (which can be any node except the target node, and the node with the closest number can be used) under the global view is calculated through a bilinear layer as the first dimension anomaly score; the node-level representation similarity of the target node in the current online course and the historical online course under the global view is calculated through a bilinear layer as the second dimension anomaly score; the node-level representation similarity of the target node under the global view and the focused view is calculated through a bilinear layer as the third dimension anomaly score; the anomaly scores of the three dimensions are weighted and summed to obtain the total cognitive anomaly score of the student corresponding to the target node in the target online course.

[0077] It should be noted that, in the above-mentioned cognitive abnormality student detection module, taking into account the randomness of random walk sampling, it can be set that in one detection process, the global feature extraction module, the focused feature extraction module and the focused feature extraction module need to perform multiple rounds of random walks. The feature representations generated by multiple rounds of random walks are respectively input into the scoring module to calculate the abnormal scores of the three dimensions and the total cognitive abnormality scores, and then the same type of scores of multiple rounds of random walks are averaged, and the abnormal scores of the three dimensions and the total cognitive abnormality scores are finally output in the form of average scores.

[0078] It should be noted that in the embodiment of the present invention, both the multimodal learning representation module and the cognitive abnormal student detection module need to be trained in advance, but the two are not jointly trained. Instead, the multimodal learning representation module is first trained independently, and after the training is completed, the learning parameters in the module are fixed and used to generate the multimodal embedding corresponding to each student, and then the cognitive abnormal student detection module is trained independently.

[0079] In addition, in an embodiment of the present invention, when pre-training the multimodal learning representation module, an auxiliary classification task for identifying the signal conversion category needs to be designed. Specifically, the method of pre-training the multimodal learning representation module through this auxiliary classification task is: using the signal conversion module additionally set in the training phase to perform different types of signal conversions on the unimodal embedding of each modality in the multimodal data, and then predicting the signal conversion category of each modality through a specific modality classification head based on the unimodal embedding after signal conversion, thereby optimizing the multimodal learning representation module through classification cross entropy loss.

[0080] In the signal conversion module of the present invention, the signal conversion category can be designed according to actual conditions. In the embodiment of the present invention, the signal conversion category is divided into five categories, namely, arrangement, time distortion, noise addition, amplitude distortion and clipping. Therefore, the final specific modality classification head also outputs these five category labels.

[0081] In addition, the contrastive learning framework is used to train the cognitive abnormal student detection module. In an embodiment of the present invention, when training the cognitive abnormal student detection module through the contrastive learning framework, it is necessary to perform focused-global contrastive learning, historical-current contrastive learning and individual-interaction contrastive learning, and use the weighted sum of the respective contrastive learning losses as the total loss function, and the loss terms of different contrastive learning can be designed according to the respective comparison objects. In contrastive learning, the construction of positive and negative sample pairs is a key link. In an embodiment of the present invention, in the three contrastive learning types, the respective positive and negative sample pairs are selected as follows:

[0082] 1) In the focused-global contrastive learning, positive and negative sample pairs are constructed for the focused view and the global view respectively; and in the focused view, the subgraph-level representation of the target node in the focused view and the node-level representation of the target node constitute the positive sample pair, and the subgraph-level representation of the target node and the node-level representation of other nodes constitute the negative sample pair; in the global view, the subgraph-level representation of the target node in the global view and the node-level representation of the target node constitute the positive sample pair, and the subgraph-level representation of the target node and the node-level representation of other nodes constitute the negative sample pair;

[0083] 2) In the history-current comparative learning, the node-level representation of the target node in the global view of the current target online course is The positive sample pair is formed by the average value of the node-level representation of the target node in the global view of all historical online courses stored in the memory bank, and the node-level representation of the target node in the global view of the current target online course is The negative sample pairs are formed with the average of the node-level representations of other nodes in the global view of all historical online courses stored in the memory bank;

[0084] 3) In individual-interaction contrastive learning, the subgraph-level representation of the target node in the hypergraph view and the subgraph-level representation of the target node in the global view constitute the positive sample pairs, and the subgraph-level representation of the target node in the hypergraph view and the subgraph-level representation of other nodes in the global view constitute the negative sample pairs.

[0085] Similarly, based on the same inventive concept, Figure 3 As shown, another preferred embodiment of the present invention also provides an online learning cognitive abnormal student detection system based on multimodal representation, which includes:

[0086] The multimodal data collection module is used to collect multimodal data of each student’s learning process in each online course on the online learning platform;

[0087] The detection object designation module is used to allow the management user to designate the student users who need to be tested for cognitive abnormalities through the GUI interface;

[0088] A cognitive anomaly detection module, used to obtain multimodal data corresponding to the specified student user from the multimodal data acquisition module according to the student user specified in the detection object specifying module, and then obtain a cognitive anomaly score for the specified student user according to the online learning cognitive anomaly student detection method based on multimodal representation as in the above embodiment;

[0089] The memory module is used to store the data generated by the other three system modules in the history online courses on the online learning platform.

[0090] It should be noted that the above-mentioned multimodal data acquisition module can be built in the background server of the online learning platform to record the multimodal data of each student's learning process in real time. The detection object designation module can be located on the front-end interface of the online learning platform, and provide designated functions through buttons and the like. Of course, the detection object can also be set by default in the detection object designation module, for example, all students are detected by default. If the management user does not modify the default settings, the detection is performed according to the default settings, and each student on the platform is used as a detection object, and then the cognitive abnormality scores of all students are obtained. Through the relative height of the cognitive abnormality scores, students who need to be focused on can be further screened to assist in the development of online teaching. In addition, the threshold of the cognitive abnormality score can also be set. If the cognitive abnormality score of a student exceeds the threshold, a corresponding reminder can be made. The reminder method can be implemented by a fixed module or display logic in the user interface, or it can be implemented in the form of system messages, pop-ups, or emails, text messages, etc.

[0091] It should also be noted that the online learning cognitive abnormal student detection method based on multimodal representation described in S1 to S3 of the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, Figure 4 As shown, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the online learning cognitive abnormal student detection method based on multimodal representation provided in the above embodiment, which includes a memory and a processor;

[0092] The memory is used to store computer programs;

[0093] The processor is used to implement the online learning cognitive abnormal student detection method based on multimodal representation in the above embodiment when executing the computer program.

[0094] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0095] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method for detecting students with cognitive abnormalities in online learning based on multimodal representation provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the method for detecting students with cognitive abnormalities in online learning based on multimodal representation in the above embodiment can be implemented.

[0096] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0097] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0098] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0099] The following will use a specific example to demonstrate the specific implementation and technical effects of the online learning cognitive abnormality student detection method based on multimodal representation described in S1 to S3 above.

[0100] Example

[0101] In this embodiment, the implementation of the online learning cognitive abnormal student detection method based on multimodal representation described in S1 to S3 is specifically demonstrated, and its specific module structure and training framework are as follows: Figure 5 The following is a detailed description of the overall implementation process.

[0102] 1. Preprocessing of online classroom data

[0103] 1.1. Acquisition and preprocessing of multimodal educational data

[0104] This embodiment collects multimodal educational data from multiple online classes in a class on the same day from multiple sources, of which the data of the last class is used as test data and the rest are used as training data. For the sake of convenience, the following description is given using three modal data as an example. First, the facial expressions and behavioral data of course participants are collected through the video recording system of the online learning device used by students; second, the audio information of teachers and students in the classroom is collected simultaneously through recording equipment; finally, the text information generated in the classroom is collected through computer equipment. After the initial multimodal educational data is collected, the data is standardized to ensure the consistency of the subsequent module inputs. Finally, three initial inputs of the multimodal cognitive representation learning module are generated: , where v, w, and t are for video mode, audio mode, and text mode respectively. Participants in the same online class need to collect their corresponding multimodal data respectively.

[0105] 1.2. Acquisition and preprocessing of basic information of classroom students and interactive perception data

[0106] Before the online class begins, the system will collect basic information about the students (such as age, gender, etc.). This information, as the initial features of the students, makes up for the difficulty in collecting in-depth information about the students in online learning scenarios with a large number of students. In addition, this embodiment also collects the interactions between students in real time during online classroom teaching. After the collection and processing of the above information, the student data in the classroom can be initially constructed into graph data. , where each student is considered as a node , the interaction information between students is regarded as edge , and the basic information of students is regarded as the initial node information This graph-structured data representation facilitates the application of graph neural network (GNN) methods in subsequent modules to detect students with cognitive abnormalities.

[0107] 2. Construction and training of online learning cognitive abnormality student detection model

[0108] 2.1 Model Framework

[0109] 2.1.1. Cognitive Representation Learning Based on Multimodal Classroom Data

[0110] The cognitive representation learning module based on multimodal classroom data consists of two key elements, namely encoder and modality-specific transformation classifiers . Encoder Includes modality-specific encoders based on temporal convolutions and Transformer-based shared encoder ,in, Responsible for modeling and analyzing the differences between multimodal data, Es realizes the exchange of information between modalities. Finally, the student multimodal cognitive features obtained from the encoder are used as a modality-specific transformation classifier The input of each modality is used to identify the transformation type of each modality for training the cognitive representation learning module. In addition, by splicing a cognitive classification head for cognitive recognition tasks ,Mapping students’ multimodal cognitive features into cognitive vectors with specific meanings, assists teachers in understanding students’ cognitive situations.,The implementation of these key components will be described in the following,paragraphs.

[0111] First, a modality-specific encoder is defined Considering the heterogeneity of multimodal signals, we first use a separate encoder for each modality, based on the temporal convolutional network (TCN) as the backbone to capture low-level modality-related information, where the key components are two dilated causal convolutional layers. In addition, the receptive field is increased by dilated convolution, and the time step is The dilated convolution on can be defined as:

[0112]

[0113] In the formula, is the mode number, is the expansion coefficient, Is the size of . Each convolutional layer is followed by a weight normalization layer for the convolutional filters, an activation layer for regularization, and a dropout layer. Then, a residual connection is created between the input and output of the block, where a 1 × 1 convolution is introduced to eliminate the channel number mismatch between the input and output. Finally, a modality-specific projection head (implemented using MLP) and layer normalization are applied to map the low-level features to a higher-dimensional embedding space. Finally, the output of the modality-specific encoder Ep is:

[0114]

[0115] in Representing modality The feature representation of .

[0116] A shared encoder is then defined. This is done through the Transformer, where each modality recognizes components of other modalities that are highly correlated with itself through an attention mechanism to achieve better signal conversion classification. For example, when students answer questions, they need to recognize not only the cognition contained in the audio information but also the facial expressions, body movements, etc. from the video information. To achieve this, the low-level features of each modality are first stacked to form a multimodal embedding. . Then, by scaling the dot product attention, the correlation between the modes is calculated:

[0117]

[0118] Where Q, K, V represent the query, key, and value of attention respectively, and T represents the matrix transpose. For the shared encoder Es, Q, K, V are obtained through multimodal features. The final output of the attention layer is:

[0119]

[0120] in is a learnable matrix. The final output It contains students’ multimodal cognitive information, corresponds to students’ cognitive representation, and to a certain extent provides a basis for the subsequent cognitive abnormality student detection module.

[0121] Finally, through the classification head of a specific modality, the classification results of each modality data are obtained for the training of the cognitive representation learning module based on multimodal classroom data and the category probability of the labeled samples to assist teachers in providing more targeted guidance to students with cognitive abnormalities. The classification process is formulated as follows:

[0122]

[0123] in Represents a specific classification head of different modalities, whose output Is the signal conversion type label.

[0124] 2.1.2 View Construction Based on Hypergraph and Similarity Focus Enhancement

[0125] The multimodal cognitive representation of each student obtained through cognitive representation learning based on multimodal classroom data The graph constructed based on the basic information of students can be Further updated to the cognitive map of students in online classes ,in Contains the multimodal cognitive representations of all students However, there are certain limitations to analyzing students’ cognitive status only from the perspective of the students themselves and their current environment. Therefore, this embodiment designs a variety of graph enhancement techniques to construct comparative views of students’ cognitive graphs. These views are crucial for self-supervised learning tasks and can help the model mine deeper cognitive semantic information for each student.

[0126] The communication between students can have an impact on the cognition of the students on both sides of the communication. Inspired by this, this embodiment modifies the original cognitive map Perform dual hypergraph transformation to construct a hypergraph view of the cognitive graph A hypergraph is a graph whose edges can connect multiple nodes. The dual hypergraph transformation can realize the migration of the message passing mechanism between nodes in the traditional graph to the edges. Transformed into a hyperedge And the side Transformed into a supernode This method can capture the cognitive information contained in the interactions between students.

[0127] In addition, this embodiment converts the original cognitive map It is also defined as a global graph, and a focused graph is constructed based on similarity sampling , this method can improve the robustness of embedding. The previous K-nearest neighbor based sampling method is different. Considering the heterogeneity of the interaction density between different students and others due to differences in students' personalities, the method adopted in this embodiment first calculates the similarity matrix of the original attribute network ,in Representation Node The similarity score can generally be set to the inner product of the feature, and then when the similarity score of any node pair in the graph is greater than the specified threshold We retain the edge between the node pair, otherwise delete it. The formula for obtaining the focused view structure is as follows:

[0128]

[0129] in is the adjacency matrix of the global graph.

[0130] In order to comprehensively evaluate the cognitive status of students, in addition to considering the cognitive consistency between students and the class as a whole, this embodiment also considers the cognitive information of students' core social circles. The system first constructs the corresponding focus graph based on the global graph, and then uses a random walk strategy to sample the subgraph, obtains the subgraph of each node from the global graph and the focus graph of the online class students, and further constructs the sampled global graph subgraph into the above-mentioned hypergraph view.

[0131] Therefore, the processing flow executed in the above step 2.1.2 can constitute the multi-view construction module in the cognitive abnormal student detection module.

[0132] 2.1.3 Multi-granularity feature extraction under multi-comparison strategy

[0133] In order to capture students' cognitive information at multiple levels, this embodiment constructs a multi-strategy contrast network based on multiple enhanced views. It mainly includes three parts of contrast learning: focus-global contrast, individual-interaction contrast, and history-current contrast. In order to carry out subsequent contrast learning, it is necessary to extract the features of the enhanced views under the multiple contrast strategies.

[0134] The focused-global comparison includes two views, the global map and the focused map. This module uses the node-subgraph comparison method on the two views to calculate the discrimination score. The main difference between the two views is that different readout strategies are used. In addition, in order to prevent the influence of the target node information in the subgraph on the identification, the node attributes corresponding to the target node corresponding to the student to be detected in the subgraph are anonymized. The overall framework of the two views is roughly similar. First, the target node is used as the starting sampling point in the global view, a subgraph is sampled by random walk and mapped to a low-dimensional embedding through a graph convolutional network:

[0135]

[0136] in For all nodes in the subgraph Layer hidden representation, a total of L layers, is the normalized adjacency matrix, is the normalization of the angle matrix, represents the learnable parameter matrix, is the activation function. The average readout function is applied to averagely read out the potential representations of all nodes in the subgraph:

[0137]

[0138] Where: P is the total number of nodes in the subgraph, It is a sub-graph level representation under the global view.

[0139] Use the above Calculate the same graph convolutional network and map the target nodes corresponding to the students to be detected to the same dimension:

[0140]

[0141] in The target node Layer hidden representation, initial hidden representation That is, the multimodal cognitive representation of the target node .make The final embedding of the target node in the subgraph of the global view.

[0142] For the sampled subgraph of the focused view, the target node is used as the starting sampling point in the focused view, a subgraph is sampled by random walk and mapped to a low-dimensional embedding through a graph convolutional network, where the graph convolutional network uses another graph convolutional network with different parameters from the global view to obtain the representation on the focused view subgraph:

[0143]

[0144] The meanings of the parameters in the formula are basically the same as those in the global map.

[0145] To minimize the impact of noisy information on graph-level embeddings, instead of using average readout, we propose a simple yet effective similarity-based attention mechanism in a focused view that can adaptively select useful embeddings for subgraphs. Intuitively, if a node in a subgraph is more similar to the target node, it can have more weight in the pooling process, and vice versa. To allow the module to adaptively exploit similarity information, a multi-layer perceptron mapping is applied to the similarity vector to obtain the weight vector , which is expressed in the form of:

[0146]

[0147] in is a learnable parameter and b is a bias. is the similarity vector between the target node and the other nodes, from the similarity get As an attention factor, applied to obtain the sub-image level embedding of the focused view:

[0148]

[0149] Also using Compute the same graph convolutional network, mapping the target node to the same dimension on the focused view:

[0150]

[0151] in The target node Layer hidden representation, initial hidden representation That is, the multimodal cognitive representation of the target node .make The final embedding for the focused view target node.

[0152] The individual-interaction comparison is defined between the global view and the hypergraph view. It will help to fully consider the interaction information in multimodal classroom data and obtain a more representative student cognitive representation. The subgraph-level representation of the global view has been obtained above, and the subgraph-level representation of the hypergraph view is obtained through the hypergraph neural network and the readout function.

[0153] For the subgraph obtained by random walk sampling of the target node in the global view, it is converted into a subgraph under the hypergraph view through the dual hypergraph transformation. The subgraph information is first encoded through the hypergraph neural network:

[0154]

[0155] in and are the diagonal matrices of the degrees of supernodes and hyperedges, that is, the number of hyperedges to which a supernode belongs and the number of nodes contained in the hyperedge. The association matrix of the hypergraph view is used to represent the structural information of the hypergraph. is the learnable parameter matrix.

[0156] Get the output of the last layer of the hypergraph neural network for the subgraph under the hypergraph view , the super-node dimension is averaged for all super-node hidden representations of the final output of the hypergraph neural network, that is, the final hidden representations of all super-nodes are read out by average to obtain the sub-graph level representation of the target node under the hypergraph view The input of the hypergraph neural network is the representation of the original edge. For simplicity, the specific time when the interaction occurred and the duration of the interaction are encoded by default.

[0157] Different from the traditional graph contrastive learning method, in individual-interaction contrastive learning, the target node in the hypergraph view is The subgraph level representation of the target node in the global view The subgraph-level representation of The subgraph level representation of The sub-graph level representation of constitutes a negative sample pair.

[0158] Finally, based on the assumption that students' cognition is not likely to change significantly in a short period of time, this embodiment designs a historical node-current node comparison. Specifically, the target node representation obtained by the graph convolution network of the student's cognitive global view at the previous moment (the moment here corresponds to a certain online course) is compared with the target node representation obtained by the graph convolution network of the student's global view at the current moment, which is more conducive to the task of detecting students with cognitive abnormalities. After each course, the node-level representation of the global view learned in this course is compared with the target node representation of the global view learned in this course. Stored in a memory bank and retrieve different moments of historical moments stored in the memory bank Average Used for historical node-current node comparison.

[0159] The multi-strategy comparison network based on multiple enhanced views finally obtains the multi-granularity representation of all target nodes under multiple enhanced views. Taking the global view, focused view and hypergraph view as an example, we can get the subgraph level representations on the global view, focused view and hypergraph view respectively. , node-level representation on the global view and focused view subgraphs , and historical node-level representation . Where dense, spars and hyper represent the global view, focused view and hypergraph view respectively. The above results are used for subsequent model training and calculation of anomaly scores.

[0160] Therefore, the three granularity feature extraction processes executed in the above step 2.1.3 can constitute the global feature extraction module, the focused feature extraction module and the hypergraph feature extraction module in the cognitive abnormal student detection module.

[0161] 2.3 Cognitive Representation Learning and Anomaly Detection Model Training and Anomaly Score Calculation

[0162] 2.3.1 Training of the Unsupervised Multimodal Representation Learning Module

[0163] This embodiment innovatively proposes to generate a dataset with pseudo labels by signal conversion. , used for pre-training of the unsupervised multimodal representation learning module. This embodiment applies five signal conversion methods to multimodal data, namely, arrangement, time warping, noise addition, amplitude warping and clipping. These conversion methods are applied to three modes: video, audio and text, and the generated conversion signal data is input into the multimodal representation learning module together with the original multimodal signal data. The above signal conversion process can be described as follows.

[0164] 1) Gaussian noise addition: The input signal is , Gaussian white noise interference is The noise is taken from a zero-mean normal distribution, and the variance of the normal distribution is given by the formula Calculated, where is the average power of the signal, and the final output is ;

[0165] 2) Amplitude distortion: The amplitude of the original signal is interpolated by the cubic spline function The random smooth curve formed changes, and the final output is .

[0166] 3) Arrangement: The original signal is divided into non-overlapping segments, which are then shuffled in time and eventually reassembled.

[0167] 4) Time warping: The original signal is split into non-overlapping segments, half of which are randomly selected by the linear interpolation function Stretch, the other half Extrusion, where is the ith fragment of the original signal, and k is the stretch factor. Finally, the two halves are reassembled.

[0168] 5) Cropping: The original signal is divided into n non-overlapping segments, one of which is randomly divided and re-joined to its original length.

[0169] After completing the signal conversion, the signal type is used as the pseudo-label of the corresponding data, and all the data with pseudo-labels are merged into a pseudo-label dataset for training the multimodal learning representation module, that is, for training the modality-specific encoder. and shared encoder and modality-specific classification heads The training loss The calculation formula is as follows:

[0170]

[0171]

[0172] in Indicates the signal conversion category label, The total loss of the unsupervised multimodal cognitive representation learning module.

[0173] 2.3.2 Training of cognitive abnormality student detection module and calculation of cognitive abnormality score

[0174] Through the cognitive anomaly detection module, each student node is obtained After obtaining the multi-view and multi-level representations of the proposed method, the similarities between the positive and negative sample pairs are calculated respectively, and the losses of the three comparison strategies are calculated respectively.

[0175] For focus-global contrast learning, this embodiment uses a node-subgraph contrast method on the focus view and the global view. The subgraph level representation of the target node The node-level representation of constitutes a positive sample pair, and the target node The subgraph level representation of other nodes The node-level representation of constitutes a negative sample pair; in the global view, the target node in the global view The subgraph level representation of the target node The node-level representation of constitutes a positive sample pair, and the target node The subgraph level representation of other nodes The node-level representation of constitutes a negative sample pair;

[0176] Different from the previous methods, this embodiment uses the same discriminator on the two views, so that the information interaction between the two views is enhanced. Specifically, a dual linear model (i.e., a bilinear layer ) respectively obtain the similarity of the positive and negative sample pairs embedded in the focused view and the global view:

[0177]

[0178]

[0179] in is a shared learnable weight matrix, and They represent the global view and the focused view respectively.

[0180] In view Global View When the similarity of the positive sample pair is Similarity with negative sample pairs Find the difference as the anomaly score

[0181]

[0182] The goal of focused-global contrastive learning is to make the similarity score of positive samples closer to 1 and vice versa. Therefore, in this embodiment, the loss function of focused-global contrastive learning is defined as the contrast loss between the subgraph-level representation and the node-level representation in the global view. Comparative loss between subgraph-level representation and node-level representation in focused view sum:

[0183]

[0184]

[0185] For historical node-current node comparison learning, this embodiment stores the target student node in the memory bank. Average of node-level representations obtained by graph convolutional networks in history courses With the current course For comparison, the embedding similarity of the positive sample pair Embedding similarity with negative sample pairs They are:

[0186]

[0187]

[0188] in Discriminator in contrastive learning with focused-global shared.

[0189] Due to the general consistency of individual cognition, small group cognition and group cognition in the classroom, this embodiment defines the historical node-current node comparison loss as:

[0190]

[0191] When the comparison pair is a positive sample pair, is set to 1 when the comparison pair is a negative sample pair. is set to 0 and Replace with The sum of the positive and negative sample comparison losses is the final .

[0192] Finally, for individual-interaction contrastive learning, the target nodes for interaction between students are The subgraph-level embedding on the hypergraph view is compared with the subgraph-level embedding on the global view for individual students. The subgraph level representation of the target node in the global view The subgraph-level representation of The subgraph level representation of The sub-graph level representation of composes a negative sample pair.

[0193] In this embodiment, individual-interaction contrast loss It is defined as the contrast loss between the sub-graph level representation in the hypergraph view and the global view, as follows:

[0194]

[0195] Combining the above losses, the total loss of the final training cognitive abnormal student detection module is obtained:

[0196]

[0197] in , , is a hyperparameter.

[0198] This embodiment integrates the abnormal information from the perspectives of multiple enhanced views and multiple comparison strategies, and additionally considers the similarity of the cross-view comparison of the target student node-level embedding in the focused view and the global view:

[0199]

[0200] As one of the bases for calculating abnormal scores, the scoring module The cognitive abnormality score can be expressed as:

[0201]

[0202] Where δ is a hyperparameter.

[0203] It should be noted that in calculating the above You need to calculate first , and , and the calculation process of these three similarity coefficients requires not only the target node The information of other nodes is also needed Information of other nodes Just select any one of the nodes in the figure. The information of the target node can be used in the entire graph The node numbers of adjacent nodes.

[0204] Since the random walk strategy used in sampling subgraphs has a certain degree of randomness, this embodiment performs multiple rounds of detection under the setting of multiple rounds of random walks, and obtains the average of the above detection anomaly scores under multiple rounds. Used to determine the final degree of cognitive abnormality of students, For Students The final total cognitive abnormality score. Therefore, this process of calculating the final cognitive abnormality of students can constitute the scoring module in the cognitive abnormality student detection module. The cognitive abnormality student detection module used in the actual reasoning link can be composed of the above-mentioned multi-view construction module, global feature extraction module, focus feature extraction module, hypergraph feature extraction module and scoring module.

[0205] 3. Detection of students with cognitive abnormalities in online learning

[0206] 1) Transformer-based multimodal cognitive representation learning

[0207] First, through the unsupervised training method proposed in this embodiment, based on the loss The multimodal representation learning module is pre-trained; then, multimodal teaching data of a large-scale student online learning is used as input samples and input into the trained multimodal representation learning module, and finally the cognitive representation of each student is obtained from the online classroom multimodal data for the downstream cognitive abnormality student detection task.

[0208] 2) Detection of cognitively abnormal students based on multi-strategy comparison of multiple enhanced views

[0209] The student cognitive representation obtained by the multimodal representation learning module is used as the student's feature, and the student cognitive global view is constructed according to the student interaction collected in the classroom. Then, multi-strategy comparative learning based on multiple enhanced views is performed to obtain the cognitive abnormality score of each student, as well as the cognitive abnormality performance at various levels, such as the cognitive abnormality of students in their highly similar core social circles:

[0210]

[0211] in: It indicates the degree of abnormality exhibited by the target student’s cognition compared with the class’s overall cognition;

[0212] The degree of abnormality that the target student exhibits due to cognitive differences in the class and core social circle;

[0213] It indicates the degree of abnormality of the target student's cognition at the temporal level;

[0214] It represents the overall cognitive abnormality score of the target students obtained by weighting the above three factors.

[0215] The main goal of the present invention is to evaluate the degree of cognitive abnormality of each student, so as to assist teachers in quickly and accurately identifying students with cognitive abnormalities in large-scale online learning scenarios. Therefore, the application of unsupervised multimodal cognitive representation learning and multi-view multi-strategy comparison of cognitive anomaly detection in the field of education has strong practicality and helps to promote students' cognitive development in online learning environments.

[0216] In order to verify the technical effect of the method proposed in the present invention, it is verified in an actual online learning scenario. The data set selected in this embodiment is a multimodal education data set collected in an online classroom to record multimodal data in the student learning process. During the course collected by the data set, the lecturer will arrange reasonable discussions, questions and answers, and written homework links so that data can be obtained from multiple modes. In addition, in order to more intuitively reflect the ability of the model proposed in the present invention to detect students with cognitive abnormalities, a certain number of lower grade students are arranged in the classroom as typical students with cognitive abnormalities to be detected. The personal basic information of students in the classroom is collected as the initial features of each student before the course starts, and the interaction between students is required to be implemented with online equipment for real-time collection. In the constructed student relationship graph, each student in this data set is constructed as a node in the graph, and each interaction between students is constructed as an edge in the graph.

[0217] The evaluation index of this embodiment uses the most commonly used AUC in the field of anomaly detection. It is the area under the ROC curve, with a value range of 0 to 1. The larger the value, the better the model effect. The comparison between the method of the present invention and the unsupervised node-level anomaly detection method for static graphs is shown in Table 1. For the sake of fairness, subsequent experiments are all conducted under the above-mentioned multimodal education dataset with an anomaly rate of 10%, and the benchmark method uses the method of the present invention to obtain cognitive representation as the node feature of the input graph.

[0218] Table 1 Detection results of cognitive abnormal students by the proposed method on multimodal education dataset

[0219]

[0220] The experimental results show that the method of the present invention has a significant improvement over the benchmark method. One main reason is that the method of the present invention adopts multiple comparison strategies to effectively model the dependencies between different influencing factors of students' cognition. The above-mentioned comparison methods are: Dominant, an anomaly detection model based on graph autoencoder, which is derived from the existing technical literature: Ding K, Li J, Bhanushali R, et al. Deep anomaly detection on attributed networks[C] / / Proceedings of the 2019 SIAM international conference on datamining. Society for Industrial and Applied Mathematics, 2019: 594-602.; CoLA, an anomaly detection model based on node-level comparison, which is derived from the existing technical literature: Liu Y, Li Z, Pan S, et al.Anomaly detection on attributed networks via contrastive self-supervisedlearning[J]. IEEE transactions on neural networks and learning systems, 2021,33(6): 2378-2392.; SL-GAD, an anomaly detection model based on contrast and generation, which is derived from the existing technical literature: Zheng Y, Jin M, Liu Y, et al. Generative and contrastive self-supervisedlearning for graph anomaly detection[J]. IEEE Transactions on Knowledge andData Engineering, 2021, 35(12): 12220-12233.

[0221] In addition, this embodiment further conducts an ablation study to better examine the contribution of different modules proposed in the present invention. The experimental results are shown in Table 2, and the specific details of each ablation variant are as follows:

[0222] w / o multimodal cognition: Based on the complete framework (S1~S3) proposed by this invention, the multimodal representation learning module is deleted and the basic information of students is used as the initial node features of the input graph data ;

[0223] w / o Focus-Global: Based on the complete framework proposed by the present invention, the Focus-Global comparison strategy is not adopted;

[0224] w / o individual-interaction: Based on the complete framework proposed in this invention, the individual-interaction comparison strategy is not adopted;

[0225] w / o historical node-current node: Based on the complete framework proposed by the present invention, the historical node-current node comparison strategy is not adopted.

[0226] Table 2 Ablation experiment results of different modules of the present invention on multimodal education dataset

[0227]

[0228] The ablation experiment results show that the multiple contrast strategies based on enhanced views proposed in the present invention all have significant effects. The main reason is that the contrast strategy has been designed in accordance with the online learning scenario, making it possible to analyze students' cognition from multiple perspectives in the online classroom. In addition, the multimodal cognitive representation learning module has a great influence on the experimental results, which illustrates the effectiveness of the module in extracting the universal cognitive representation of students in multimodal educational data in an unsupervised scenario. This is mainly due to its unsupervised training method based on signal conversion and the multimodal feature extraction and fusion module.

[0229] In summary, the present invention can provide an efficient and accurate solution for detecting students with cognitive abnormalities for unsupervised online learning scenarios.

[0230] The above-described embodiments are only some preferred implementations of the present invention, but are not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for detecting students with cognitive abnormalities in online learning based on multimodal representation, characterized in that: include: S1. Obtain multimodal data of each student's learning process in the target online course, wherein the multimodal data includes video data recording students' facial expressions and body movements, audio data recording online classroom sounds, and text data recording online classroom interactive discussion texts; construct a student relationship graph by taking all students as nodes, interactions between students as edges, and basic information registered by students on the platform as node features; S2. Input the multimodal data of each student in the target online course into a multimodal learning representation module pre-trained by the auxiliary classification task to obtain the multimodal embedding corresponding to each student; in the multimodal learning representation module, firstly, for each of the three modalities of video, audio and text in each multimodal data, respectively, use the modality-specific encoder to encode the data of different modalities to obtain their respective single-modal embeddings, and then splice the single-modal embeddings of the three modalities and input them into the Transformer-based shared encoder to fuse them into multimodal embeddings; S3, input the multimodal embeddings and student relationship graphs corresponding to all students in the target online course into the cognitive abnormality student detection module pre-trained by the contrastive learning framework to obtain the cognitive abnormality score of the student corresponding to the target node; The cognitive abnormal student detection module consists of a multi-view construction module, a global feature extraction module, a focused feature extraction module, a hypergraph feature extraction module and a scoring module; the multi-view construction module is used to combine the student relationship graph and the multimodal embedding to construct a global view, and at the same time, the global view is converted into a focused view through the edge screening operation of node similarity; in the global feature extraction module, for the target node corresponding to the student to be detected, a subgraph-level representation and a node-level representation of the target node under the global view are generated; the focused feature extraction module is used to generate a subgraph-level representation and a node-level representation of the target node under the focused view for the target node corresponding to the student to be detected; the hypergraph feature extraction module is used to generate a subgraph-level representation of the target node under the hypergraph view through a dual hypergraph transformation and a hypergraph neural network; in the scoring module, according to the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module and the hypergraph feature extraction module respectively under multiple rounds of random walks, combined with the historical information in the memory bank, a multi-dimensional abnormality score is performed through a bilinear layer, and the final cognitive abnormality total score of the target node is obtained by fusion.

2. The method for detecting students with cognitive abnormalities in online learning based on multimodal representation as claimed in claim 1, characterized in that: In S2, when the multimodal learning representation module is pre-trained through an auxiliary classification task, the signal conversion module additionally set in the training phase is used to perform different categories of signal conversion on the unimodal embedding of each modality in the multimodal data, and then the signal conversion category of each modality is predicted through a specific modality classification head based on the unimodal embedding after signal conversion, so as to optimize the multimodal learning representation module through classification cross entropy loss.

3. The method for detecting students with cognitive abnormalities in online learning based on multimodal representation as claimed in claim 2, characterized in that: The signal conversion categories in the signal conversion module are divided into five categories, namely, arrangement, time distortion, noise addition, amplitude distortion and clipping.

4. The method for detecting students with cognitive abnormalities in online learning based on multimodal representation as claimed in claim 1, characterized in that: In S3, the internal processes of each submodule of the cognitive abnormal student detection module are as follows: In the multi-view construction module, the node features of the student nodes in the student relationship graph are replaced with the corresponding multimodal embeddings to form a global view; the edge connections in the global view are screened based on the feature similarity between the nodes, and the edges whose feature similarity between the nodes is lower than the threshold are deleted, thereby converting the global view into a focused view; In the global feature extraction module, for the target node corresponding to the student to be detected, the target node is first used as the starting sampling point under the global view, a subgraph is sampled by random walk and input into the first graph convolutional network for layer-by-layer aggregation to generate a hidden representation, the node dimension of all node hidden representations finally output by the first graph convolutional network is averaged to obtain the subgraph-level representation of the target node under the global view; at the same time, the node features of the target node in the global view are input into the first graph convolutional network to map them to the node-level representation of the target node under the global view; In the focused feature extraction module, for the target node corresponding to the student to be detected, the target node is first used as the starting sampling point in the focused view, a subgraph is sampled by random walk and input into the second graph convolution network for layer-by-layer aggregation to generate a hidden representation, the similarity of each other node relative to the target node is calculated based on the hidden representation of all nodes finally output by the second graph convolution network, and then the similarity is mapped to a weight through a multi-layer perceptron, and then the node hidden representation of all nodes finally output by the second graph convolution network is weightedly fused to obtain a subgraph-level representation of the target node in the focused view; at the same time, the node features of the target node in the focused view are input into the second graph convolution network to map them to a node-level representation of the target node in the focused view; In the hypergraph feature extraction module, the subgraph collected in the global feature extraction module is first subjected to dual hypergraph transformation to obtain the subgraph under the hypergraph view and input into the hypergraph neural network for encoding, and the supernode dimension is averaged for all supernode hidden representations finally output by the hypergraph neural network to obtain the subgraph-level representation of the target node under the hypergraph view; In the scoring module, according to the multiple feature representations extracted by the global feature extraction module, the focused feature extraction module, and the hypergraph feature extraction module, the node-level representation similarity between the target node and other nodes under the global view is calculated through a bilinear layer as the first dimension anomaly score; the node-level representation similarity between the target node in the current online course and the historical online course under the global view is calculated through a bilinear layer as the second dimension anomaly score; the node-level representation similarity between the target node under the global view and the focused view is calculated through a bilinear layer as the third dimension anomaly score; the anomaly scores of the three dimensions are weighted and summed to obtain the total cognitive anomaly score of the student corresponding to the target node in the target online course.

5. The method for detecting students with cognitive abnormalities in online learning based on multimodal representation as claimed in claim 4, characterized in that: In the cognitive abnormality student detection module, the global feature extraction module, the focused feature extraction module and the focused feature extraction module need to perform multiple rounds of random walks. The feature representations generated by the multiple rounds of random walks are respectively input into the scoring module to calculate the abnormal scores of the three dimensions and the total score of cognitive abnormalities. Then, the scores of the same category of the multiple rounds of random walks are averaged, and the abnormal scores of the three dimensions and the total score of cognitive abnormalities are finally output in the form of average scores.

6. The method for detecting students with cognitive abnormalities in online learning based on multimodal representation as claimed in claim 1, characterized in that: In S3, when training the cognitive abnormal student detection module through the contrastive learning framework, it is necessary to perform focus-global contrastive learning, history-current contrastive learning and individual-interaction contrastive learning, and use the weighted sum of the respective contrastive learning losses as the total loss function; In the focus-global contrast learning, positive and negative sample pairs are constructed for the focus view and the global view respectively; In the focused view, the subgraph-level representation of the target node in the focused view and the node-level representation of the target node constitute a positive sample pair, and the subgraph-level representation of the target node and the node-level representation of other nodes constitute a negative sample pair; in the global view, the subgraph-level representation of the target node in the global view and the node-level representation of the target node constitute a positive sample pair, and the subgraph-level representation of the target node and the node-level representation of other nodes constitute a negative sample pair; In the history-current comparative learning, the node-level representation of the target node in the global view of the current target online course and the average value of the node-level representation of the target node in the global view of all historical online courses stored in the memory bank constitute a positive sample pair, and the node-level representation of the target node in the global view of the current target online course and the average value of the node-level representation of other nodes in the global view of all historical online courses stored in the memory bank constitute a negative sample pair; In the individual-interaction contrastive learning, the subgraph-level representation of the target node in the hypergraph view and the subgraph-level representation of the target node in the global view constitute positive sample pairs, and the subgraph-level representation of the target node in the hypergraph view and the subgraph-level representation of other nodes in the global view constitute negative sample pairs.

7. The method for detecting students with cognitive abnormalities in online learning based on multimodal representation as claimed in claim 1, characterized in that: The multimodal learning representation module is first trained independently. After the training is completed, the learning parameters in the module are fixed and used to generate the multimodal embedding corresponding to each student. Then, the cognitive abnormality student detection module is trained independently.

8. An online learning cognitive abnormal student detection system based on multimodal representation, characterized in that: include: The multimodal data collection module is used to collect multimodal data of each student’s learning process in each online course on the online learning platform; The detection object designation module is used to allow the management user to designate the student users who need to be tested for cognitive abnormalities through the GUI interface; A cognitive anomaly detection module, for obtaining multimodal data corresponding to a designated student user from the multimodal data acquisition module according to the designated student user in the detection object designation module, and then obtaining a cognitive anomaly score of the designated student user according to the online learning cognitive anomaly student detection method based on multimodal representation as described in any one of claims 1 to 7; The memory module is used to store the data generated by the other three system modules in the history online courses on the online learning platform.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for detecting students with cognitive abnormalities in online learning based on multimodal representation as described in any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the method for detecting students with cognitive abnormalities in online learning based on multimodal representation as described in any one of claims 1 to 7 when executing the computer program.

Images