Multimodal learning behavior analysis method, system, and storage medium

Abstract

This invention discloses a method, system, and storage medium for multimodal learning behavior analysis, applied in the field of artificial intelligence technology. It enables high-performance and highly interpretable multimodal learning behavior analysis, providing reliable analytical explanations for educational decision-making processes. The method includes: acquiring multimodal behavior data of the learning process of the object to be analyzed and preprocessing it to obtain multimodal sequence data; performing collaborative embedding representation on the multimodal sequence data to obtain initial feature representations; constructing a learning behavior data association graph based on the initial feature representations; decoupling the learning behavior data association graph using a graph decoupling neural network to obtain a multimodal decoupling graph; constructing an attribute routing mechanism based on the relationships between nodes in the multimodal decoupling graph; updating the node embedding representations and graph structure through the attribute routing mechanism to obtain target feature representations and target graph structures; performing learning behavior analysis based on the target feature representations and visualizing the target graph structure to obtain visualized learning behavior analysis results.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal learning behavior analysis method, system, and storage medium. Background Technology

[0002] With the rise of big data in education and artificial intelligence (AI) technology, the integration of AI with education has become a crucial tool in intelligent education practices. Intelligent education is also shifting from internet-based approaches to practical applications such as precision teaching assistance based on the analysis of students' entire learning behavior data. However, with the increasing volume of student learning behavior data, high-dimensional features are difficult for the average person to understand and lack a meaningful connection to metadata. There is currently no good method for providing reliable analytical explanations for educational decision-making processes while achieving high performance and high interpretability. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this invention proposes a multimodal learning behavior analysis method, system, and storage medium, which can achieve high-performance and highly interpretable multimodal learning behavior analysis, providing reliable analysis and interpretation for the educational decision-making process.

[0004] On one hand, embodiments of the present invention provide a multimodal learning behavior analysis method, including the following steps:

[0005] Acquire multimodal behavioral data of the learning process of the object to be analyzed;

[0006] The multimodal behavior data is preprocessed to obtain the corresponding multimodal sequence data;

[0007] Based on the multimodal sequence data, a collaborative embedding representation is performed to obtain an initial feature representation;

[0008] Construct a data association graph of learning behavior based on the initial feature representation;

[0009] The learning behavior data association graph is decoupled by a graph decoupling neural network to obtain a multimodal decoupling graph;

[0010] An attribute routing mechanism is constructed based on the relationships between the nodes in the multimodal decoupling graph.

[0011] The node embedding representation and graph structure of the multimodal decoupling graph are updated iteratively through the attribute routing mechanism to obtain the target feature representation and target graph structure;

[0012] Learning behavior analysis is performed based on the target feature representation, and the target graph structure is visualized to obtain visualized learning behavior analysis results.

[0013] According to an embodiment of the present invention, a multimodal learning behavior analysis method has at least the following beneficial effects: First, this embodiment acquires multimodal behavior data of the object to be analyzed during the learning process, and preprocesses the acquired multimodal behavior data to obtain corresponding multimodal sequence data, facilitating deep feature learning. Next, this embodiment performs collaborative embedding representation based on the multimodal sequence data to obtain initial feature representations, thereby mapping multiple modal information to the same embedding space through collaborative embedding, enhancing the complementarity between modalities. Then, a learning behavior data association graph is constructed based on the obtained initial feature representations to improve the data representation capability. Simultaneously, a graph decoupling neural network is used to decouple the learning behavior association graph, obtaining a multimodal decoupling graph. Furthermore, this embodiment constructs an attribute routing mechanism based on the relationships between nodes in the multimodal decoupling graph, and updates and iterates the node embedding representations and graph structure of the multimodal decoupling graph through the attribute routing mechanism to obtain target feature representations and target graph structures. This effectively controls the transmission of information within the model through the constructed attribute routing mechanism, improving the accuracy and transparency of the learning behavior analysis results. Furthermore, this embodiment performs learning behavior analysis based on the target feature representation and visualizes the target graph structure to obtain visualized learning behavior analysis results, thereby achieving high-performance and highly interpretable multimodal learning behavior analysis and providing reliable analysis and interpretation for the educational decision-making process.

[0014] According to some embodiments of the present invention, the preprocessing of the multimodal behavioral data to obtain corresponding multimodal sequence data includes:

[0015] Based on the data format of the multimodal behavioral data, corresponding data cleaning is performed to obtain multimodal cleaned data;

[0016] The multimodal cleaning data is serialized to obtain the multimodal sequence data.

[0017] According to some embodiments of the present invention, the step of performing cooperative embedding representation based on the multimodal sequence data to obtain an initial feature representation includes:

[0018] The initial feature representation is obtained by performing the collaborative embedding representation of different modal sequence data in the multimodal sequence data through a convolutional neural network and a long short-term memory network.

[0019] According to some embodiments of the present invention, constructing a learning behavior data association graph based on the initial feature representation includes:

[0020] The association relationships between time slice nodes are obtained based on the initial feature representation; wherein, the time slice node is a time slice in the multimodal sequence data;

[0021] Construct an adjacency matrix based on the aforementioned relationships;

[0022] The learning behavior data association graph is constructed based on the adjacency matrix and the time slice nodes.

[0023] According to some embodiments of the present invention, the step of constructing an attribute routing mechanism based on the relationships between nodes in the multimodal decoupling graph includes:

[0024] Calculate the first difference score matrix between the nodes of each modal decoupling graph in the multimodal decoupling graph; wherein, the first difference score matrix is ​​a mask matrix for intramodal information transmission;

[0025] Calculate the second difference score matrix of the same node in the multimodal decoupling graph in different modes; wherein the second difference score matrix is ​​a mask matrix for cross-modal information transmission.

[0026] According to some embodiments of the present invention, the step of updating and iterating the node embedding representation and graph structure of the multimodal decoupling graph through the attribute routing mechanism to obtain the target feature representation and target graph structure includes:

[0027] Based on the transmission of the first difference score matrix control information in the time dimension within each modal decoupling diagram in the multimodal decoupling diagram, the time dimension representation update is obtained;

[0028] Based on the control information of the second difference score matrix, the transmission of modal dimensions between the various modal decoupling diagrams in the multimodal decoupling diagram is obtained, and the modal dimension representation update is obtained;

[0029] The node embedding representation is updated based on the time dimension representation update and the modality dimension representation update to obtain the target feature representation;

[0030] The graph structure is updated based on the target feature representation to obtain the target graph structure.

[0031] According to some embodiments of the present invention, the step of performing learning behavior analysis based on the target feature representation and visualizing the target graph structure to obtain visualized learning behavior analysis results includes:

[0032] The target feature representation is input into a preset graph decoupled learning behavior analysis model to perform learning behavior analysis and obtain the learning behavior analysis results.

[0033] The target graph structure is visualized based on the mask matrix to obtain a preset visualization image; wherein, the preset visualization image includes a heatmap and a structure visualization image;

[0034] The visual learning behavior analysis result is obtained based on the learning behavior analysis result and the preset visualization image.

[0035] On the other hand, embodiments of the present invention also provide a multimodal learning behavior analysis system, including:

[0036] The data acquisition module is used to acquire multimodal behavioral data of the learning process of the object to be analyzed;

[0037] The preprocessing module is used to preprocess the multimodal behavioral data to obtain corresponding multimodal sequence data;

[0038] An embedding representation module is used to perform collaborative embedding representation based on the multimodal sequence data to obtain an initial feature representation;

[0039] The association graph construction module is used to construct an association graph of learning behavior data based on the initial feature representation;

[0040] The decoupling module is used to decouple the learning behavior data association graph through a graph decoupling neural network to obtain a multimodal decoupling graph;

[0041] The routing construction module constructs an attribute routing mechanism based on the relationships between the nodes in the multimodal decoupling graph.

[0042] The routing update module is used to update and iterate the node embedding representation and graph structure of the multimodal decoupling graph through the attribute routing mechanism to obtain the target feature representation and target graph structure;

[0043] The results analysis module is used to perform learning behavior analysis based on the target feature representation and visualize the target graph structure to obtain visualized learning behavior analysis results.

[0044] On the other hand, embodiments of the present invention also provide a multimodal learning behavior analysis system, including:

[0045] At least one processor;

[0046] At least one memory for storing at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the multimodal learning behavior analysis method as described in the above embodiments.

[0048] On the other hand, embodiments of the present invention also provide a computer storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the multimodal learning behavior analysis method as described in the above embodiments. Attached Figure Description

[0049] Figure 1 This is a flowchart of the multimodal learning behavior analysis method provided in the embodiments of the present invention;

[0050] Figure 2 This is a block diagram illustrating the principle of the multimodal learning behavior analysis system provided in this embodiment of the invention. Detailed Implementation

[0051] The embodiments described in this application should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0054] With the rise of big data in education and artificial intelligence (AI) technology, the integration of AI with pedagogy has become a crucial tool in intelligent education practices. Intelligent education is also shifting from internet-based approaches to practical applications such as precision teaching assistance based on comprehensive student behavior data analysis. Students exhibit diverse individual behaviors, reflecting their learning styles, psychological characteristics, and emotional traits to a certain extent. By leveraging deep learning-based behavioral analysis models to mine behavioral data from the entire student learning process—including multimodal data such as sound, video, and facial expressions—we can gain a more comprehensive and efficient understanding of students' learning processes, uncover learning patterns, and expand the depth and breadth of learning analysis research, thus providing services for the learning process. Currently, most multimodal learning behavior analysis models perceive various behavioral information through various intelligent recognition technologies, collecting multimodal data for unified representation to accurately determine learning progress. These methods often combine machine learning approaches, focusing on algorithmic model building, but neglecting the issue of data interpretability. Furthermore, as student learning behavior data increases, high-dimensional features become difficult for ordinary people to understand and lack a meaningful connection with metadata. These deep learning-based models contain uncertainties, and most of their results are weakly correlated with educational and teaching principles, lacking interpretability. This poses a serious threat to the application of these technologies in practical education and teaching, especially in outcome-sensitive tasks such as academic early warning and knowledge tracking.

[0055] One embodiment of the present invention provides a multimodal learning behavior analysis method, system, and storage medium, which can achieve high-performance and highly interpretable multimodal learning behavior analysis, providing reliable analytical interpretation for the educational decision-making process. (See also...) Figure 1 The method in this embodiment of the invention includes, but is not limited to, steps S110, S120, S130, S140, S150, S160, S170 and S180.

[0056] Specifically, the application process of the method in this embodiment of the invention includes, but is not limited to, the following steps:

[0057] S110: Obtain multimodal behavioral data of the learning process of the object to be analyzed.

[0058] S120: Preprocess the multimodal behavioral data to obtain the corresponding multimodal sequence data.

[0059] S130: Perform collaborative embedding representation based on multimodal sequence data to obtain initial feature representation.

[0060] S140: Construct a data association graph of learning behavior based on the initial feature representation.

[0061] S150: Decouple the association graph of learning behavior data by using a graph decoupling neural network to obtain a multimodal decoupling graph.

[0062] S160: Construct an attribute routing mechanism based on the relationships between nodes in the multimodal decoupling graph.

[0063] S170: Update the node embedding representation and graph structure of the multimodal decoupling graph through the attribute routing mechanism to obtain the target feature representation and target graph structure.

[0064] S180: Analyze learning behavior based on the target feature representation and visualize the target graph structure to obtain visualized learning behavior analysis results.

[0065] In this specific embodiment, the first step is to acquire multimodal behavioral data of the learning process of the object to be analyzed. Data is the foundation for driving model analysis. Compared to single-modal data, multimodal data has diverse sources, enabling data complementarity and effectively enhancing the analysis results. This embodiment acquires multimodal behavioral data of the object to be analyzed during the learning process to enhance the complementarity between data. For example, the multimodal behavioral data acquired in this embodiment includes physiological data, psychological data, behavioral data, and mixed data. For physiological data a... tThis embodiment uses bio-data acquisition technology to collect neurobiological data from students, such as eye movement frequency, electroencephalogram (EEG), electrocardiogram (ECG), and skin conductance. This data reflects students' emotions and physical health. Simultaneously, this embodiment uses IoT sensing technology and wearable devices to collect vital signs data related to the focus and activity levels of the subjects being analyzed, such as body temperature, blood pressure, and heart rate. Regarding psychological data (b)... t Unstructured data, such as comments from the target audience, is obtained from the learning platform through web crawling technology. Emotional information, such as facial expressions, is acquired using sentiment recognition technology, and the spoken content of the target audience is collected using automatic speech recognition technology. Next, for the behavioral data c... t This embodiment uses a campus card to record various activities on campus, such as the number of times one enters and exits the library, the amount of electronic resources searched, and the frequency of book borrowing. For hybrid data d... t This embodiment uses log search technology to collect learner clickstream, quiz data, and other data such as the number of times courseware is viewed and the frequency of interaction, based on the data stored on the teaching management platform. Furthermore, this embodiment preprocesses the multimodal behavioral data separately to obtain corresponding multimodal sequence data. Since the acquired data is multimodal behavioral data, and different modalities are represented differently, it is necessary to preprocess the multimodal behavioral data separately, i.e., perform corresponding preprocessing operations on the behavioral data of different modalities to obtain the corresponding multimodal sequence data, in order to facilitate deep feature learning.

[0066] Furthermore, this embodiment performs collaborative embedding representation on multimodal sequence data to obtain initial feature representation. Since there are certain semantic barriers between behavioral data of different modalities, this embodiment performs collaborative embedding representation on the preprocessed multimodal sequence data to map information from multiple modalities into the same embedding space, thereby breaking down semantic barriers between modalities and enhancing their complementarity. Next, this embodiment constructs a learning behavior data association graph based on the initial feature representation. Compared to sequence data, graph-structured data not only retains temporal information but also structural information. Simultaneously, different educational decision-making tasks require different attribute information from the data, such as learning style and learning time, which can be reflected through different graph structures providing decision interpretations. This embodiment constructs a learning behavior data association graph based on the obtained initial feature representation to enhance the data's representational capability. Then, this embodiment decouples the learning behavior data association graph using a graph decoupling neural network to obtain a multimodal decoupled graph. This embodiment addresses the semantic gaps in attention-based interpretation methods by decoupling the learning behavior data association graph. It associates extracted abstract features with specific modal attributes, enriching and refining node representations, reducing message loss during subsequent feature extraction, and enhancing the effectiveness and interpretability of learning behavior analysis. The multimodal decoupling graph includes high-dimensional features with specific semantics. Furthermore, this embodiment constructs an attribute routing mechanism based on the relationships between nodes in the multimodal decoupling graph. After graph decoupling, the embedding representation of each node is divided into specific modal attribute representations, providing a separate channel to guide signal flow and extracting valuable information from the nodes of the learning behavior association graph for learning behavior analysis. This embodiment extracts valuable information for learning behavior analysis by introducing the constructed attribute routing mechanism during embedding propagation to guide signal flow. Next, this embodiment updates and iterates the node embedding representations and graph structure of the multimodal decoupling graph through the attribute routing mechanism to obtain the target feature representation and target graph structure. This effectively controls the transmission of information within the model through the constructed attribute routing mechanism, improving the accuracy and transparency of the learning behavior analysis results. Furthermore, this embodiment performs learning behavior analysis based on the target feature representation and visualizes the target graph structure to obtain visualized learning behavior analysis results, thereby achieving high-performance and highly interpretable multimodal learning behavior analysis and providing reliable analysis and interpretation for the educational decision-making process.

[0067] In some embodiments of the present invention, multimodal behavioral data is preprocessed to obtain corresponding multimodal sequence data, including but not limited to:

[0068] Based on the data format of the multimodal behavioral data, appropriate data cleaning is performed to obtain multimodal cleaned data.

[0069] Multimodal cleaned data is serialized to obtain multimodal sequence data.

[0070] In this specific embodiment, the preprocessing of multimodal behavioral data includes data cleaning and data serialization. Specifically, this embodiment performs corresponding preprocessing for different modal behavioral data. First, this embodiment performs data cleaning according to the data format of the multimodal behavioral data to obtain multimodal cleaned data. Then, the multimodal cleaned data is serialized to obtain multimodal sequence data. For example, this embodiment removes stop words, spaces, and symbols from text data, and simultaneously performs stemming enhancement. Next, this embodiment transforms the enhanced and transformed data into sequence data through tokenization. For example, sentence-level text data uses a word piece tokenizer, and document-level text data uses a sentence piece tokenizer. Additionally, for image data, this embodiment first performs image enhancement processing based on image quality and noise levels, such as super-resolution and dehazing. Next, this embodiment segments the data into sequences in a top-down order through patching. Simultaneously, this embodiment performs signal enhancement operations such as high-pass filtering and low-pass filtering on speech, physiological, and psychological data, and then divides it into specific sequences at certain time intervals. After the above operations, the multimodal behavioral data can be unified into multimodal sequence data. The multimodal sequence data is represented as shown in equation (1):

[0071] S k ={w k,1 ,w k,2 ,…,w k,i ,…,w k,t} (1)

[0072] Where w k,i This represents a time slice in the sequence under the k-mode.

[0073] In some embodiments of the present invention, an initial feature representation is obtained by performing a cooperative embedding representation based on multimodal sequence data, including but not limited to:

[0074] The initial feature representation is obtained by co-embedding different modal sequence data in multimodal sequence data through convolutional neural networks and long short-term memory networks.

[0075] In this specific embodiment, a collaborative embedding representation is performed using convolutional neural networks (CNNs) and long short-term memory networks (LSTMs). Specifically, this embodiment uses convolutional neural networks and long short-term memory networks to perform collaborative embedding representation on the transformed different modal sequence data, such as image modal data and text modal data, so as to map multiple modal information into the same embedding space, enhance the complementarity between modalities, and facilitate subsequent similarity-based mapping and structured operations to obtain the corresponding embedding representation as shown in Equation (2):

[0076] X k ={x k,1 ,x k,2 ,…,x k,i ,…,x k,t} (2)

[0077] Where, in the formula This represents the embedded representation of the data corresponding to time slice i in the k-mode.

[0078] Furthermore, in this embodiment, the embedding representation obtained by the above equation is downsampled by feature pooling in the modality dimension to fuse the local features of each modality, thereby obtaining the initial embedding representation of the displacement of each time slice, which serves as the initial feature representation, as shown in equations (3) and (4) below:

[0079]

[0080]

[0081] Where, in the formula This represents the student's learning behavior state characteristics at time t, i.e., the initial characteristic representation.

[0082] In some embodiments of the present invention, a learning behavior data association graph is constructed based on an initial feature representation, including but not limited to:

[0083] The relationships between time-slice nodes are obtained based on the initial feature representation. Here, a time-slice node is a time slice in the multimodal sequence data.

[0084] Construct an adjacency matrix based on the association relationships.

[0085] Construct a data association graph of learning behavior based on the adjacency matrix and time slice nodes.

[0086] In this specific embodiment, the association relationship between time slice nodes is first obtained based on the initial feature representation. Specifically, each time slice w in the multimodal sequence data... iAs nodes in the learning behavior data association graph, specifically time slice nodes. Then, in this embodiment, an adjacency matrix is ​​constructed based on the association relationships, thereby building the learning behavior data association graph based on the adjacency matrix and the time slice nodes. Specifically, the association... Figure 1 Generally, it consists of nodes and the edges connecting the nodes. This embodiment uses each time slice w in the multimodal sequence data. i As node v in the learning behavior data association graph i Connecting edges represent the relationships between nodes, typically represented by an adjacency matrix. The adjacency matrix can also represent the degree of dependency between nodes. For example, v i With v j If there is a relationship between them, then A ij =1, otherwise A ij =0. Furthermore, in order to obtain the initial graph structure, this embodiment is based on the initial feature representation of the nodes. The similarity matrix is ​​used to calculate the cosine similarity between node pairs in the embedding space, resulting in a weighted adjacency matrix, as shown in equation (5) below:

[0087]

[0088] Where w p It is a learnable weight parameter, h i and h j Representative node v i With v j The initial embedding is represented by cos(·), where cos(·) represents the cosine similarity.

[0089] Furthermore, to preserve the sequence information of the time-series data, the nodes in the graph are connected according to the sequence order in the multimodal sequence data, thereby constructing a learning behavior data association graph based on the adjacency matrix and time-slice nodes. Simultaneously, the similarity-based graph construction method is dynamic and heuristic, updating according to the node information, thus allowing the final learned graph structure to be visualized as structural information, providing a degree of interpretability. After the above operations, the multimodal sequence data S... k Convert it into a learning behavior data association graph consisting of time slice nodes and connecting edges. in Let n represent the number of nodes in the graph, and let n represent the number of nodes in the sample. The relationships between these nodes can be described as follows: Use adjacency matrix This indicates that the initial embedding representation H is obtained simultaneously. 0 Feature matrices that can be used as graph nodes Where c represents the dimension of the feature vector.

[0090] It should be noted that, unlike the attention mechanism which directly uses the original embeddings of nodes to calculate the contribution of each node to the learning behavior analysis, this embodiment uses a graph decoupling neural network to decouple the learning behavior data association graph, thus obtaining a multimodal decoupling graph. Specifically, this embodiment first encodes a single dimension of the initial embedding representation of a node as the input signal of the neuron into the hidden neuron. The initial embedding representation of a node can be decomposed as shown in the following equation (6):

[0091] h i ={h i,1 ,h i,2 ,…,h i,m ,…h i,c} (6)

[0092] Where, in the formula h represents the initial embedding representation of a node. i,m Represents node v i The m-th input signal.

[0093] Furthermore, in this embodiment, the initial embedding representation of each node is divided into blocks based on the linear model as the neuron. To associate the neuron's output signal with specific semantics, this embodiment uses a supervised task to link the neuron's output signal with specific modal attribute features. Here, the specific modal attribute corresponds to the embedding representation x of the original modal data time node. k,i The feature representations on the time slices of the original modality data are used as supervision labels and associated with specific modality attributes. Therefore, the neuron representation for a specific modality is shown in Equation (7):

[0094] h i,k =W k h i +b i,k (7)

[0095] Where k is the modal attribute information indicator, W k It is a trainable weight matrix, b i,k It is a learnable bias vector, and the linear model can derive the contribution value of each input signal from the weight matrix, thus having good interpretability.

[0096] Furthermore, since signal extraction without supervision cannot guarantee that the extracted attribute information is consistent with the desired specific attribute, this embodiment introduces a multi-label classification loss in the signal extraction process of neurons, using the original modal attribute labels to constrain the attribute feature extraction process. The specific formulas are shown in equations (8) and (9) below:

[0097]

[0098]

[0099] In the above formula, K is the number of attribute information, and P i,k These are real attribute tags. This is the probability of predicting that attribute. It's easy to understand that the graph decoupling operation in this specific embodiment can be understood as the association graph of the learning behavior data. Decomposition yields multiple modal decoupling maps G. k ={V,E}, although the connection structure of nodes and edges remains unchanged, the initial embedding representation h of nodes on each modal decoupling graph is changed. i,k Since these represent specific modal attributes, there are differences between modalities. Therefore, graph decoupling can overcome the semantic gaps in attention-based interpretation methods by associating extracted abstract features with specific modal attributes, enriching and refining node representations, mitigating information loss in subsequent feature extraction processes, enhancing the effectiveness of learning behavior analysis, and improving interpretability.

[0100] In some embodiments of the present invention, an attribute routing mechanism is constructed based on the relationships between nodes in a multimodal decoupling graph, including but not limited to:

[0101] Calculate the first difference score matrix between nodes in each modal decoupling graph. The first difference score matrix is ​​a mask matrix for intramodal information transmission.

[0102] Calculate the second difference score matrix of the same node in the multimodal decoupling graph across different modes. The second difference score matrix is ​​a mask matrix for cross-modal information transmission.

[0103] In this specific embodiment, the attribute routing mechanism is first constructed based on the relationship between the nodes in the multimodal decoupling graph. Specifically, the first difference score matrix between the nodes in each modal decoupling graph is calculated, and the first difference score matrix is ​​used as the mask matrix for intramodal information transmission. In addition, the second difference score matrix of the same node in different modalities in the multimodal decoupling graph is calculated, and the second difference score matrix is ​​used as the mask matrix for cross-modal information transmission. During the learning process of the object to be analyzed, the learning behavior will be affected by the previous learning behavior. In order to realize the temporal correlation of the learning behavior information, the first difference score matrix is ​​constructed to realize the transmission of the learning behavior information of the object to be analyzed in the time dimension. The difference score matrix between each node interaction, i.e., the first difference score matrix, is calculated on a single decoupled modal decoupling graph, and is used as the mask matrix for intramodal information transmission. The calculation formula is shown in the following formula (10):

[0104]

[0105] Where h i,k and h j,k represents the initial embedding representation on a given modal decoupling graph, where k is the modal indicator.

[0106] Furthermore, the learning behavior data of the object to be analyzed exhibits complementarity and difference across different modalities. In this embodiment, a second difference score matrix is ​​calculated to realize the transfer of the learning behavior of the object to be analyzed across the modal dimension. In this embodiment, the difference score matrix of the same node in different modalities is calculated on each modal decoupling graph, that is, the second difference score matrix, which serves as a mask matrix for cross-modal information transfer. The calculation formula is shown in the following formula (11):

[0107]

[0108] Where h i,k and h j,k represents the initial embedding representation on a given modal decoupling graph, k is the modal indicator, and K represents the number of different modal decoupling graphs.

[0109] In some embodiments of the present invention, the node embedding representation and graph structure of the multimodal decoupling graph are updated iteratively through an attribute routing mechanism to obtain the target feature representation and target graph structure, including but not limited to:

[0110] Based on the transmission of the control information from the first difference fraction matrix in the time dimension of each modal decoupling diagram within the multimodal decoupling diagram, the time dimension representation update is obtained.

[0111] Based on the transmission of the second difference fraction matrix control information in the modal dimensions between the various modal decoupling diagrams in the multimodal decoupling diagram, the modal dimension representation update is obtained.

[0112] The node embedding representation is updated based on the time dimension update and the modality dimension update to obtain the target feature representation.

[0113] The target graph structure is obtained by updating the graph structure based on the target feature representation.

[0114] In this specific embodiment, the first step is to transmit the information in the first difference score matrix within each modal decoupling graph in the time dimension to obtain the time dimension representation update. Next, the second difference score matrix is ​​used to transmit the information in the second difference score matrix between each modal decoupling graph in the modal dimension to obtain the modal dimension representation update. Then, the node embedding representation and graph structure are updated according to the time dimension representation update and the modal dimension representation update respectively to obtain the corresponding target feature representation and target graph structure. Specifically, after calculating the mask matrix for information transmission within the modality, i.e., the first difference score matrix, this embodiment uses the mask matrix to control the transmission in the modal decoupling graph to update the node representation. The formula for updating the node representation in the time dimension within the modality is shown in equation (12) below:

[0115]

[0116] Where, in the formula The node embedding representation represents the node's update over time. Represents node v i The set of neighboring nodes.

[0117] Furthermore, after calculating the mask matrix for cross-modal information transmission, i.e. the second difference score matrix, this embodiment controls the transmission of information between decoupling graphs of different modalities based on the mask matrix to perform node representation updates. The node representation update in the modal dimension is shown in the following equation (13):

[0118]

[0119] Where, in the formula The node embedding representation that represents the update of a node in the modality dimension. Represents node v i The set of neighboring nodes.

[0120] Furthermore, in this embodiment, the node embedding representation, i.e. the target feature representation, is obtained by updating the obtained time dimension representation and modality dimension representation through iterative updates. The updated target feature representation is shown in equation (14) below:

[0121]

[0122] Where μ represents the learnable bias vector, This represents the temporary feature embedding representation of the node.

[0123] Furthermore, after calculating the temporary feature update embedding representation of the target node, this embodiment updates the final node embedding representation according to the information update principle on the graph. The updated node embedding representation on the entire graph is shown in Equation (15):

[0124]

[0125] Where, in the formula This represents the node feature representation of the upper layer in the graph structure. This represents the node feature representation of the current network layer.

[0126] Furthermore, in this embodiment, based on the updated target node feature representation of the current network layer, the graph structure in the current state is calculated and updated again according to Equation (5), that is, the target graph structure.

[0127] In some embodiments of the present invention, learning behavior analysis is performed based on the target feature representation, and the target graph structure is visualized to obtain visualized learning behavior analysis results, including but not limited to:

[0128] The target feature representation is input into a preset graph to decouple the learning behavior analysis model for learning behavior analysis, and the learning behavior analysis results are obtained.

[0129] The target graph structure is visualized based on the mask matrix to obtain a preset visualization image. This preset visualization image includes a heatmap and a structural visualization image.

[0130] Visualized learning behavior analysis results are obtained based on the learning behavior analysis results and preset visualization images.

[0131] In this specific embodiment, the target feature representation is first decoupled from the learning behavior analysis model information learning behavior analysis by inputting a preset graph. Specifically, different educational decision-making needs require different learning behavior analysis tasks, which in turn require different graph structure inputs. The overall educational decision-making tasks can be divided into node classification tasks, such as grade prediction and knowledge tracking, and graph classification tasks, such as sentiment analysis and learning prediction. At the same time, this also corresponds to different levels of interpretation. In order to realize the decision-making and interpretation generation of multiple tasks in a unified framework, the graph tasks are transformed into index generation tasks, generating an overall classification index and a node sequence index for decision-making, as shown in the following equation (16):

[0132]

[0133] In this equation, softmax is used to generate the index. The graph classification task performs an aggregation operation of agg on the node feature matrix to obtain a unique vector representation for generating the overall classification index. Furthermore, this embodiment uses the cross-entropy loss function to constrain the classifier, as shown in equation (17):

[0134]

[0135] Wherein, D train S is the defined training set, and S is the training sample in the training set. The learning behavior analysis model constructed in the above way, namely the pre-defined graph decoupled learning behavior analysis model, can realize a variety of educational decision-making tasks.

[0136] Furthermore, in order to train the preset graph decoupling learning behavior analysis model, this embodiment divides the collected multimodal behavior data into training and testing sets in the deep learning modeling of multimodal learning behavior, and trains and verifies the preset graph decoupling learning behavior analysis model designed according to the above steps. During the training process, the learning behavior analysis model adopts a dynamic association graph structure. During the training process, the graph structure will be iteratively updated according to the learned node features. At the same time, in order to further enhance the effect of graph decoupling and routing mechanism, the attribute routing mechanism designed in U iterations updates the node representation. Furthermore, in order to achieve end-to-end unified training, a unified objective function and optimization path for the multimodal learning behavior analysis model are adopted. The final objective function of the learning behavior analysis model is shown in the following equation (18):

[0137]

[0138] Where α and β are the classification losses, respectively. and decoupling loss The weight coefficients are such that, in this embodiment, the iteration number of the attribute routing mechanism is U = 3.

[0139] Furthermore, in order to generate a structured interpretation of the learning analysis results, given a pre-trained preset graph, the learning behavior analysis model Φ and the learning behavior analysis prediction result Y are decoupled, and the subgraph G of the sample graph is identified. s The node feature X that has the most influence on the prediction of the predefined graph decoupled learning behavior analysis model Φ. s The explanation is generated by using a subset of the subsets, and the overall explanation process is formally optimized using mutual information (MI) as shown in Equation (19):

[0140]

[0141] In this equation, f(·) represents the entropy calculation function. It should be noted that f(Y) is fixed because the decision parameters of the pre-trained graph decoupling learning behavior analysis model are fixed. Therefore, the overall optimization function becomes minimizing f(Y|G=G). s X = X s As shown in equation (20):

[0142]

[0143] Furthermore, in this embodiment, the target graph structure is visualized based on the mask matrix to obtain a preset visualization image. The preset visualization image includes a heatmap and a structural visualization graph. Specifically, this embodiment uses a constructed attribute routing mechanism to achieve information flow transmission and graph structure updates. Therefore, this embodiment obtains the preset visualization image by visualizing the mask matrix of the attribute routing mechanism. This embodiment extends the traditional method of using an attention mechanism based on feature importance to characterize node and edge weights, giving features specific semantics and overcoming the problems of inaccurate and non-unique interpretations caused by the lack of semantics in attention weights. It also characterizes the graph structure at a fine-grained level. For example, this embodiment randomly selects an object to be analyzed and collects its learning behavior data. The student's multimodal learning behavior data is preprocessed and initially embedded, and an initial graph structure is constructed according to feature similarity for information transmission. Further, after the graph is constructed, the fused feature representation is decoupled using a supervised task, the node feature representation is updated, and an H is generated. k ={h 1,k ,…,h i,k Then, in this embodiment, the feature representation is input into the training model. After processing by the attribute routing mechanism, the mask values ​​q for each node and region under a specific modality and the mask values ​​c between multiple modalities are output. Next, after the preset graph decoupling learning behavior analysis model is trained, it is input into the interpretation generation optimization process to obtain the subgraph G that has the greatest impact on the result. s and the most influential subset of node features X sThen, this embodiment uses matplotlib to draw the heatmap and corresponding graph structure corresponding to the mask values ​​of the obtained attribute routing mechanism, and gives different color labels according to the magnitude of the numerical weight, thereby visualizing the target graph structure and obtaining a preset visualization image. It should be noted that the preset visualization image obtained in this embodiment is divided into a heatmap and a structure visualization image. Among them, the darker the color, the greater the weight, and the greater the impact on the learning analysis results. On the one hand, this visualization can intuitively reflect the importance of different nodes and regions, which is conducive to guiding information transmission and reflecting the importance of specific information. On the other hand, different modalities also have different weights, which can reflect the importance between modalities from the heatmap, thereby realizing the importance of information of a specific node in a specific modality. Furthermore, this embodiment obtains the visualized learning behavior analysis results by combining the learning behavior analysis results and the preset visualization image.

[0144] It should be noted that in some embodiments of the present invention, this embodiment supports multimodal learning behavior analysis through a preset graph decoupled learning behavior analysis model, and generates visualization and natural language interactive explanations based on the message flow of the model decision-making process. For example, suppose an object Z to be analyzed is learning a course, and the system backend records variables related to the student's learning process. Now, it is necessary to predict the final exam score of object Z (divided into five levels: failing, passing, average, good, and excellent). It is necessary to perform shallow feature analysis on data from different modalities, then normalize and decouple the shallow features for representation learning, and fuse them according to a routing mechanism to obtain a global feature representation as the basis for analysis. In this embodiment, for data from different modalities, data cleaning is first performed, and different deep learning algorithms are used to obtain the initial node vector representation X, constructing a graph structure G. Then, in this embodiment, the nodes in the graph structure are decoupled according to the existing multimodal data attributes to obtain the decoupled representation h. i,k The node representation is updated according to the decoupled features and the routing mechanism, and the node features are fused to obtain h. i Furthermore, based on feature h i Make overall and partial decisions, calculate (G) s ,X s And an explanation is given through visualization based on the updated graph structure.

[0145] An embodiment of the present invention also provides a multimodal learning behavior analysis system, comprising:

[0146] The data acquisition module is used to acquire multimodal behavioral data of the learning process of the object to be analyzed.

[0147] The preprocessing module is used to preprocess multimodal behavioral data to obtain corresponding multimodal sequence data.

[0148] The embedding representation module is used to perform collaborative embedding representation based on multimodal sequence data to obtain initial feature representations.

[0149] The association graph construction module is used to construct an association graph of learning behavior data based on the initial feature representation.

[0150] The decoupling module is used to decouple the association graph of learning behavior data through the graph decoupling neural network to obtain a multimodal decoupling graph.

[0151] The routing construction module builds an attribute routing mechanism based on the relationships between nodes in the multimodal decoupling graph.

[0152] The routing update module is used to update the node embedding representation and graph structure of the multimodal decoupled graph through the attribute routing mechanism to obtain the target feature representation and target graph structure.

[0153] The results analysis module is used to analyze learning behavior based on the target feature representation and visualize the target graph structure to obtain visualized learning behavior analysis results.

[0154] Reference Figure 2 An embodiment of the present invention also provides a multimodal learning behavior analysis system, comprising:

[0155] At least one processor 210.

[0156] At least one memory 220 is used to store at least one program.

[0157] When at least one program is executed by at least one processor 210, the at least one processor 210 implements the multimodal learning behavior analysis method as described in the above embodiments.

[0158] An embodiment of the present invention also provides a computer-readable storage medium storing computer-executable instructions that are executed by one or more control processors, for example, performing the steps described in the above embodiments.

[0159] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0160] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A multi-modal learning behavior analysis method, characterized in that, The method comprises the following steps: acquiring multi-modal behavior data of a learning process of an object to be analyzed; preprocessing the multi-modal behavior data to obtain corresponding multi-modal sequence data; performing collaborative embedding representation according to the multi-modal sequence data to obtain initial feature representation; constructing a learning behavior data association graph according to the initial feature representation; decoupling the learning behavior data association graph by a graph decoupling neural network to obtain a multi-modal decoupled graph; constructing an attribute routing mechanism according to the relationship between nodes in the multi-modal decoupled graph; updating and iterating the node embedding representation and the graph structure of the multi-modal decoupled graph by the attribute routing mechanism to obtain target feature representation and target graph structure; performing learning behavior analysis according to the target feature representation and visualizing the target graph structure to obtain visualized learning behavior analysis results; wherein the constructing a learning behavior data association graph according to the initial feature representation comprises: obtaining the association relationship between time slice nodes according to the initial feature representation; wherein the time slice nodes are time slices in the multi-modal sequence data; constructing an adjacency matrix according to the association relationship; constructing the learning behavior data association graph according to the adjacency matrix and the time slice nodes; wherein the decoupling the learning behavior data association graph by a graph decoupling neural network to obtain a multi-modal decoupled graph comprises: encoding a single dimension of the initial embedding representation of a node as an input signal of a neuron to a hidden neuron; dividing the initial embedding representation of each node into blocks according to a linear model as a neuron; associating the output signal of the neuron with specific modal attribute features through a supervised task; introducing a multi-label classification loss in the signal extraction process of the neuron to constrain the attribute feature extraction process through the original modal attribute label; wherein the constructing an attribute routing mechanism according to the relationship between nodes in the multi-modal decoupled graph comprises: calculating a first difference score matrix between the nodes of each modal decoupled graph in the multi-modal decoupled graph; wherein the first difference score matrix is a mask matrix of intra-modal information transmission; calculating a second difference score matrix of the same nodes in different modalities in the multi-modal decoupled graph; wherein the second difference score matrix is a mask matrix of cross-modal information transmission; wherein the updating and iterating the node embedding representation and the graph structure of the multi-modal decoupled graph by the attribute routing mechanism to obtain target feature representation and target graph structure comprises: controlling the transmission of information in the time dimension within each modal decoupled graph in the multi-modal decoupled graph according to the first difference score matrix to obtain time dimension representation update; controlling the transmission of information in the modal dimension between each modal decoupled graph in the multi-modal decoupled graph according to the second difference score matrix to obtain modal dimension representation update; updating the node embedding representation according to the time dimension representation update and the modal dimension representation update to obtain the target feature representation; updating the graph structure according to the target feature representation to obtain the target graph structure. 2.The multi-modal learning behavior analysis method of claim 1, wherein, The preprocessing of the multi-modal behavior data obtains corresponding multi-modal sequence data, including: According to the data form of the multi-modal behavior data, corresponding data cleaning is performed to obtain multi-modal cleaning data; The multi-modal cleaning data is serialized to obtain the multi-modal sequence data. 3.The multi-modal learning behavior analysis method of claim 1, wherein, The collaborative embedding representation according to the multi-modal sequence data obtains an initial feature representation, including: The different modal sequence data in the multi-modal sequence data is subjected to the collaborative embedding representation through a convolutional neural network and a long short-term memory network to obtain the initial feature representation. 4.The multi-modal learning behavior analysis method of claim 1, wherein, The learning behavior analysis according to the target feature representation and the visualization of the target graph structure obtain a visualized learning behavior analysis result, including: The target feature representation is input into a preset graph decoupling learning behavior analysis model to perform learning behavior analysis, and a learning behavior analysis result is obtained; According to the mask matrix, the target graph structure is visualized to obtain a preset visual image; wherein the preset visual image includes a heat map and a structure visualization graph; According to the learning behavior analysis result and the preset visual image, the visualized learning behavior analysis result is obtained.

5. A multi-modal learning behavior analysis system, characterized by, It includes: A data acquisition module for acquiring multi-modal behavior data of a learning process of an object to be analyzed; A preprocessing module for preprocessing the multi-modal behavior data to obtain corresponding multi-modal sequence data; An embedding representation module for collaborative embedding representation according to the multi-modal sequence data to obtain an initial feature representation; An associated graph construction module for constructing a learning behavior data associated graph according to the initial feature representation; A decoupling module for decoupling the learning behavior data associated graph through a graph decoupling neural network to obtain a multi-modal decoupled graph; A routing construction module for constructing an attribute routing mechanism according to the relationship between each node in the multi-modal decoupled graph; A routing update module for updating and iterating the node embedding representation and the graph structure of the multi-modal decoupled graph through the attribute routing mechanism to obtain a target feature representation and a target graph structure; A result analysis module for learning behavior analysis according to the target feature representation and visualizing the target graph structure to obtain a visualized learning behavior analysis result; Wherein, the construction of the learning behavior data associated graph according to the initial feature representation includes: According to the initial feature representation, the association relationship between time slice nodes is obtained; wherein the time slice node is a time slice in the multi-modal sequence data; According to the association relationship, an adjacency matrix is constructed; According to the adjacency matrix and the time slice node, the learning behavior data associated graph is constructed; Wherein, the decoupling of the learning behavior data associated graph through the graph decoupling neural network obtains a multi-modal decoupled graph, including: The single dimension of the initial embedding representation of the node is encoded as the input signal of the neuron to the hidden neuron; According to the linear model as the neuron, the initial embedding representation of each node is divided into blocks; Through a supervised task, the neuron output signal is associated with specific modal attribute features. In the signal extraction process of neurons, a multi-label classification loss is introduced, and the attribute feature extraction process is constrained by the original modal attribute label. The attribute routing mechanism is constructed according to the relationship between the nodes in the multi-modal decoupling graph, including: Calculate the first difference score matrix between the nodes of each modal decoupling graph in the multi-modal decoupling graph; wherein the first difference score matrix is a mask matrix of intra-modal information transmission; Calculate the second difference score matrix of the same node in different modalities in the multi-modal decoupling graph; wherein the second difference score matrix is a mask matrix of cross-modal information transmission; Wherein, the node embedding representation and the graph structure of the multi-modal decoupling graph are updated iteratively through the attribute routing mechanism to obtain the target feature representation and the target graph structure, including: According to the first difference score matrix, control the transmission of information in the time dimension of each modal decoupling graph in the multi-modal decoupling graph to obtain the time dimension representation update; According to the second difference score matrix, control the transmission of information in the modal dimension between each modal decoupling graph in the multi-modal decoupling graph to obtain the modal dimension representation update; According to the time dimension representation update and the modal dimension representation update, update the node embedding representation to obtain the target feature representation; According to the target feature representation, update the graph structure to obtain the target graph structure.

6. A multi-modal learning behavior analysis system, characterized by, Comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, at least one processor implements the multi-modal learning behavior analysis method according to any one of claims 1 to 4.

7. A computer storage medium storing a processor-executable program, characterized in that, The program executable by the processor when executed by the processor is used to implement the multi-modal learning behavior analysis method according to any one of claims 1 to 4.

Images