Gated self-attention contrastive graph neural network concept prerequisite relationship learning method

Abstract

The application discloses a concept prerequisite relation learning method based on a gated self-attention contrast graph neural network concept, relates to the technical field of natural language processing, and mainly comprises the following steps: generating a text embedding set by using a text encoder according to a target concept and a description of the target concept, and constructing a prerequisite relation graph by using prerequisite relation transitivity; taking the text embedding set as an initial node embedding set, updating the node embedding set by using a contrast loss function and a graph neural network to obtain all concept node representations; and obtaining prerequisite relations between concepts by using a binary cross-entropy loss function and a twin network. The concept prerequisite relation learning method based on the gated self-attention contrast graph neural network concept can learn prerequisite relations of concepts from sparse and unstructured knowledge concepts.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically, to a method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network. Background Technology

[0002] Existing technologies are mainly divided into two categories. First, there are handcrafted feature-based methods: Early research focused on using handcrafted features to detect prerequisite relationships between concepts. For example, RefD proposed contextual, structural, and semantic features, extracting prerequisite relationships between concepts by analyzing the structure and semantics of text. Clickstream methods also utilize Wikipedia's clickstream data to define features and use machine learning to infer prerequisite relationships between concepts. This approach captures associations between concepts by analyzing user behavior (such as clickstreams). However, these handcrafted feature-based methods rely heavily on feature engineering, requiring expert knowledge to define and extract relevant features. This is not only time-consuming and labor-intensive but may also fail to fully capture the complex relationships between concepts, as feature selection and extraction are often limited by subjective factors and domain knowledge. Second, there are deep learning-based methods: To avoid the tedious feature engineering process, more and more researchers have applied deep learning methods to learn concept prerequisite relationships in recent years. For example, MHAVGAE uses variational graph autoencoders to learn concept prerequisite relationships from the curriculum; methods such as PREEQ, M3, Conlearn, and HGAPNet use different deep learning models (such as supervised learning methods and graph neural networks) to learn concept prerequisite relationships, attempting to capture complex patterns between concepts.

[0003] However, existing technologies have the following drawbacks: First, sparsity: concept prerequisites are sparse (the number of prerequisites is much smaller than the total number of concept pairs). Existing methods rely on a large amount of training data, resulting in long information propagation paths and severe attenuation in sparse graphs, limiting aggregation capabilities. Second, insufficient transitivity modeling: they cannot effectively capture multi-hop transitive relationships between concepts. For example, if maximum likelihood is a prerequisite for a Hidden Markov Model (HMM), and the HMM is a prerequisite for a Conditional Random Field (CRF), then maximum likelihood should be a prerequisite for the CRF. Existing methods cannot simulate such multi-hop relationships. Third, limited application scenarios: most methods are designed for structured learning materials (such as textbooks) and are difficult to apply to sparse, unstructured knowledge concepts. Summary of the Invention

[0004] The purpose of this invention is to provide a concept prerequisite relation learning method based on gated self-attention contrastive graph neural network, which can learn the prerequisite relation of concepts from sparse and unstructured knowledge concepts.

[0005] This invention provides a method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network, comprising the following steps: S1: Based on the target concept and its description, a text embedding set is generated using a text encoder, and a prerequisite relationship graph is constructed using the transitivity of prerequisite relationships; S2: Based on the prerequisite relationship graph, the text embedding set is used as the initial node embedding set, and the node embedding set is updated using a contrastive loss function and a graph neural network to obtain the representations of all concept nodes; S3: Based on the representations of all concept nodes, the prerequisite relationships between concepts are obtained using a binary cross-entropy loss function and a Siamese network.

[0006] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network.

[0007] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network.

[0008] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network.

[0009] This invention also provides an application of the above-described learning method based on the conceptual prerequisite relationship of a gated self-attention contrastive graph neural network, which is applied to the development of a learning system.

[0010] The concept prerequisite relation learning method based on gated self-attention contrastive graph neural network provided by this invention has the following beneficial effects:

[0011] This invention combines contrastive learning with graph neural networks to propose a method for learning concept prerequisites without relying on structured learning materials. This method not only expands its application scenarios but also effectively guides the training of graph neural networks through a contrastive loss function, helping the model better learn the prerequisite relationships between concepts.

[0012] This invention introduces a gating mechanism and a multi-head self-attention mechanism to enhance the model's ability to update node representations on sparse graphs. The gating mechanism can capture multi-hop prerequisite relationships between concepts, while the multi-head self-attention mechanism can focus on multiple relationships and features in parallel, thereby more effectively capturing and integrating these similar relationships. This improvement solves the problems of long information propagation paths and information attenuation in traditional graph neural networks when processing sparse graphs, improving the model's ability to learn and capture concept relationships, and is therefore the focus of this invention.

[0013] This invention utilizes existing conceptual prerequisite pairs to construct a conceptual prerequisite relationship graph and updates the concept representations through a gated graph neural network. This construction method enables the model to better capture the dependencies between concepts, especially in sparse and unstructured knowledge concepts. By constructing a conceptual prerequisite relationship graph and utilizing a gated graph neural network for information propagation, this invention achieves the learning of prerequisite relationships from sparse and unstructured knowledge concepts. Attached Figure Description

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0015] Figure 1 This is a flowchart of the concept prerequisite relation learning method based on gated self-attention contrastive graph neural network provided by the present invention;

[0016] Figure 2 This is a schematic diagram of the overall framework design of the concept prerequisite relation learning method based on gated self-attention contrast graph neural network provided by the present invention;

[0017] Figure 3 This is a schematic diagram of the original and extended prerequisite relationships provided by the present invention;

[0018] Figure 4 This is a schematic diagram of the conceptual prerequisite diagram structure and connection matrix provided by the present invention;

[0019] Figure 5 This is a structural block diagram of the computer device provided by the present invention. Detailed Implementation

[0020] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 This diagram illustrates the concept prerequisite relationship learning method based on a gated self-attention contrastive graph neural network according to this embodiment. In this embodiment, the concept prerequisite relationship learning method based on a gated self-attention contrastive graph neural network includes the following steps:

[0022] S1: Based on the target concept and its description, generate a set of text embeddings using a text encoder, and construct a prerequisite relation graph using the transitivity of prerequisite relations;

[0023] S2: Based on the aforementioned prerequisite relation graph, using the text embedding set as the initial node embedding set, the node embedding set is updated using a contrastive loss function and a graph neural network to obtain the representations of all concept nodes;

[0024] In one exemplary embodiment, step S2 specifically includes:

[0025] S21: Train the graph neural network using the contrastive loss function to obtain a trained graph neural network;

[0026] S22: Based on the prerequisite relation graph, using the text embedding set as the initial node embedding set, update the node embedding set using the trained graph neural network;

[0027] In one exemplary embodiment, the graph neural network is as follows:

[0028] ,

[0029] ,

[0030] ,

[0031] ,

[0032] ,

[0033] ,

[0034] ,

[0035] in, Indicates an update to the door; Indicates that the door is being reset; Indicates the candidate hidden state; Represents a node At time step The representation of the node (i.e., the final hidden state update), that is, the updated node representation; and Representing nodes respectively and At time step The representation of; This represents the Sigmid activation function; , , and , , These are the parameters to be learned; , , , respectively, represent the linear transformation parameter matrices of the graph neighbor aggregation part, which are learnable parameter matrices; Represents a node The set of neighboring nodes; It is a node with neighboring nodes Edge weights between them; Represents the hyperbolic tangent activation function; Attention weights; The input feature matrix; , , It is the projection matrix under the head h, and the linear transformation of the ReLU activation function; The dimension of the Query / Key vector, used for scaling to avoid gradient explosion; This is the feature matrix output by the h-th head; It is a normalized exponential function; The final output feature matrix; The output is the result of concatenating all heads along the feature dimension; This is the feature matrix output by the m-th head; To output a linear transformation matrix (projecting the spliced ​​result back to the original or desired output dimension).

[0036] In one exemplary embodiment, the contrastive loss function is as follows:

[0037] ,

[0038] ,

[0039] in, To compare loss functions; The number of knowledge concepts (i.e., the number of knowledge concepts in the dataset); Represents cross-entropy operation; and Represent matrices respectively The row and number List; Representation matrix The row and number Column elements are used to measure the similarity between text embeddings and graph node embeddings; and These represent the target distributions of text embedding and node embedding, respectively. To control the regularization of the similarity matrix The tradeoff coefficient between the identity matrix and the set; Represents a set of embedding pairs. This represents the node embedding in the graph. The text embedding corresponding to the node; This is the temperature coefficient.

[0040] S3: Based on the representation of all the concept nodes, the prerequisite relationships between concepts are obtained using the binary cross-entropy loss function and Siamese network;

[0041] In one exemplary embodiment, step S3 specifically includes:

[0042] S31: The Siamese network is trained using the binary cross-entropy loss function to obtain a trained Siamese network;

[0043] S32: Based on the representations of all concept nodes, the preconditions between concepts are obtained using the trained Siamese network;

[0044] In one exemplary embodiment, the twin network is as follows:

[0045] ,

[0046] ,

[0047] in, This represents the result after projection by the feedforward network; Indicates the activation function; Indicates the first Node embeddings corresponding to each concept; They represent the first The and the first One concept; Representing concept pairs middle yes The probability of the prerequisites; , , , These are the parameterized weight matrix and the bias vector, respectively. yes Type function; Indicates element-wise multiplication; Represents the connection of vectors

[0048] In one exemplary embodiment, the binary cross-entropy loss function is as follows:

[0049] ,

[0050] in, Represents the binary cross-entropy loss function; This represents the training dataset for this domain; It indicates the real prerequisite relationship between pairs of concepts.

[0051] In some embodiments, the above-described method for learning concept prerequisites based on gated self-attention contrastive graph neural networks can also be implemented in the following ways.

[0052] In this embodiment, the concept prerequisite relation learning method based on gated self-attention contrastive graph neural network is applied to the development of a learning system, including the following steps:

[0053] Step 1: Based on the target concept and its description, generate a set of text embeddings using a text encoder, and construct a prerequisite relation graph using the transitivity of prerequisite relations;

[0054] Among them, the target concept and its description are respectively a set of geological concepts and their descriptive texts;

[0055] Step 2: Based on the prerequisite relation graph, using the text embedding set as the initial node embedding set, update the node embedding set using the contrastive loss function and graph neural network to obtain the representation of all concept nodes;

[0056] Step 3: Based on the representation of all concept nodes, use the binary cross-entropy loss function and Siamese network to obtain the prerequisite relationships between concepts.

[0057] In some embodiments, the above-described method for learning concept prerequisites based on gated self-attention contrastive graph neural networks can also be implemented in the following ways.

[0058] In this embodiment, the overall framework design model framework is as follows: Figure 2 As shown, this embodiment employs two encoders—a text encoder and a graph encoder. First, the first 200 words of each concept are obtained from the Wikipedia API, and the text embedding is obtained using the text encoder. Then, a concept prerequisite graph is constructed, and the concept representation is updated in the graph neural network. Because the prerequisite relationships between concepts are transitive, GAC-GNN, as a graph encoder, allows for the aggregation of node information beyond multiple steps by setting the step size. Furthermore, a concept fusion module is added to the graph neural network to utilize similar prerequisite relationships between related concepts. The GAC-GNN model uses a contrastive loss function.

[0059] Finally, the concept representations are fed into a Siamese network for prerequisite relation classification.

[0060] It should be noted that the concept set Prerequisite relationship It is a set of prerequisite relations between concepts, in which Representing concepts It is a concept The prerequisites. Based on the transitivity of prerequisites, extend the prerequisite pairs, if... yes The prerequisite relationship, yes The prerequisite relationship, then Too The prerequisite relationship. Conceptual text Depend on In The problem consists of descriptions corresponding to each concept, presented as a text document. The document is the Wikipedia description of the concept. This problem can be formally defined as: given a set of concepts... The goal is to learn functions This function can determine Is it? The prerequisite concept; considering this as a binary classification task. Specifically, if ,but yes The concept of prerequisites, otherwise no The concept of prerequisites. For example... Figure 3 The diagram shows the original prerequisites and the extended prerequisites.

[0061] For text encoding, in this embodiment, Hugging Face provides a rich set of large language models and related tools and resources, which are widely used in various NLP tasks. Among existing text model rankings, this embodiment comprehensively considers the performance and size of these models and ultimately selects mxbai-embed-large-v1 as the text encoder to generate embeddings for concept descriptions.

[0062] For graph encoding, in this embodiment, graph neural networks can capture the relationships between nodes, making them suitable for this task. In prerequisite relationship mining, a directed concept prerequisite relationship graph is first constructed using existing prerequisite relationships. Then, a graph neural network is applied to the concept prerequisite relationship graph to update the concept representation, integrating the concept prerequisite relationships into the model. GGNN (Gated Graph Neural Network) uses gated recurrent units to process node information in the graph. By iteratively updating the state of nodes, it achieves deeper information propagation and can better capture complex dependencies and long-range information. Related concepts have similar prerequisite relationships, so this embodiment designs a concept fusion module, referring to the graph encoder as GAC-GNN. Figure 4 The diagram shown contains the concept prerequisites graph structure and the connection matrix. and A schematic diagram;

[0063] Specifically, let These represent the weighted connections of the incoming and outgoing edges in the concept prerequisite graph, respectively. For example, consider the concept prerequisite pairs... Since concepts may appear repeatedly in concept prerequisite pairs, this embodiment assigns a normalized weight to each edge, calculated as the number of times the edge appears divided by the out-degree of the edge's starting node. For example... have and Two incoming edges, then middle and Value .

[0064] Next, this embodiment will describe how to update the concept representation using a gated graph neural network. First, each concept... Embedded into a feature space, vector Representing concepts of 3D real-valued eigenvectors. For the steps in the concept prerequisite diagram... Each concept at the location , by the connection matrix and Given that information dissemination can be formalized as follows:

[0065]

[0066] Related concepts should have similar prerequisite relationships with other concepts. For example, a binary tree is a prerequisite for a DAG graph, and an AVL tree is highly related to a binary tree; therefore, an AVL tree is likely also a prerequisite for a DAG graph. To leverage this observation for learning concept prerequisite relationships, a good solution is to select related concepts for the target concept and fuse information from these related concepts for target concept prerequisite relationship classification. Here, this embodiment implements... A single point of attention is used to stabilize the learning process and incorporate more information into the network.

[0067]

[0068]

[0069]

[0070] in It's a head The projection matrix is ​​then applied. Next, two projection matrices with... Linear transformations of activation functions are used to impart nonlinearity to the model.

[0071] In this embodiment, the loss function of the graph neural network is used to measure the difference between the model's predictions and the actual results, and to guide the model's optimization during training. This embodiment is based on the idea of ​​contrastive learning and uses a contrastive loss function (square_contrastive_loss) to guide the training of the graph neural network.

[0072] More formally, Yes (node ​​embedding, text embedding) pair, It is the output vector of the text encoder. This is the output of the graph encoder for the node. Then, the matrix is ​​given by the following formula:

[0073]

[0074] This is the cosine similarity matrix between text and nodes. Although it's a square matrix, it's not symmetric: rows are text, and columns are nodes. In this example, each element is multiplied by... ,in It is a hyperparameter. Let and These are graph-based non-negative similarity functions for text and nodes, respectively. Then, the graph-based similarity distributions for text and nodes are defined as follows:

[0075]

[0076] similar.

[0077] Fix a certain hyperparameter In this embodiment, the target distribution is defined as follows:

[0078]

[0079] Therefore, the loss function in this embodiment is:

[0080]

[0081] Represents cross-entropy. They represent the first Line number List. The combination of the regularized distance matrix and the identity matrix is ​​controlled to ensure that the target matrix retains similarity information without losing its initial structure. The target matrix is ​​determined through... The matrix resulting from the combination of the regularized distance matrix and the identity matrix. When the target matrix is ​​exactly equal to the identity matrix, the model only considers the features of each node itself, without considering the similarity information between nodes. In this case, the target matrix is ​​entirely determined by the regularized distance matrix. This means that the model relies entirely on the similarity information between nodes, ignoring the individual features of each node.

[0082] For Siamese networks, in this embodiment, after obtaining the final concept representation, a Siamese network is used to predict concepts. Is it a concept? The prerequisite is that the concept is fed into two feedforward networks with shared weights. The outputs are then concatenated for classification. The entire process can be represented as follows:

[0083] ,

[0084] ,

[0085] in, This represents the result after projection by the feedforward network; Indicates the activation function; Indicates the first Node embeddings corresponding to each concept; They represent the first The and the first One concept; Representing concept pairs middle yes The probability of the prerequisites; , , , These are the parameterized weight matrix and the bias vector, respectively. yes Type function; Indicates element-wise multiplication; Represents the connection of vectors

[0086] Finally, there is the cross-entropy loss function:

[0087]

[0088] in It is the training dataset. Indication concept Is it a concept? The prerequisite.

[0089] It should be noted that this embodiment combines a multimodal contrastive learning approach, using independent text encoders and graph encoders to allow the model to simultaneously learn the textual and graph structural information of concepts. `MatMul` represents matrix multiplication, used to calculate the attention score; `Scale` represents normalization; `Mask` is the mask; and `SoftMax` calculates the attention weights. In the self-attention mechanism, each input element (such as a word or node) is compared with all other elements in the sequence to measure its correlation with other elements. This captures global dependencies during feature extraction and information interaction, and assigns different weights based on similarity, amplifying important information while weakening irrelevant information. The calculation of self-attention typically involves a query (Q), a key (K), and a value (V).

[0090] It should be noted that in this embodiment, firstly, in the data preprocessing section, a text description is obtained for each concept, and an embedding vector is generated for them using a text encoder. The concept is then recorded at this stage. The text embedding is And use it as the initial embedding of the corresponding concept node. (The input to the embedding layer is...) and Then update the node embeddings in the GAC-GNN model. (Updated via gating mechanism, self-attention mechanism, and contrastive loss function) Finally, the model outputs the final node embedding. The input to the prediction layer is And concept pairs, the output is the prerequisite relation prediction value (0 or 1);

[0091] Gated Updates: In GGNN, node representation updates are achieved through a gating mechanism and graph adjacency relationships. For each node, the representations of its neighboring nodes are first passed over, and then fused and updated through GRU units. Specifically, GGNN employs the following update rules:

[0092] Initialization phase: The initial representation of each node is defined through the node's annotations or other prior information, represented as follows: ;

[0093] Propagation Phase: At each time step, the representation of a node is updated based on the state of its neighboring nodes and the weights of its edges. Gating mechanisms (such as GRU) are used to control the propagation and retention of information, thereby optimizing the information transmission process. At each time step... Node representation It will be updated using the representations of its adjacent nodes and the weights of the graph's edges. The specific update rules are as follows:

[0094]

[0095] in, Represents a node and the set of neighboring nodes, It is a node with neighboring nodes Edge weights between them and Represents a node and At time step According to GRU, the proportion of information flow is determined by control gates (update gate and reset gate).

[0096] Gating Mechanism: The gating mechanism determines the degree of update of the current node state. Specifically, GRU generates update gates based on the node state of the previous step and the current input information, controlling the fusion of old and new information. Update Gate Reset door Updated node representation , as follows:

[0097]

[0098]

[0099]

[0100]

[0101] in, express Activation function This represents the hyperbolic tangent activation function. , , and , , These are the parameters for learning.

[0102] The update via the fusion module (self-attention mechanism) is as shown in the patent:

[0103]

[0104]

[0105]

[0106] in It's a head The projection matrix below.

[0107] Updated via contrastive loss function: In recent years, contrastive learning has achieved breakthroughs in text embedding tasks. The proposed model is influenced by the ideas of contrastive learning and the classic contrastive learning framework SimCLR, aiming to pay attention to the original information of concept nodes when updating their embeddings. Specifically, for concept sets... Using node embedding sets and text embedding set Comparative learning. Sets Denotes a set of embedding pairs, where This represents the node embedding in the graph. The text embeddings corresponding to the nodes are represented. Based on this, a similarity matrix is ​​constructed. This matrix is ​​square, but not symmetrical; its rows correspond to the text embeddings of concepts, and its columns correspond to the graph node embeddings of concepts. The elements in the matrix... This is used to measure the similarity between text embeddings and graph node embeddings, and its calculation method is as follows:

[0108]

[0109] Each element uses a temperature coefficient. Scaling is applied. Loss function. The definition is as follows:

[0110]

[0111] Here, This represents the cross-entropy operation, used to measure the difference between the predicted distribution and the true distribution. and Represent matrices respectively The row and number Column. Parameters and These represent the target distributions for text embedding and node embedding, respectively. The parameters are... Controlling the regularization of the similarity matrix A trade-off between the target matrix and the identity matrix is ​​struck to ensure that the target matrix retains both similarity information and the original structure. When the similarity matrix is ​​large, Increased influence on the target label means the model will pay more attention to the similarity relationships between concepts. Conversely, when When the size is small, the model tends to use a standard identity matrix, where each sample is matched only with its corresponding category, without considering the similarity between concepts. Connect to the output layer of the graph encoder. and Finally, we obtain the updated set of node embedding vectors for the concept nodes. .

[0112] In the Siamese network prediction layer: once all concept node representations are obtained, the Siamese network is used to predict concepts. Is it a concept? The prerequisite. , The output (corresponding) , Connect these elements to predict relationships under given conditions. The entire process is as follows:

[0113]

[0114]

[0115] in, This represents the result after being projected by the feedforward network. , , , These are the parameterized weight matrix and the bias vector, respectively. yes The function type is Element-wise. It represents element-wise multiplication. This represents the concatenation of vectors. Finally, the training process is performed using the binary cross-entropy loss function:

[0116]

[0117] in, This represents the training dataset for that domain. It indicates the real prerequisite relationship between pairs of concepts.

[0118] It should be noted that, in this embodiment, as Figure 2As shown in the diagram, the first and third parts represent common methods for prerequisite relation learning. The first part involves data preprocessing, obtaining textual descriptions of each concept through the Wikipedia API and expanding the density of the prerequisite relation graph through transitivity. The third part is a Siamese network, a neural network structure composed of two or more identical sub-networks that share the same weights and parameters. Siamese networks were first applied to tasks such as signature verification and face recognition, and their main characteristic is learning by comparing the similarity between two inputs. In a Siamese network, two or more identical sub-networks simultaneously process different inputs and calculate the similarity or difference between the inputs using a similarity metric (such as cosine similarity, Euclidean distance, etc.). This structure helps the model learn the relationships between inputs without direct label information and is widely used in metric learning, matching tasks, and relational reasoning. Regarding the second part of the diagram, multimodal contrastive learning is first introduced into the prerequisite relation learning task. Through independent text encoders and graph encoders, the model simultaneously learns the textual information and graph structure information of the concept.

[0119] First, use the official Wikipedia API to extract the Wikipedia article description corresponding to the target concept, select the first 200 words to create its text description, and use a text encoder to generate the corresponding text embedding.

[0120] Next, a prerequisite relation graph is constructed using the transitivity of prerequisite relations. In order to capture multi-hop prerequisite relations, GAC-GNN selects gating and self-interaction mechanisms to enhance the representation of the embedded information of concepts.

[0121] Finally, these embeddings are fed into a Siamese network to predict the prerequisite relationships between concepts.

[0122] The training process of the model is as follows: The input parameters accepted are (a set of text embeddings). Node embedding set Comparison of loss functions (A prerequisite graph containing nodes and edges); gating mechanism updates the node embedding set. (Can be expressed by a formula); Concept fusion update node embedding set (Can be expressed by a formula); control the node embedding set through loss. Update; updated set of node embeddings Send it into the twin network.

[0123] The technical solution of the present invention will be verified through experiments below;

[0124] 1. Datasets. Statistical data for the three published datasets are shown in Table 1.

[0125] Experts have annotated this set of concepts and the prerequisite relationships between them, and released them along with the dataset. Statistics for the dataset are shown in Table 1.

[0126] Table 1: Statistics of the dataset

[0127]

[0128] 2. Evaluation methods: Widely adopted metrics are used to evaluate the performance of all models, including accuracy (ACC), precision, recall, F1 score, and AUC. Higher values ​​indicate better model performance.

[0129] Baseline methods include: RefD 2015: RefD is a simple link-based metric used to measure preconditional relationships between concepts; Clickstream 2019: Utilizing Wikipedia's clickstream data, it defines 8 features and uses machine learning methods to obtain preconditional relationships between Wikipedia concepts; Clickstream 2021: Building upon Clickstream 2019, it introduces related concepts and defines 32 features using machine learning methods to obtain preconditional relationships between Wikipedia concepts; M3 2019: It merges features generated by LSTM with handcrafted features; PREREQ 2019: PREREQ obtains latent representations of concepts through a pairwise latent Dirichlet assignment model and uses Siamese networks to identify prerequisite relations; MHAVGAE2022: Proposes an end-to-end graph network-based model called Multi-Head Attention Variational Graph Autoencoder, which learns prerequisite relations between concepts through a resource concept graph; ConLearn2022: A context-aware concept prerequisite relation learning method that leverages pre-trained language models BERT and GNN to enhance the model's performance on concept prerequisite relation classification tasks; HGAPNet2023: Proposes a graph neural network-based method that uses the attention of nodes on heterogeneous graphs to extract prerequisite concepts for a given concept.

[0130] 3. Implementation Details. To compare the performance of all models, all baseline methods used five-fold cross-validation to fully utilize the data and obtain stable performance estimates.

[0131] 4. Results Comparison and Analysis

[0132] All methods were tested, and the five-fold cross-validation results for all methods are reported, with the best values ​​marked in bold. The experimental results are shown in Table 2. PREREQ uses the five-fold cross-validation results from the original paper. PREREQ, HGAPNet, and MHAVGAE were run on datasets other than AL, and the clickstream method was run on datasets other than LectureBank.

[0133] Table 2: Comparison of Experimental Results

[0134]

[0135] As can be seen from Table 2, the performance of the proposed method consistently outperforms the baseline methods on all datasets. Several observations can be drawn from the results: (1) Clickflow achieved satisfactory results on all datasets because it uses many handcrafted features, but this requires excessive preprocessing. (2) MHAVGAE achieved the highest recall on both the MOOC and LectureBank datasets because it utilizes vertex latent attention representations and designs a gated fusion mechanism to transform latent attention into latent information of concepts, thereby identifying most prerequisite relationships. However, its pre-performance is low, resulting in a low F1 score. (3) The RefD method also has higher recall on the Data Mining and LectureBank datasets because its proposed link-based metric can indeed measure prerequisite relationships between concepts. However, its pre-performance is low, resulting in a low F1 score. (4) Among the existing baseline methods, the most competitive baselines are ConLearn and HGAPNet. Their performance on the MOOC, UC, and LectureBank datasets closely follows the proposed method because ConLearn uses GGNN to capture multi-hop prerequisite relationships, and HGAPNet uses an attention mechanism. However, ConLearn's conceptual representation update is unsupervised, which may lead to higher loss. HGAPNet uses heterogeneous graph neural networks and relies on attention mechanisms. If the graph is sparse, the attention mechanism may not be able to fully capture the relationships between different types of nodes and edges, thus degrading the model's performance.

[0136] 5. Ablation experiments; evaluate the role of these components in the final performance of the model. Specifically, the following questions will be addressed: (1) How do other graph neural network architectures perform in extracting concept premise relations on the same graph structure? (2) What is the impact of using a contrastive loss function in a graph neural network? (3) In the prediction network, use a Siamese network. How does this compare to simple node vector connections in the prediction layer? (4) How does the initial concept representation affect performance?

[0137] Ablation is created by removing them one by one. Specifically, the graph model is replaced with a regular GNN, the contrastive loss function is removed from the graph model, the final node embeddings are used for prediction through fully connected layers, and BERT is used to generate embeddings for concepts instead of large model embeddings.

[0138] Table 3: Ablation Experiment Comparison Table

[0139]

[0140] (1) To address the first problem, a Graph Convolutional Network (GCN) was chosen, which is one of the existing strong models. Furthermore, since GCN is designed for relation learning, it is well-suited for determining premise relations. Table 3 presents the results for all datasets. The results show that the model of this invention significantly outperforms GCN.

[0141] (2) In the second question, the role of the contrastive loss function in the model was explored. Table 3 clearly shows that the performance drops significantly after the contrastive loss function is removed.

[0142] (3) For the third problem, the Siamese network was removed from the prediction network, and the node embedding vectors of the two concepts were used for label prediction instead. The goal of this invention is to investigate the effect of combining concept representations through basic vector operations. The results again show that fully connected layers are significantly worse than Siamese networks.

[0143] (4) In the final experiment, the effectiveness of text representation based on the large model was investigated. Sentence BERT was used to create embeddings for concepts and documents. In the proposed model, a document is represented only as the average of the vectors of the concepts appearing in that document. The results show that sentence BERT embeddings are always worse than large model embeddings.

[0144] This invention extracts textual embedding representations of concepts from large-scale language models, captures multi-hop prerequisite relations of concepts using a gating mechanism, and fuses information from related concepts using a multi-head self-attention mechanism to learn prerequisite relations from sparse and unstructured knowledge concepts. No handcrafted features are used in the model, making it easy to implement in downstream applications. Experiments on three datasets demonstrate that this invention achieves state-of-the-art performance compared to existing methods.

[0145] This embodiment provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the above-described method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0146] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the above-described method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network.

[0147] like Figure 5 As shown, the computer device 120 may include: at least one processor 121, such as a central processing unit (CPU), at least one communication interface 123, memory 124, and at least one communication bus 122. The communication bus 122 is used to enable communication between these components. The communication interface 123 may include a display screen and a keyboard; optionally, the communication interface 123 may also include a standard wired interface or a wireless interface. The memory 124 may be high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 124 may also be at least one storage device located remotely from the aforementioned processor 121. The memory 124 stores application programs, and the processor 121 calls the program code stored in the memory 124 to execute any of the aforementioned method steps. The communication bus 122 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 122 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5The term 124 is represented by a single line, but this does not imply a single bus or a single type of bus. The memory 124 may include volatile memory, such as random-access memory (RAM); it may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may include combinations of the above types of memory. The processor 121 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. The processor 121 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Optionally, the memory 124 is also used to store program instructions. The processor 121 can call the program instructions to implement the concept prerequisite relation learning method based on gated self-attention contrastive graph neural network as described in this embodiment.

[0148] This embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for learning conceptual prerequisite relationships based on a gated self-attention contrastive graph neural network.

[0149] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for learning concept prerequisite relationships based on a gated self-attention contrastive graph neural network, characterized in that, Includes the following steps: S1: Based on the target concept and its description, generate a set of text embeddings using a text encoder, and construct a prerequisite relation graph using the transitivity of prerequisite relations; S2: Based on the aforementioned prerequisite relation graph, using the text embedding set as the initial node embedding set, the node embedding set is updated using a contrastive loss function and a graph neural network to obtain the representations of all concept nodes; S3: Based on the representations of all concept nodes, using the binary cross-entropy loss function and Siamese network, the prerequisite relationships between concepts are obtained, specifically including: S31: The Siamese network is trained using the binary cross-entropy loss function to obtain a trained Siamese network; S32: Based on the representations of all concept nodes, the preconditions between concepts are obtained using the trained Siamese network; The twin network is as follows: ， ， in, This represents the result after projection by the feedforward network; Indicates the activation function; Indicates the first Node embeddings corresponding to each concept; They represent the first The and the first One concept; Representing concept pairs middle yes The probability of the prerequisites; , , , These are the parameterized weight matrix and the bias vector, respectively. yes Type function; Indicates element-wise multiplication; Indicates the connection of vectors; The binary cross-entropy loss function is as follows: ， in, Represents the binary cross-entropy loss function; This represents the training dataset for this domain; It indicates the real prerequisite relationship between pairs of concepts.

2. The concept prerequisite relation learning method based on gated self-attention contrastive graph neural network according to claim 1, characterized in that, Step S2 specifically includes: S21: Train the graph neural network using the contrastive loss function to obtain a trained graph neural network; S22: Based on the prerequisite relation graph, using the text embedding set as the initial node embedding set, update the node embedding set using the trained graph neural network.

3. The concept prerequisite relation learning method based on gated self-attention contrastive graph neural network according to claim 1, characterized in that, The graph neural network is as follows: ， ， ， ， ， ， ， in, Indicates an update to the door; Indicates that the door is being reset; Indicates the candidate hidden state; Represents a node At time step The representation of the node is the final hidden state update, which is the updated node representation. and Representing nodes respectively and At time step The representation of; This represents the Sigmid activation function; , , and , , These are the parameters to be learned; , , , respectively, represent the linear transformation parameter matrices of the graph neighbor aggregation part, which are learnable parameter matrices; Represents a node The set of neighboring nodes; It is a node with neighboring nodes Edge weights between them; Represents the hyperbolic tangent activation function; Attention weights; The input feature matrix; , , It is the projection matrix under the head h, and the linear transformation of the ReLU activation function; The dimension of the Query / Key vector, used for scaling to avoid gradient explosion; This is the feature matrix output by the h-th head; It is a normalized exponential function; The final output feature matrix; The output is the result of concatenating all heads along the feature dimension; This is the feature matrix output by the m-th head; To output the linear transformation matrix.

4. The concept prerequisite relation learning method based on gated self-attention contrastive graph neural network according to claim 1, characterized in that, The contrastive loss function is as follows: ， ， in, To compare loss functions; This refers to the number of knowledge concepts, i.e., the number of knowledge concepts in the dataset. Represents cross-entropy operation; and Represent matrices respectively The row and number List; Representation matrix The row and number Column elements are used to measure the similarity between text embeddings and graph node embeddings; and These represent the target distributions of text embedding and node embedding, respectively. To control the regularization of the similarity matrix The tradeoff coefficient between the identity matrix and the set; Represents a set of embedding pairs. This represents the node embedding in the graph. The text embedding corresponding to the node; This is the temperature coefficient.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the concept prerequisite relation learning method based on gated self-attention contrastive graph neural network as described in any one of claims 1 to 4.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the concept prerequisite relation learning method based on gated self-attention contrastive graph neural network as described in any one of claims 1 to 4.

7. An application of the concept prerequisite relation learning method based on gated self-attention contrastive graph neural network as described in any one of claims 1 to 4, characterized in that, It is used in the development of learning systems.

Images