Neural network concept predetermination relation learning method based on gated self-attention contrast graph

Abstract

The invention discloses a neural network concept predetermination relation learning method based on a gated self-attention contrast graph, and relates to the technical field of natural language processing, and the method mainly comprises the steps: generating a text embedding set through a text encoder according to a target concept and the description thereof, utilizing the precedence relation transitivity to construct a precedence relation graph; taking the text embedding set as an initial node embedding set, and updating the node embedding set by utilizing a contrast loss function and a graph neural network to obtain all concept node representations; a binary cross entropy loss function and a twin network are used to obtain a prerequisite relationship between concepts. By implementing the neural network concept prerequisite relation learning method based on the gated self-attention contrast graph provided by the invention, the prerequisite condition relation of the concept can be learned from sparse and unstructured knowledge concepts.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, more particularly, to a concept prerequisite relationship learning method based on a gated self-attention contrastive graph neural network. BACKGROUND

[0002] The existing technologies mainly fall into two categories. One is the method based on manual features: the earliest research focuses on using manual features to detect the prerequisite relationship between concepts; for example: RefD proposes contextual, structural, and semantic features, and extracts the prerequisite relationship between concepts by analyzing the structure and semantics of the text; there is also a click flow, which uses the click flow data of Wikipedia to define features and uses machine learning methods to infer the prerequisite relationship between concepts; this method captures the association between concepts by analyzing user behavior (such as click flow); however, these methods based on manual features rely on a large amount of feature engineering, which requires expert knowledge to define and extract relevant features; this not only consumes time and effort, but also may not be able to fully capture the complex relationship between concepts, because the selection and extraction of features are often limited by subjective factors and domain knowledge; the other is the method based on deep learning: in order to avoid the cumbersome feature engineering process, more and more researchers have applied deep learning methods to concept prerequisite relationship learning in recent years; for example: MHAVGAE applies a variational graph autoencoder to learn the prerequisite relationship between concepts from courses; PREEQ, M3, Conlearn, and HGAPNet use different deep learning models (such as supervised learning methods, graph neural networks, etc.) to learn the prerequisite relationship between concepts, trying to capture the complex patterns between concepts.

[0003] However, the existing technologies have the following defects: one is the sparsity problem: the concept prerequisite relationship is sparse (the number of relationships is much smaller than the total number of concept pairs), the existing methods rely on a large amount of training data, the information propagation path is long in the sparse graph, the decay is serious, and the aggregation ability is limited; two is the lack of modeling of transitivity: it is difficult to effectively capture the multi-hop transitive relationship between concepts, for example, if the maximum likelihood is the prerequisite of the hidden Markov model, and the hidden Markov model is the prerequisite of the conditional random field, then the maximum likelihood should be the prerequisite of the conditional random field; the existing methods cannot simulate this multi-hop relationship; three is the limited application scenarios: most methods are designed for structured learning materials (such as textbooks), and it is difficult to apply to sparse and unstructured knowledge concepts. SUMMARY

[0004] The purpose of the present application is to provide a concept prerequisite relationship learning method based on a gated self-attention contrastive graph neural network, which can learn the prerequisite relationship between concepts from sparse and unstructured knowledge concepts.

[0005] The application provides a concept prerequisite relationship learning method based on a gated self-attention contrast graph neural network concept, comprising the following steps: S1: generating a text embedding set by using a text encoder according to a target concept and a description thereof, and constructing a prerequisite relationship graph by using prerequisite relationship transitivity; S2: updating a node embedding set by using a contrast loss function and a graph neural network according to the prerequisite relationship graph and taking the text embedding set as an initial node embedding set, to obtain all concept node representations; and S3: obtaining prerequisite relationships between concepts by using a binary cross-entropy loss function and a twin network according to the all concept node representations.

[0006] The application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the concept prerequisite relationship learning method based on the gated self-attention contrast graph neural network concept.

[0007] The application further provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the concept prerequisite relationship learning method based on the gated self-attention contrast graph neural network concept when executing the program.

[0008] The application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the concept prerequisite relationship learning method based on the gated self-attention contrast graph neural network concept.

[0009] The application further provides an application of the concept prerequisite relationship learning method based on the gated self-attention contrast graph neural network concept, which is applied to learning system development.

[0010] The concept prerequisite relationship learning method based on the gated self-attention contrast graph neural network concept has the following beneficial effects: The application combines contrast learning and a graph neural network, and proposes a concept prerequisite relationship learning method without relying on structured learning materials. This method not only expands the application scenarios, but also effectively guides the training of the graph neural network through a contrast loss function, helping the model to better learn the prerequisite relationships between concepts.

[0011] The application introduces a gating mechanism and a multi-head self-attention mechanism to enhance the ability of the model to update node representations on a sparse graph. The gating mechanism can capture multi-hop prerequisite relationships between concepts, and the multi-head self-attention mechanism can focus on multiple relationships and features in parallel, thereby more effectively capturing and integrating these similar relationships. This improvement solves the problem of long information propagation path and information decay when traditional graph neural networks process sparse graphs, and improves the learning and capturing ability of the model to concept relationships, which is therefore the protection point of the application.

[0012] The present application utilizes existing concept prerequisite conditions to construct a concept prerequisite relationship graph, and updates the representation of the concept through a gated graph neural network. This construction method enables the model to better capture the dependency relationship between concepts, especially in sparse and unstructured knowledge concepts. By constructing a concept prerequisite relationship graph and utilizing a gated graph neural network for information propagation, the present application realizes learning of prerequisite relationship from sparse and unstructured knowledge concepts. BRIEF DESCRIPTION OF DRAWINGS

[0013] The present application will be further described below in conjunction with the accompanying drawings and embodiments, in which: Figure 1 is a flowchart of the concept prerequisite relationship learning method based on the gated self-attention contrastive graph neural network provided by the present application; Figure 2 is a schematic diagram of the overall framework design of the concept prerequisite relationship learning method based on the gated self-attention contrastive graph neural network provided by the present application; Figure 3 is a schematic diagram of the original prerequisite relationship and the expanded prerequisite relationship provided by the present application; Figure 4 is a schematic diagram of the concept prerequisite condition graph structure and the connection matrix provided by the present application; Figure 5 is a structural block diagram of a computer device provided by the present application. DETAILED DESCRIPTION

[0014] In order to have a clearer understanding of the technical features, objectives and effects of the present application, the specific embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0015] Figure 1 shows a schematic diagram of the concept prerequisite relationship learning method based on the gated self-attention contrastive graph neural network of the present embodiment. In the present embodiment, the concept prerequisite relationship learning method based on the gated self-attention contrastive graph neural network includes the following steps: S1: generating a text embedding set using a text encoder according to a target concept and its description, and constructing a prerequisite relationship graph using prerequisite relationship transitivity; S2: updating a node embedding set using a contrastive loss function and a graph neural network according to the prerequisite relationship graph and taking the text embedding set as an initial node embedding set, to obtain all concept node representations; In an exemplary embodiment, step S2 specifically includes: S21: training the graph neural network using the contrastive loss function to obtain a trained graph neural network; S22: updating the node embedding set using the trained graph neural network according to the prerequisite relationship graph and taking the text embedding set as an initial node embedding set. In an exemplary embodiment, the graph neural network is as follows: , , , , , , , wherein, represents an update gate; represents a reset gate; represents a candidate hidden state; represents a node at time step , i.e., the final hidden state update, i.e., the updated node representation; and represent the node and at time step ; represents a Sigmid activation function; , , and , , are learned parameters, respectively; , , represent linear transformation parameter matrices of the graph neighbor aggregation part, are learnable parameter matrices, respectively; represents a set of node and neighbor nodes; is an edge weight between node and neighbor node ; represents a hyperbolic tangent activation function; is an attention weight; is an input feature matrix; , , is a projection matrix under head h, a linear transformation of a ReLU activation function; is a dimension of a Query / Key vector, used for scaling to avoid gradient explosion; is a feature matrix output by the hth head; is a normalized exponential function; is a final output feature matrix; The output is the result of concatenating all heads along the feature dimension; This is the feature matrix output by the m-th head; To output a linear transformation matrix (projecting the spliced ​​result back to the original or desired output dimension).

[0016] In one exemplary embodiment, the contrastive loss function is as follows: , , in, To compare loss functions; The number of knowledge concepts (i.e., the number of knowledge concepts in the dataset); Represents cross-entropy operation; and Represent matrices respectively The row and number List; Representation matrix The row and number Column elements are used to measure the similarity between text embeddings and graph node embeddings; and These represent the target distributions of text embedding and node embedding, respectively. To control the regularization of the similarity matrix The tradeoff coefficient between the identity matrix and the set; Represents a set of embedding pairs. This represents the node embedding in the graph. The text embedding corresponding to the node; This is the temperature coefficient.

[0017] S3: Based on the representation of all the concept nodes, the prerequisite relationships between concepts are obtained using the binary cross-entropy loss function and Siamese network; In one exemplary embodiment, step S3 specifically includes: S31: The Siamese network is trained using the binary cross-entropy loss function to obtain a trained Siamese network; S32: Based on the representations of all concept nodes, the pre-trained Siamese network is used to obtain the prerequisite relationships between concepts; In one exemplary embodiment, the twin network is as follows: , , in, This represents the result after projection by the feedforward network; Indicates the activation function; representing node embeddings of the th concept; representing the th and th concepts, respectively; representing the probability that the th concept is a prerequisite of the th concept; , , , are parameterized weight matrices and bias vectors, respectively; is a type function; represents element-wise multiplication; represents concatenation of vectors

[0018] In an exemplary embodiment, the binary cross-entropy loss function is as follows: , wherein, represents the binary cross-entropy loss function; represents the training data set of the domain; represents the true prerequisite relationship between the concept pairs.

[0019] In some embodiments, the above-mentioned gated self-attention contrastive graph neural network concept prerequisite relationship learning method can also be implemented in the following way.

[0020] In this embodiment, the gated self-attention contrastive graph neural network concept prerequisite relationship learning method is applied to the development of a learning system, which includes the following steps: Step 1: According to the target concepts and their descriptions, a set of text embeddings is generated using a text encoder, and a prerequisite relationship graph is constructed using the prerequisite relationship transitivity. Wherein, the target concepts and their descriptions are a set of geology concepts and their description texts, respectively. Step 2: According to the prerequisite relationship graph, the set of text embeddings is used as the initial set of node embeddings, and the set of node embeddings is updated using the contrastive loss function and the graph neural network, to obtain the representation of all concept nodes. Step 3: According to the representation of all concept nodes, the prerequisite relationship between concepts is obtained using the binary cross-entropy loss function and the twin network.

[0021] In some embodiments, the above-mentioned gated self-attention contrastive graph neural network concept prerequisite relationship learning method can also be implemented in the following way.

[0022] In this embodiment, the model framework of the overall framework design is as follows​Figure 2 The embodiment adopts two encoders, a text encoder and a graph encoder. First, the top 200 word descriptions of each concept are obtained from the API of Wikipedia, and the text embedding is obtained using the text encoder. Then, the concept prerequisite graph is established, and the concept representation is updated in the graph neural network. Since the prerequisite relationship between concepts is transitive, GAC-GNN as a graph encoder allows the aggregation of multi-step out node information by setting the step length; in addition, a concept fusion module is added in the graph neural network to utilize the similar prerequisite relationship of related concepts; the GAC-GNN model uses a contrastive loss function.

[0023] Finally, the concept representation is fed into a Siamese network for prerequisite relationship classification.

[0024] It should be noted that the concept set , the prerequisite relationship pair is a set of prerequisite relationships between concepts, where represents the concept is the prerequisite relationship of the concept . According to the transitivity of the prerequisite relationship, the prerequisite relationship pair is expanded, if is the prerequisite relationship of , is the prerequisite relationship of , then is also the prerequisite relationship of . The concept text is composed of the description corresponding to concepts in , and is presented as a text document. The document is the description of the concept in Wikipedia. This problem can be formally defined as: given a concept set , the goal is to learn a function , which can determine whether is the prerequisite concept of ; consider this a binary classification task. Specifically, if , is the prerequisite concept of , otherwise is not the prerequisite concept of . As shown in Figure 3 , the original prerequisite relationship and the expanded prerequisite relationship are shown in the schematic diagram; For text encoding, in the present embodiment, Hugging Face provides a rich large language model and its related tools and resources, which are widely used in various NLP tasks. In the existing text model ranking, the performance and size of these models are considered comprehensively, and finally mxbai-embed-large-v1 is selected as the text encoder to generate the embedding for the concept description.

[0025] For graph encoding, in the present embodiment, the graph neural network can capture the relationship between nodes, which makes the graph neural network suitable for this task. In the prerequisite relationship mining, first, the directed concept prerequisite graph is constructed by using the existing prerequisite relationship, and then the graph neural network is applied on the concept prerequisite graph to update the concept representation, and the concept prerequisite relationship is integrated into the model. The GGNN (Gated Graph Neural Network) adopts a gated recurrent unit to process the node information in the graph, and updates the state of the node through iteration to realize deeper information propagation, which can better capture complex dependency relationships and long-range information. Related concepts have similar prerequisite relationships, so the present embodiment designs a concept fusion module, which is called GAC-GNN. As shown in Figure 4 is a schematic diagram of the concept prerequisite graph structure and the connection matrix and . Specifically, let respectively represent the weighted connections of the incoming edges and outgoing edges in the concept prerequisite graph. For example, consider the concept prerequisite pair . Since the concept may repeatedly appear in the concept prerequisite pair, the present embodiment assigns a normalized weight to each edge, which is calculated as the number of occurrences of the edge divided by the out-degree of the starting node of the edge. For example has and two incoming edges, then the values of and are .

[0026] Next, the present embodiment will introduce how to update the representation of the concept by the gated graph neural network. First, each concept is embedded into a feature space, and the vector represents the dimensional real-valued feature vector of the concept . For each concept at step in the concept prerequisite graph, it is given by the connection matrix and , and the information propagation can be formalized as:

[0027] Related concepts should have similar prerequisite relationships with other concepts. For example, binary tree is a prerequisite of DAG graph, and AVL tree is highly related to binary tree, so AVL tree is likely to be a prerequisite of DAG graph as well. To exploit this observation for concept prerequisite learning, a good solution is to select related concepts for the target concept and fuse information from these related concepts for the target concept prerequisite classification. Here, the embodiment implements an independent attention head to stabilize the learning process and incorporate more information into the network.

[0028]

[0029]

[0030]

[0031] where is the projection matrix under the head . Then, two linear transformations with activation functions are applied to endow the model with nonlinearity.

[0032] For the contrastive learning module, in the embodiment, the loss function of the graph neural network is used to measure the gap between the model prediction result and the actual result, and guide the model to optimize in the training process. Based on the idea of contrastive learning, the embodiment uses a contrastive loss function (square_contrastive_loss) to guide the trained graph neural network.

[0033] More formally, is the pair of (node embedding, text embedding), is the text encoder output vector, is the graph encoder output of the node. Then, the matrix given by the following formula:

[0034] is the cosine similarity matrix between text and nodes. Although it is square, this matrix is not symmetric: rows are texts and columns are nodes. The embodiment multiplies each element by , where is a hyperparameter. Let and be the graph-based non-negative similarity functions for text and nodes, respectively. Then, the graph-based similarity distributions for text and nodes are defined as follows:

[0035] similarly.

[0036] Fixing a certain hyperparameter The target distribution is defined as follows in this embodiment:

[0037] Therefore, the loss function of this embodiment is:

[0038] represents the cross-entropy, represents the similarity between the th row and the th column. Controlling the combination of the regularized distance matrix and the identity matrix ensures that the target matrix both preserves the similarity information and does not lose the initial structure. The target matrix is the matrix obtained by controlling the combination of the regularized distance matrix and the identity matrix. When , the target matrix is exactly equal to the identity matrix. This means that the model only considers the node's own features, without considering the similarity information between nodes. When , the target matrix is completely determined by the regularized distance matrix. This means that the model completely relies on the similarity information between nodes, ignoring the node's own features.

[0039] For the Siamese network, in this embodiment, after obtaining the final concept representation, a Siamese network is used to predict whether the concept is a prerequisite for the concept . The concept representation is fed into two feedforward networks with shared weights. Then the outputs are connected for classification. The whole process can be represented as follows: , , where, represents the result projected by the feedforward network; represents the activation function; represents the node embedding corresponding to the th concept; represent the th and th concepts, respectively; represents the probability that the concept pair is a prerequisite for the concept ; , , , are parameterized weight matrices and bias vectors, respectively; is type function; representing element-wise multiplication; representing concatenation of vectors

[0040] Finally, there is a cross-entropy loss function:

[0041] where is the training data set, indicates the concept is a prerequisite for the concept .

[0042] It should be noted that the embodiment combines the method of multi-modal contrast learning, and through independent text encoder and graph encoder, the model learns the text information and graph structure information of the concept. MatMul represents matrix multiplication, which is used to calculate the attention score, Scale represents normalization, Mask is a mask, and SoftMax calculates the attention weight. In the self-attention mechanism, each input element (such as a word or a node) will calculate the similarity with all other elements in the sequence to measure its association with other elements, so as to capture global dependencies in the feature extraction and information interaction process, and different weights are assigned according to the similarity, so that important information can be amplified, and irrelevant information can be weakened. The calculation of self-attention usually involves query (Query, Q), key (Key, K) and value (Value, V); It should be noted that in the embodiment, first, in the data preprocessing part, the text description of each concept is obtained, and the text encoder is used to generate the embedding vector for them. The text embedding of the concept at this time is recorded as , and it is used as the initial embedding of the corresponding concept node . (The input of the embedding layer is and ) Then update the node embedding in the model GAC-GNN (update through the gating mechanism and the self-attention mechanism and the contrast loss function), and finally the model outputs the final node embedding . The input of the prediction layer is , and the concept pair, and the output is the prerequisite relationship prediction value (0 or 1); Gating mechanism update: in the GGNN, the update of the node representation is realized through the gating mechanism and the adjacency relationship of the graph. For each node, first, the representations of its adjacent nodes are passed over, and then they are fused and updated through the GRU unit. Specifically, the GGNN adopts the following update rule: Initialization phase: The initial representation of each node is defined by its annotation or other prior information, denoted as ; Propagation phase: At each time step, the representation of a node is updated based on the states of its neighboring nodes and the weights of the edges. The propagation and memory of information are controlled by a gating mechanism (such as GRU) to optimize the information transmission process. At each time step , the node representation is updated by the representations of its neighboring nodes and the weights of the edges of the graph. The specific update rule is as follows:

[0043] where represents the set of nodes and neighboring nodes, is the edge weight between node and neighboring node , and represent the representations of node and at time step , and GRU controls the proportion of information flow through the control gate (update gate and reset gate).

[0044] Gating mechanism: The gating mechanism determines the degree of updating the current node state. Specifically, GRU generates an update gate based on the node state of the previous step and the current input information to control the fusion of new and old information. The update gate , reset gate , updated node representation , is as follows:

[0045]

[0046]

[0047]

[0048] where represents the activation function, represents the hyperbolic tangent activation function, , , , , , , are learning parameters.

[0049] Updated by fusion module (self-attention mechanism) as shown in the patent:

[0050]

[0051]

[0052] where is the head projection matrix.

[0053] Contrastive loss function update: In recent years, contrastive learning has made breakthroughs in text embedding tasks. The proposed model is influenced by the idea of contrastive learning and the classic framework of contrastive learning SimCLR, hoping to focus on its own original information when updating the embedding of concept nodes. Specifically, for a concept set , the node embedding set and the text embedding set are used for contrastive learning. Set represents the set of embedding pairs, where represents the node embedding in the graph, represents the text embedding corresponding to the node. Based on this, a similarity matrix is constructed, which is a square matrix but not symmetric. The rows correspond to the text embedding of the concept, and the columns correspond to the graph node embedding of the concept. The elements in the matrix are used to measure the similarity between the text embedding and the graph node embedding, which are calculated as follows:

[0054] Each element is scaled by a temperature coefficient . The loss function is defined as follows:

[0055] Here, represents the cross-entropy operation, which is used to measure the difference between the predicted distribution and the true distribution. Among them, and represent the first row and the first column of the matrix . Parameters and represent the target distribution of text embedding and node embedding, respectively. Among them, the parameter controls the regularization of the similarity matrix a trade-off between the identity matrix and the similarity matrix, thus ensuring that the target matrix can both maintain the similarity information and preserve the original structure. When is large, the similarity matrix has a stronger influence on the target labels, which means that the model will pay more attention to the similarity between concepts. Conversely, when is small, the model tends to use the standard identity matrix, in which case each sample is only matched with its corresponding class without considering the similarity between concepts. By connecting the output layer of the graph encoder and , the final updated node embedding vector set of the concept node is obtained.

[0056] In the twin network prediction layer: once all the concept node representations are obtained, a twin network is used to predict whether the concept is a prerequisite for the concept . The outputs of , (corresponding to , ) are connected to predict the relationship under the prerequisite condition. The whole process is as follows:

[0057]

[0058] wherein, represents the result after being projected by the feedforward network. , , , are parameterized weight matrices and bias vectors, respectively. is a type function. represents element-wise multiplication, represents vector concatenation. Finally, the binary cross-entropy loss function is used for the training process:

[0059] wherein, represents the training data set of the domain, represents the true prerequisite relationship between the concept pairs.

[0060] It should be noted that in the present embodiment, as Figure 2As shown in the figure, the first and third parts are the common methods of learning the direction of prerequisite relationship. The first part belongs to data preprocessing, and the text description of each concept is obtained through the Wikipedia API. The density of the prerequisite graph is expanded through the prerequisite transitivity. The third part is a Siamese Network, which is a neural network structure composed of two or more identical sub-networks that share the same weights and parameters. Siamese Network is first applied to tasks such as signature verification and face recognition. Its main feature is to learn by comparing the similarity between two inputs. In Siamese Network, two or more identical sub-networks process different inputs at the same time, and the similarity or difference between the inputs is calculated through a similarity measure such as cosine similarity, Euclidean distance, etc. This structure can help the model learn the relationship between inputs without direct label information, and is widely used in metric learning, matching tasks and relationship reasoning. As for the second part of the figure, first, the idea of multi-modal contrastive learning is introduced in the prerequisite relationship learning task. Through independent text encoders and graph encoders, the model learns the text information and graph structure information of the concept at the same time.

[0061] First, the Wikipedia article description corresponding to the target concept is extracted using the official Wikipedia API, and the first 200 words are selected to create the text description and generate the corresponding text embedding using the text encoder.

[0062] Next, a prerequisite relationship graph is constructed using the prerequisite transitivity. In order to capture multi-hop prerequisite relationships, GAC-GNN selects a gating mechanism and self-interaction mechanism to enhance the representation of concept embedding information.

[0063] Finally, these embeddings are input into the Siamese Network to predict the prerequisite relationship between concepts.

[0064] Among them, the training process of the model is: the input parameters accepted (a set of text embeddings , a set of node embeddings , a contrastive loss loss function , a prerequisite relationship graph containing nodes and edges); the gating mechanism updates the set of node embeddings (the formula can be represented as); the concept fusion updates the set of node embeddings (the formula can be represented as); the node embedding set is updated through loss control; the updated node embedding set is input into the Siamese Network.

[0065] Next, the technical solutions of the present application will be verified through experiments; 1. Dataset. The statistical data of three published datasets is shown in Table 1.

[0066] Experts have annotated this set of concepts and the prerequisite relationships between them, and released them along with the dataset. Statistics for the dataset are shown in Table 1.

[0067] Table 1: Statistics of the dataset

[0068] 2. Evaluation methods: Widely adopted metrics are used to evaluate the performance of all models, including accuracy (ACC), precision, recall, F1 score, and AUC. Higher values ​​indicate better model performance.

[0069] Baseline methods include: RefD 2015: RefD is a simple link-based metric used to measure preconditional relationships between concepts; Clickstream 2019: Utilizing Wikipedia's clickstream data, it defines 8 features and uses machine learning methods to obtain preconditional relationships between Wikipedia concepts; Clickstream 2021: Building upon Clickstream 2019, it introduces related concepts and defines 32 features using machine learning methods to obtain preconditional relationships between Wikipedia concepts; M3 2019: It merges features generated by LSTM with handcrafted features; PREREQ 2019: PREREQ obtains latent representations of concepts through a pairwise latent Dirichlet assignment model and uses Siamese networks to identify prerequisite relations; MHAVGAE2022: Proposes an end-to-end graph network-based model called Multi-Head Attention Variational Graph Autoencoder, which learns prerequisite relations between concepts through a resource concept graph; ConLearn2022: A context-aware concept prerequisite relation learning method that leverages pre-trained language models BERT and GNN to enhance the model's performance on concept prerequisite relation classification tasks; HGAPNet2023: Proposes a graph neural network-based method that uses the attention of nodes on heterogeneous graphs to extract prerequisite concepts for a given concept.

[0070] 3. Implementation Details. To compare the performance of all models, all baseline methods used five-fold cross-validation to fully utilize the data and obtain stable performance estimates.

[0071] 4. Results Comparison and Analysis All methods were tested, and the five-fold cross-validation results for all methods are reported, with the best values ​​marked in bold. The experimental results are shown in Table 2. PREREQ uses the five-fold cross-validation results from the original paper. PREREQ, HGAPNet, and MHAVGAE were run on datasets other than AL, and the clickstream method was run on datasets other than LectureBank.

[0072] Table 2: Comparison table of experimental results

[0073] From Table 2, it can be seen that the performance of the method of the present application is always better than the baseline method on all data sets. From the results, the following observations can be made: (1) Clickstream achieved satisfactory results on all data sets, as it used many hand-crafted features, but it required too much preprocessing work. (2) MHAVGAE achieved the highest recall on both the MOOC and LectureBank data sets, as it utilized vertex latent attention representation and designed a gating fusion mechanism to transform latent attention into latent information of concepts, thereby identifying most prerequisite relationships, however, its pre performance was lower, resulting in a lower F1 score. (3) The recall of the RefD method on the Data Mining and LectureBank data sets was also higher than that of the method, as the link-based measure proposed by it can indeed measure prerequisite relationships between concepts. However, its pre performance was lower, resulting in a lower F1 score. (4) Among the existing baseline methods, the most competitive baselines are ConLearn and HGAPNet. They performed second best on the MOOC, UC, and LectureBank data sets, as ConLearn uses GGNN to capture multi-hop prerequisite relationships, and HGAPNet uses an attention mechanism. However, the concept representation update of ConLearn is unsupervised, which can lead to a higher loss. HGAPNet uses a heterogeneous graph neural network and relies on an attention mechanism, which can not fully capture the relationships between different types of nodes and edges if the graph is sparse, thereby reducing the performance of the model.

[0074] 5. Ablation experiments; evaluate the role of these components in the final performance of the model. Specifically, the following questions will be addressed: (1) How do other graph neural network architectures perform in prerequisite relationship extraction on the same graph structure? (2) What is the impact of using a contrastive loss function in the graph neural network? (3) In the prediction network, use a twin network. How does this compare to a simple node vector connection in the prediction layer? (4) How does the initial concept representation affect performance? Ablation is created by removing them one by one. Specifically, replace the graph model with a normal GNN, remove the contrastive loss function in the graph model, use the final node embedding to predict through a fully connected layer, and use bert to generate embedding for concepts to replace the large model embedding.

[0075] Table 3: Comparison table of ablation experiments

[0076] (1) To solve the first problem, a graph convolutional network (GCN) is chosen, which is one of the existing strong models. In addition, since GCN is designed for relation learning, it makes it suitable for determining premise relations. Table 3 gives the results for all datasets. The results show that the model of the present application is indeed significantly better than GCN.

[0077] (2) In the second problem, the role of the contrastive loss function in the model is explored. Table 3 clearly shows that the performance decreases significantly after removing the contrastive loss function.

[0078] (3) For the third problem, the siamese network is removed from the prediction network, and by connecting the node embedding vectors of two concepts, they are used for label prediction. The goal of the present application is to study the effect of combining concept representations through basic vector operations. The results again show that the fully connected layer is significantly worse than the siamese network.

[0079] (4) In the last experiment, the effectiveness of text representations based on large models is studied. Sentence BERT is used to create embeddings for concepts and documents. In the proposed model, a document is only represented as the average of the vectors of the concepts that appear in that document. The results show that the sentence BERT embeddings are always worse than the large model embeddings.

[0080] The present application obtains text embedding representations of concepts from large language models, captures multi-hop prerequisite relationships of concepts using a gating mechanism, and fuses information of related concepts using a multi-head self-attention mechanism. No handcrafted features are used in the model, which makes the model easy to implement in downstream applications. Experiments on three datasets show that the present application has the best performance compared to existing methods.

[0081] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the concept prerequisite relationship learning method based on the gated self-attention contrastive graph neural network described above. The storage medium can be a disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0082] The embodiment provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the concept prerequisite relationship learning method of the gated self-attention contrast graph neural network based on the gated self-attention when executing the program.

[0083] As shown in Figure 5 The computer device 120 can comprise at least one processor 121, such as a central processing unit (CPU), at least one communication interface 123, a memory 124 and at least one communication bus 122. The communication bus 122 is used to realize the connection and communication among the components. The communication interface 123 can comprise a display, a keyboard, and the optional communication interface 123 can further comprise a standard wired interface and a wireless interface. The memory 124 can be a high-speed random access memory (RAM), and can also be a non-volatile memory, such as at least one disk memory. The memory 124 can also be at least one storage device located away from the aforementioned processor 121. The memory 124 stores an application program, and the processor 121 calls the program code stored in the memory 124 to execute any of the above method steps. The communication bus 122 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 122 can be divided into an address bus, a data bus and a control bus. For the sake of representation, Figure 5The bus 123 is used to connect the above-mentioned elements in the system 120, and only one bus is represented, but it does not mean that there is only one bus or only one type of bus. Among them, the memory 124 can include volatile memory such as random-access memory (RAM); the memory can also include non-volatile memory such as flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 124 can also include a combination of the above-mentioned types of memory. Among them, the processor 121 can be a central processing unit (CPU), a network processor (NP) or a combination of CPU and NP. The processor 121 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. Alternatively, the memory 124 is also used to store program instructions. The processor 121 can call the program instructions to implement the concept prerequisite relationship learning method based on the gated self-attention contrastive graph neural network as described in the embodiment.

[0084] The embodiment provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the concept prerequisite relationship learning method based on the gated self-attention contrastive graph neural network as described above.

[0085] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these all belong to the protection of the present application.

Claims

1. A method for learning concept prerequisite relations based on gated self-attention contrastive graph neural networks, characterized in that, The method comprises the following steps: S1: generating a set of text embeddings by using a text encoder according to a target concept and a description thereof, and constructing a prerequisite relationship graph by using prerequisite relationship transitivity; S2: updating a set of node embeddings by using a contrastive loss function and a graph neural network according to the set of text embeddings as an initial set of node embeddings and the prerequisite relationship graph, to obtain all concept node representations; S3: obtaining prerequisite relationships between concepts by using a binary cross-entropy loss function and a twin network according to the all concept node representations.

2. The gated self-attention based contrast map neural network conceptual prerequisite relationship learning method according to claim 1, characterized in that, Step S2 specifically comprises: S21: training the graph neural network by using the contrastive loss function to obtain a trained graph neural network; S22: updating the set of node embeddings by using the trained graph neural network according to the set of text embeddings as the initial set of node embeddings and the prerequisite relationship graph.

3. The gated self-attention based contrast map neural network conceptual prerequisite relationship learning method according to claim 1, characterized in that, The graph neural network is as shown in the following formula: ， ， ， ， ， ， ， wherein, denotes an update gate; denotes a reset gate; denotes a candidate hidden state; denotes a node at time step , i.e., a final hidden state update, i.e., an updated node representation; and denote a node and at time step ; denotes a Sigmid activation function; , , and , , are learned parameters, respectively; , , denote linear transformation parameter matrices of the graph neighbor aggregation part, are learnable parameter matrices, respectively; denotes a node and a set of neighbor nodes; is an edge weight between a node and a neighbor node ; denotes a hyperbolic tangent activation function; is an attention weight; is an input feature matrix; , , is a projection matrix under a head h, a linear transformation of a ReLU activation function; is a dimension of a Query / Key vector, used for scaling to avoid gradient explosion; is a feature matrix output by an h-th head; is a normalized exponential function; is a final output feature matrix; is an output of splicing all heads along the feature dimension; is a feature matrix output by an m-th head; is an output linear transformation matrix.

4. The gated self-attention based contrast map neural network concept precedence relationship learning method according to claim 1, characterized in that, The contrastive loss function is as shown in the following formula: ， ， in, To compare loss functions; This refers to the number of knowledge concepts, i.e., the number of knowledge concepts in the dataset. Represents cross-entropy operation; and Represent matrices respectively The row and number List; Representation matrix The row and number Column elements are used to measure the similarity between text embeddings and graph node embeddings; and These represent the target distributions of text embedding and node embedding, respectively. To control the regularization of the similarity matrix The tradeoff coefficient between the identity matrix and the set; Represents a set of embedding pairs. This represents the node embedding in the graph. The text embedding corresponding to the node; This is the temperature coefficient.

5. The gated self-attention based contrast map neural network concept precedence relationship learning method according to claim 1, characterized in that, Step S3 specifically comprises: S31: training the twin network by using the binary cross-entropy loss function to obtain a trained twin network; S32: obtaining prerequisite relationships between concepts by using the trained twin network according to the all concept node representations.

6. The gated self-attention based contrast map neural network concept precedence relationship learning method according to claim 1, characterized in that, The twin network is as shown in the following formula: ， ， where, denotes the result projected by the feedforward network; denotes an activation function; denotes the node embedding corresponding to the th concept; denote the th and th concepts, respectively; denotes the probability that a concept is in is a prerequisite for ; , , , are parameterized weight matrices and bias vectors, respectively; is a type function; denotes element-wise multiplication; denotes concatenation of vectors.

7. The gated self-attention based contrast map neural network concept precedence relationship learning method according to claim 1, characterized in that, The binary cross-entropy loss function is as shown in the following formula: ， wherein, denotes a binary cross-entropy loss function; denotes a training dataset for the domain; denotes the true prerequisite relations between the concept pairs.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the prerequisite relationship learning method based on the gated self-attention contrastive graph neural network concept according to any one of claims 1-7.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the prerequisite relationship learning method based on the gated self-attention contrastive graph neural network concept according to any one of claims 1-7.

10. The application of the concept of learning the prerequisite relationship based on the gated self-attention contrastive graph neural network according to any one of claims 1-7, characterized in that, The application is applied to learning system development.

Images