A curriculum learning based self-supervised pre-training system and method

Abstract

The application provides a curriculum learning-based self-supervised pre-training system and method, relates to the technical field of graph structure data, and comprises the following steps: inputting sample graph structure data into a structured perception capable mask autoencoder in a training process through a first masker, obtaining vectorized representation of each node in the sample graph structure data, and determining an edge set constituting a curriculum based on a first residual error; selecting edges to be masked based on the edge set constituting the curriculum and the sample graph structure data through a second masker, obtaining an edge set selected based on a first difficulty score; reconstructing a reconstruction graph structure data based on a perturbed graph as input through the structured perception capable mask autoencoder; calculating a loss value based on the sample graph structure data and the reconstruction graph structure data, and continuing training based on the loss value until a training end condition is met, and obtaining a trained structured perception capable mask autoencoder. The system can be trained without labels.

Description

Technical Field

[0001] This invention relates to the field of graph structure data technology, and in particular to a self-supervised pre-training system and method based on course learning. Background Technology

[0002] Graph-structured data is ubiquitous in networks, such as social networks, reference networks, and e-commerce networks. Currently, graph neural networks learn from graph-structured data through supervised or semi-supervised learning with task-specific labels. However, in practice, acquiring or collecting a large number of task-specific labels is a challenging task. Therefore, how to train an effective model with limited or even no labels has become an urgent problem to be solved.

[0003] Based on this, the present invention proposes a label-free self-supervised pre-training system to enable training on graph structure data without labels. Summary of the Invention

[0004] This invention provides a self-supervised pre-training system and method based on course learning, so as to achieve label-free training of graph structure data.

[0005] In a first aspect, this invention proposes a self-supervised pre-training system based on course learning, the system comprising:

[0006] The first masker inputs the sample graph structure data into a structure-aware masking autoencoder during the training process to obtain a vectorized representation of each node in the sample graph structure data, calculates the first residual error, and selects the edges to be masked from the sample graph structure data based on the first difficulty score represented by the first residual error to determine the set of edges that constitute the course.

[0007] The second masker, based on the set of edges constituting the course and the sample graph structure data, selects the edges to be masked, thus obtaining the set of edges selected based on the first difficulty score.

[0008] A structure-aware mask autoencoder performs masking based on the edge set selected by the first difficulty score to obtain a perturbation map, and uses the perturbation map as input to reconstruct the reconstructed map structure data.

[0009] Based on the sample graph structure data and the reconstructed graph structure data, a loss value is calculated, and training continues based on the loss value until the training termination condition is met, resulting in a trained mask autoencoder with structure awareness.

[0010] Optionally, continuing training based on the loss value includes:

[0011] The first mask and the second mask input the sample graph structure data into the structure-aware mask autoencoder obtained from the previous training, and obtain a new vectorized representation of each node in the sample graph structure data.

[0012] Based on the new vectorized representation of each node in the sample graph structure data, a second residual error is obtained. Based on the second difficulty score represented by the second residual error, a set of edges selected based on the second difficulty score is obtained, wherein the second difficulty score is greater than the first difficulty score.

[0013] A structure-aware mask autoencoder uses a set of edges selected based on the second difficulty score to mask and obtain a new perturbation map. The new perturbation map is then used as input to reconstruct the new reconstructed map structure data.

[0014] Based on the sample graph structure data and the new reconstructed graph structure data, a new loss value is calculated, and training continues based on the new loss value.

[0015] Optionally, the calculation yields a first residual error. Based on the first difficulty score represented by the first residual error, edges to be masked are selected from the sample graph structure data to determine the set of edges constituting the course, including:

[0016] Based on the vectorized representation of each node, the similarity between the vectorized representations of every two nodes is calculated to obtain multiple similarity scores.

[0017] Based on multiple similarities, the first residual error of the edge connecting every two nodes is calculated, and the first residual error is used to characterize the first difficulty score for reconstructing the sample graph structure data;

[0018] Based on the first residual error and the sample graph structure data, a residual graph is obtained;

[0019] The first difficulty scores are sorted, and the edges to be masked are selected from the residual graph to determine the set of edges that constitute the course.

[0020] Optionally, the step of selecting edges to be masked based on the edge set constituting the course and the sample graph structure data to obtain the selected edge set based on the first difficulty score includes:

[0021] Based on the sample graph structure data, the edges to be masked in the sample graph structure data are randomly selected as the first edge set. The randomly selected edges to be masked do not include the edges in the edge set that constitutes the course.

[0022] The selected edge set based on the first edge set and the edge set constituting the course is determined.

[0023] Optionally, determining the selected edge set based on the first difficulty score according to the first edge set and the edge set constituting the course further includes:

[0024] The upper bound of the mask ratio and the random selection ratio are determined by the segmentation ratio. The upper bound of the mask ratio is used to determine the number of edges selected from the set of edges constituting the course, and the random selection ratio is used to determine the number of edges selected from the sample image structure data.

[0025] According to the upper bound of the masking ratio, select the edges to be masked from the set of edges constituting the course as the second set of edges;

[0026] Based on the sample graph structure data and the random selection ratio, the edges to be masked in the sample graph structure data are randomly selected as the third edge set, and the randomly selected edges to be masked do not include the edges in the second edge set.

[0027] Based on the edges in the second edge set and the edges in the third edge set, the edge set that constitutes the selection based on the first difficulty score is determined.

[0028] Optionally, the structure-aware mask autoencoder includes a GNN encoder and a GNN decoder. The perturbation map is obtained by masking the edge set selected based on the first difficulty score, and the perturbation map is used as input for reconstruction to obtain reconstructed graph structure data, including:

[0029] The nodes in the perturbation graph are encoded by the GNN encoder to obtain the node representation for each layer.

[0030] The GNN decoder decodes the node representation of each layer and reconstructs it based on the inherent similarity between nodes to obtain the reconstructed graph structure data.

[0031] Optionally, the GNN decoder includes: a cross-correlation decoder and a multilayer perceptron; the GNN decoder decodes the node representation of each layer and reconstructs the data based on the inherent similarity between nodes to obtain the reconstructed graph structure data, including:

[0032] Based on the inherent similarity between any two nodes, the edge embedding of any two nodes is calculated by the cross-correlation decoder;

[0033] A multilayer perceptron is used to predict the existence probability of each edge embedding.

[0034] The reconstruction is performed based on the existence probability to obtain the reconstructed graph structure data.

[0035] Optionally, the loss function for calculating the loss value includes a first loss function, a second loss function, and a third loss function;

[0036] The loss value between the vectorized representations of any two nodes is calculated using the first loss function;

[0037] The edge set constituting the course is constructed by selecting edges with smaller residual errors in the sample image structure data using a second loss function.

[0038] The number of edges in the selected edge set is optimized using a third loss function.

[0039] Optionally, the formula for calculating the loss function is as follows:

[0040]

[0041] The formula for calculating the first loss function is as follows: H represents the perturbation graph obtained by masking the edge set selected based on the first difficulty score, H represents all node representations of the sample graph structure data, A represents the adjacency matrix of the sample graph structure data, ENC(·) represents the GNN encoder, and DEC(·) represents the GNN decoder.

[0042] The formula for calculating the second loss function is βΣ. i，j S i，j R ij β represents a balance hyperparameter, S ij This represents the selected set of edges. A represents the residual error of the edge. ij This represents the sample graph structure data. Represents the reconstructed graph structure data, where ||·|| represents the l2 norm;

[0043] The third loss function is calculated as f(S; λ), where f(S; λ) = λ||SA||, λ represents the regularization term, A represents the adjacency matrix of the sample graph structure data, and S represents the mask matrix used to indicate the selection of the edge set.

[0044] In a second aspect of this invention, a self-supervised pre-training method based on course learning is proposed, the method comprising:

[0045] The sample graph structure data is input into the structure-aware mask autoencoder during the training process through the first mask, and the vectorized representation of each node in the sample graph structure data is obtained. The first residual error is calculated. Based on the first difficulty score represented by the first residual error, the edges to be masked are selected from the sample graph structure data to determine the set of edges that constitute the course.

[0046] The second masker selects the edges to be masked based on the set of edges constituting the course and the sample graph structure data, thereby obtaining the set of edges selected based on the first difficulty score.

[0047] A perturbation map is obtained by masking the selected edge set based on the first difficulty score using a structure-aware mask autoencoder. The perturbation map is then used as input to reconstruct the reconstructed graph structure data.

[0048] Based on the sample graph structure data and the reconstructed graph structure data, a loss value is calculated, and training continues based on the loss value until the training termination condition is met, resulting in a trained mask autoencoder with structure awareness.

[0049] Optionally, the method further includes:

[0050] The first mask and the second mask input the sample graph structure data into the structure-aware mask autoencoder obtained from the previous training, and obtain a new vectorized representation of each node in the sample graph structure data.

[0051] Based on the new vectorized representation of each node in the sample graph structure data, a second residual error is obtained. Based on the second difficulty score represented by the second residual error, a set of edges selected based on the second difficulty score is obtained, wherein the second difficulty score is greater than the first difficulty score.

[0052] A structure-aware mask autoencoder uses a set of edges selected based on the second difficulty score to mask and obtain a new perturbation map. The new perturbation map is then used as input to reconstruct the new reconstructed map structure data.

[0053] Based on the sample graph structure data and the new reconstructed graph structure data, a new loss value is calculated, and training continues based on the new loss value.

[0054] Optionally, the calculation yields a first residual error. Based on the first difficulty score represented by the first residual error, edges to be masked are selected from the sample graph structure data to determine the set of edges constituting the course, including:

[0055] Based on the vectorized representation of each node, the similarity between the vectorized representations of every two nodes is calculated to obtain multiple similarity scores.

[0056] Based on multiple similarities, the first residual error of the edge connecting every two nodes is calculated, and the first residual error is used to characterize the first difficulty score for reconstructing the sample graph structure data;

[0057] Based on the first residual error and the sample graph structure data, a residual graph is obtained;

[0058] The first difficulty scores are sorted, and the edges to be masked are selected from the residual graph to determine the set of edges that constitute the course.

[0059] Optionally, the step of selecting edges to be masked based on the edge set constituting the course and the sample graph structure data to obtain the selected edge set based on the first difficulty score includes:

[0060] Based on the sample graph structure data, the edges to be masked in the sample graph structure data are randomly selected as the first edge set. The randomly selected edges to be masked do not include the edges in the edge set that constitutes the course.

[0061] The selected edge set based on the first edge set and the edge set constituting the course is determined.

[0062] Optionally, determining the selected edge set based on the first difficulty score according to the first edge set and the edge set constituting the course further includes:

[0063] The upper bound of the mask ratio and the random selection ratio are determined by the segmentation ratio. The upper bound of the mask ratio is used to determine the number of edges selected from the set of edges constituting the course, and the random selection ratio is used to determine the number of edges selected from the sample image structure data.

[0064] According to the upper bound of the masking ratio, select the edges to be masked from the set of edges constituting the course as the second set of edges;

[0065] Based on the sample graph structure data and the random selection ratio, the edges to be masked in the sample graph structure data are randomly selected as the third edge set, and the randomly selected edges to be masked do not include the edges in the second edge set.

[0066] Based on the edges in the second edge set and the edges in the third edge set, the edge set that constitutes the selection based on the first difficulty score is determined.

[0067] Optionally, the structure-aware mask autoencoder includes a GNN encoder and a GNN decoder. The perturbation map is obtained by masking the edge set selected based on the first difficulty score, and the perturbation map is used as input for reconstruction to obtain reconstructed graph structure data, including:

[0068] The nodes in the perturbation graph are encoded by the GNN encoder to obtain the node representation for each layer.

[0069] The GNN decoder decodes the node representation of each layer and reconstructs it based on the inherent similarity between nodes to obtain the reconstructed graph structure data.

[0070] Optionally, the GNN decoder includes: a cross-correlation decoder and a multilayer perceptron; the GNN decoder decodes the node representation of each layer and reconstructs the data based on the inherent similarity between nodes to obtain the reconstructed graph structure data, including:

[0071] Based on the inherent similarity between any two nodes, the edge embedding of any two nodes is calculated by the cross-correlation decoder;

[0072] A multilayer perceptron is used to predict the existence probability of each edge embedding.

[0073] The reconstruction is performed based on the existence probability to obtain the reconstructed graph structure data.

[0074] Optionally, the loss function for calculating the loss value includes a first loss function, a second loss function, and a third loss function;

[0075] The loss value between the vectorized representations of any two nodes is calculated using the first loss function;

[0076] The edge set constituting the course is constructed by selecting edges with smaller residual errors in the sample image structure data using a second loss function.

[0077] The number of edges in the selected edge set is optimized using a third loss function.

[0078] Optionally, the formula for calculating the loss function is as follows:

[0079]

[0080] The formula for calculating the first loss function is as follows: H represents the perturbation graph obtained by masking the edge set selected based on the first difficulty score, H represents all node representations of the sample graph structure data, A represents the adjacency matrix of the sample graph structure data, ENC(·) represents the GNN encoder, and DEC(·) represents the GNN decoder.

[0081] The formula for calculating the second loss function is β∑ i，j S ij R ij β represents a balance hyperparameter, S ij This represents the selected set of edges. A represents the residual error of the edge. ij This represents the sample graph structure data. Represents the reconstructed graph structure data, where ||·|| represents the l2 norm;

[0082] The third loss function is calculated as f(S; λ), where f(S; λ) = λ||SA||, λ represents the regularization term, A represents the adjacency matrix of the sample graph structure data, and S represents the mask matrix used to indicate the selection of the edge set.

[0083] This invention offers the following advantages: It proposes a self-supervised pre-training system and method based on curriculum learning. A first masker inputs sample graph structure data into a structure-aware masking autoencoder during training to obtain a vectorized representation of each node in the sample graph structure data. A first residual error is calculated. Based on a first difficulty score represented by the first residual error, edges to be masked are selected from the sample graph structure data to determine the edge set constituting the curriculum. A second masker selects edges to be masked based on the edge set constituting the curriculum and the sample graph structure data, obtaining a selected edge set based on the first difficulty score. The structure-aware masking autoencoder performs masking based on the selected edge set with the first difficulty score to obtain a perturbation graph. The perturbation graph is used as input for reconstruction to obtain reconstructed graph structure data. Based on the sample graph structure data and the reconstructed graph structure data, a loss value is calculated, and training continues based on the loss value until the training termination condition is met, resulting in a trained structure-aware masking autoencoder. The self-supervised training method proposed in this invention does not require training labels. Instead, it integrates training data samples for reconstruction tasks in a meaningful order from easy to difficult, learns information-based graph neural network representations, and enables the pre-trained system to learn step by step from easy to difficult. It selects appropriate data samples from previous tasks whose difficulty matches the training state of the structure-aware mask autoencoder, achieves more powerful node representations, and exhibits better performance in downstream tasks. Attached Figure Description

[0084] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is an architecture diagram of a self-supervised pre-training system based on course learning provided in an embodiment of the present invention;

[0086] Figure 2 This is a schematic diagram of the framework of a self-supervised pre-training system based on course learning provided in an embodiment of the present invention;

[0087] Figure 3 This is a flowchart illustrating the steps of a self-supervised pre-training method based on course learning, provided in an embodiment of the present invention. Detailed Implementation

[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] Graph-structured data is ubiquitous in networks, such as social networks, reference networks, and e-commerce networks. Currently, Graph Neural Networks (GNNs) learn from graph-structured data through supervised or semi-supervised learning with task-specific labels. However, in practice, acquiring or collecting a large number of task-specific labels is a challenging task. Therefore, how to train an effective model with limited or even no labels has become an urgent problem to be solved.

[0090] Therefore, this invention proposes a label-free self-supervised pre-training system to train an effective model. Simultaneously, this invention considers starting with easier data samples and gradually transitioning to more difficult ones to propose a meaningful order of training data samples, allowing the training system to learn more informative graph neural network representations. Furthermore, this invention also considers the training state of the training system and proposes a feasible scheduling strategy to gradually utilize the training data samples for reconstruction. The following describes a customized structure-aware edge reconstruction task proposed in this invention. Through a self-supervised method, the difficulty of reconstructing training data samples is adjusted. Based on an adaptive learning strategy, edge reconstruction is strengthened during training, thereby restoring missing edges in the sample graph structure data. The specific implementation technical solution is described below.

[0091] In this embodiment of the invention, curriculum learning (CL) refers to a training strategy that mimics the human learning process, advocating that the model start learning from easy samples and gradually progress to more complex samples and knowledge. Self-supervised learning (SSL) refers to a commonly used type of supervised training method in computer vision and natural language processing. In this method, training is conducted by designing information representations from previous tasks without relying on labels, and mainly includes two types: contrastive methods and generative methods.

[0092] In a first aspect of the present invention, a self-supervised pre-training system based on course learning is proposed, see [reference]. Figure 1 , Figure 1 This is an architecture diagram of a self-supervised pre-training system based on course learning proposed in an embodiment of the present invention. The system includes:

[0093] The first masker inputs the sample graph structure data into a structure-aware masking autoencoder during the training process to obtain a vectorized representation of each node in the sample graph structure data, calculates the first residual error, and selects the edges to be masked from the sample graph structure data based on the first difficulty score represented by the first residual error to determine the set of edges that constitute the course.

[0094] The second masker, based on the set of edges constituting the course and the sample graph structure data, selects the edges to be masked, thus obtaining the set of edges selected based on the first difficulty score.

[0095] A structure-aware mask autoencoder performs masking based on the edge set selected by the first difficulty score to obtain a perturbation map, and uses the perturbation map as input to reconstruct the reconstructed map structure data.

[0096] Based on the sample graph structure data and the reconstructed graph structure data, a loss value is calculated, and training continues based on the loss value until the training termination condition is met, resulting in a trained mask autoencoder with structure awareness.

[0097] To enable those skilled in the art to better understand the self-supervised pre-training system based on course learning provided by this invention, Figure 2 This diagram illustrates a framework schematic of a self-supervised pre-training system based on course learning, as provided in an embodiment of the present invention. The following section will combine... Figure 2 This plan will be described in detail below:

[0098] Specifically, the self-supervised pre-training system based on curriculum learning includes a first mask, a second mask, and a structure-aware mask autoencoder, wherein the first mask... Figure 2 The middle one is the adaptive mask scheduler, and the second mask is in Figure 2 The middle part is a complexity-guided course mask. The first mask can select the set of edges constituting the course from easy to difficult based on the course learning. The second mask can adaptively select the set of edges to be masked based on the set of edges constituting the course and the sample graph structure data. The structure-aware mask autoencoder takes the output of the second mask as input and reconstructs the masked edge set to obtain the reconstructed graph structure data.

[0099] This invention considers that randomly sampling edges for masking to construct a perturbation graph for training a structure-aware masked autoencoder may introduce optimization difficulties during reconstruction. This may be due to the different roles and relationships of different edges in the graph. The perturbation graph is obtained by masking a subset of edges from randomly selected sample graph structure data. For example, a perturbation graph with less structural information may be difficult for a structure-aware masked autoencoder in the early stages of training because the encoded node vector representation may not be well trained. Therefore, this invention proposes a complexity-guided first masker that selects edges from simple to difficult for masking, making the optimization process for the structure-aware masked autoencoder smoother.

[0100] Specifically, in the first masker, during the training of the structure-aware mask autoencoder, any sample graph structure data from the training dataset is used as input to the structure-aware mask autoencoder to obtain a vectorized representation of each node in the sample graph structure data. Then, a first residual error is calculated between the vectorized representations of any two nodes. Based on the difficulty of reconstructing the edges of the sample graph structure data represented by the first residual error, a first difficulty score is used. Edges to be masked are selected from the sample graph structure data to determine the set of edges constituting the course. When selecting edges belonging to the sample graph structure data based on the first difficulty score, the edges are selected in an order from easy to difficult, so that the training process of the structure-aware mask autoencoder can proceed from easy to difficult, enabling the structure-aware mask autoencoder to obtain more powerful node representations in the sample graph structure data.

[0101] In an optional embodiment of the present invention, the calculation of the first residual error, and the selection of edges to be masked from the sample graph structure data based on the first difficulty score represented by the first residual error, to determine the edge set constituting the course, specifically includes: firstly, based on the vectorized representation of each node in the sample graph structure data, calculating the similarity between the vectorized representations of every two nodes to obtain multiple similarities. Then, based on the multiple similarities, calculating the first residual error of the edge connecting every two nodes. For example, for the vector representations of nodes u and v, the similarity between nodes u and v is calculated to be 0.8, and the first residual error of the edge connecting nodes u and v is calculated to be 1 - 0.8 = 0.2; for the vector representations of nodes i and j, the similarity between nodes i and j is calculated to be 0.1, and the first residual error of the edge connecting nodes i and j is calculated to be 1 - 0.1 = 0.9.

[0102] The first residual error is used to characterize the first difficulty score for reconstructing the sample graph structure data. The first residual error can also represent the confidence level of the structure-aware mask autoencoder in the existence of an edge between two nodes. A smaller residual error indicates a higher probability that an edge exists between the two nodes, meaning that edges corresponding to smaller residual errors are easier for the structure-aware mask autoencoder to learn and understand, and are easier to reconstruct subsequently. For example, when the first residual error between nodes u and v is 0.2, it indicates a higher probability that an edge exists between nodes u and v; when the first residual error between nodes i and j is 0.9, it indicates a lower probability that an edge exists between nodes i and j.

[0103] By combining the first residual error and the sample plot structure data, a residual plot is obtained (see reference). Figure 2 (The residual graph in the image). Then, the first difficulty scores are sorted, and the edges to be masked are selected from the residual graph to determine the set of edges constituting the course. Figure 2 Using the residual plot in the example, in Figure 2 In the residual plot, the residual errors corresponding to edges inside the residual plot are relatively small, while the residual errors corresponding to edges outside the residual plot are relatively large. This means that edges inside the residual plot are easier to reconstruct, while edges outside the residual plot are more difficult to reconstruct. Therefore, the process of selecting edges from easy to difficult based on the scores of the first difficulty level can be found in [reference needed]. Figure 2 An example of the process of selecting edges from easy to difficult in the first mask. During the first training of a structure-aware mask autoencoder pair, the residual map is produced by the untrained encoder. During the second training, a new residual map is produced by the encoder obtained from the previous training.

[0104] The perturbation graph is obtained by masking the edge set selected based on the first difficulty score. This masked perturbation graph is then used as input to a structure-aware masked autoencoder for reconstruction, yielding the reconstructed graph structure data. This invention employs a perturbation-reconstruction self-supervised learning paradigm to improve the quality of the learned node representations. The calculation formula for the perturbation graph is as follows: Where A is the adjacency matrix of the sample graph structure data, A mask Let be the adjacency matrix of the edge set selected based on the first difficulty score. Based on this, the node representations obtained in the above manner are considered information-rich because they must contain sufficient structural and attribute information to reconstruct the original structure from the perturbation graph. These structures can be deeply encoded into the node representations, thus providing stable data for subsequent downstream tasks.

[0105] This invention, through implementing a masking technique on the edge set of a course composed of edges selected based on a lower initial difficulty score, can significantly reduce the task complexity of a structure-aware masked autoencoder as a graph learning model. This is because the graph learning model will primarily focus on learning the basic features and properties of the edge set constituting the course, which have low residual errors and are relatively easy to learn. Therefore, this approach can enhance the overall efficiency and effectiveness of the graph learning model in the learning process.

[0106] This invention also considers the edge selection problem during training. Selecting the appropriate number of edges during training is a challenging task, as edge selection is directly related to discrete optimization problems within a maximal topological space. However, selecting the edge set constituting the course solely based on the difficulty level ranking of the first difficulty score may continuously mask easily processed edges during training, leading to getting stuck on these edges and impairing the generalization ability of the graph learning model. Therefore, this invention proposes a second masker to progressively and autonomously increase the number of edges requiring masking during training. Based on the sample graph structure data and the edge set constituting the course, the edges to be masked are further selected, resulting in a set of edges selected based on the first difficulty score, which serves as the training input for the structure-aware masking autoencoder.

[0107] In one optional embodiment of the invention, firstly, based on the sample graph structure data, a random ratio is introduced to randomly select edges to be masked in the sample graph structure data. Multiple edges selected based on the random ratio are used as a first edge set. The randomly selected edges to be masked do not include edges in the edge set constituting the course. For example, when the sample graph structure data contains 0-9 edges, and the edge set constituting the course contains edges 1, 2, 4, 7, and 8, then the randomly selected edges do not include edges 1, 2, 4, 7, and 8, and may be edges 0, 3, and 5. Then, based on the first edge set and the edge set constituting the course, the edge set selected based on the first difficulty score is determined. For example, the edges in the selected edge set may be edges 1, 2, 4, 7, 8, 0, 3, and 5.

[0108] To further alleviate the problem of impaired generalization ability of graph learning models, this invention introduces a segmentation ratio to promote the number of randomly selected edges. Specifically, the number of edges selected from the set of edges constituting the course and the set of randomly selected edges is first determined by the segmentation ratio. That is, the upper bound of the mask ratio and the random selection ratio are determined by the segmentation ratio, where the upper bound of the mask ratio is used to determine the number of edges selected from the set of edges constituting the course, and the random selection ratio is used to determine the number of edges selected from the sample image structure data. The segmentation ratio encourages the selection of more randomly selected edges as the final set of selected edges. A smaller segmentation ratio means that more edges are randomly selected. The segmentation ratio in this invention aims to achieve a balance between exploring and utilizing the number of edges. Therefore, by adjusting the proportion of randomly selected edges, the overfitting problem can be managed. For example, the upper bound of the mask ratio can be determined to be 0.4 and the random selection ratio to be 0.4 based on the segmentation ratio; the mask ratio can also be determined to be 0.6 and the random selection ratio to be 0.2 based on the segmentation ratio; or the mask ratio can be determined to be 0.4 and the random selection ratio to be 0.6 based on the segmentation ratio.

[0109] Then, edges to be masked are selected from the edge set constituting the course according to the upper bound of the masking ratio as the second edge set. The upper bound of the masking ratio is used to determine the number of edges selected from the sample graph structure data and the edge set constituting the course. For example, when the sample graph structure data contains 0-9 edges, when the edge set constituting the course contains 5 edges (edge ​​1, edge 2, edge 4, edge 7, and edge 8), and the upper bound of the masking ratio is 0.4, it means 5 * 0.4 = 2, and 2 edges (edge ​​1 and edge 2) are selected from the edge set constituting the course.

[0110] Furthermore, based on the sample graph structure data and the random selection ratio, edges to be masked in the sample graph structure data are randomly selected as the third edge set. The randomly selected edges to be masked do not include edges from the second edge set. For example, when the sample graph structure data contains 0-9 edges, and the edges in the second edge set are edge 1 and edge 2, the randomly selected edges will not be edge 1 and edge 2, but will only be selected from edges 0, 3 to 9. Based on the random selection ratio of 0.4, edges 0, 5, and 6 are selected as the third edge set.

[0111] Finally, the edges in the first edge set and the edges in the second edge set are determined to constitute the edge set selected based on the first difficulty score. That is, when the sample graph structure data contains 0-9 edges, when the edges in the second edge set are edge 1 and edge 2, and the edges in the third edge set are edge 0, edge 5, edge 6, and edge 9, the edges in the edge set selected based on the first difficulty score are edge 0, edge 1, edge 2, edge 5, and edge 6. The process of determining the selected edge set based on the second mask can be found in [reference needed]. Figure 2 The second mask diagram is shown in the image.

[0112] In an optional embodiment of the present invention, the perturbation graph obtained by masking the edge set selected based on the first difficulty score is input into a structure-aware mask autoencoder, and reconstructed using the structure-aware mask autoencoder to obtain reconstructed graph structure data. By using the perturbation graph constructed from the edge set whose difficulty matches the current training state of the structure-aware mask autoencoder selected by the first and second masks as training data samples, the structure-aware mask autoencoder can learn more powerful node representations, and the structure-aware mask autoencoder obtained after training can exhibit better performance in downstream tasks.

[0113] Specifically, the structure-aware mask autoencoder includes a GNN encoder and a GNN decoder. The process of masking the selected edge set based on the first difficulty score to obtain a perturbation graph, and using the perturbation graph as input to reconstruct the reconstructed graph structure data, includes: firstly, encoding the nodes in the perturbation graph obtained by masking the selected edge set based on the first difficulty score using the GNN encoder to obtain the node representation for each layer.

[0114] Then, the node representations of each layer are decoded using the GNN decoder, and reconstructed based on the inherent similarity between nodes to obtain the reconstructed graph structure data. Specifically, the GNN decoder includes a cross-correlation decoder and a multilayer perceptron. Decoding the node representations of each layer using the GNN decoder and reconstructing based on the inherent similarity between nodes to obtain the reconstructed graph structure data includes: calculating the edge embedding between any two nodes based on the inherent similarity between any two nodes using the cross-correlation decoder; then using the multilayer perceptron to predict the existence probability of each edge embedding; and reconstructing the graph structure data based on the existence probability to obtain the reconstructed graph structure data.

[0115] For example, the reconstructed graph structure data can be obtained by representing the input sample graph structure data as G(V, E), where V and E represent the sets of edges and nodes, respectively. This graph can also be represented as G(H, A), where H and A represent the node representation and adjacency matrix, respectively. A GNN encoder is used to encode the input sample graph structure data and obtain a node representation, the formula for which is shown below:

[0116]

[0117] in, This represents the embedding of node v at the k-th layer. N represents the embedding of node u in the (k-1)th layer. v ={u:(v,u)∈E} represents the set of all direct neighbors of node v, AGG(·) represents the information aggregation function used to aggregate information about neighboring nodes, and COM(·) represents the combination function used to update the node embedding using the aggregated information.

[0118] This invention employs a multi-layer GNN encoder for stacking, enabling the acquisition of k node representations for any node v. The node representation obtained after encoding using the above GNN encoder is as follows: H = ENC(H,A)∈R N×d Where N represents the number of nodes in the sample graph structure data, d represents the dimension, and ENC(·) represents the GNN encoder.

[0119] For example, a multi-layer GNN encoder can represent different meanings of node u. Suppose node u is a user Y. The first-layer GNN encoder might capture user Y's interests (playing basketball), while the second-layer GNN encoder might capture user Y's geographical location (the basketball court). When it's necessary to determine whether there is a connection between two different users, it's necessary to base the judgment on finer-grained features, that is, to determine whether different layers of different nodes are similar. For example, the first-layer GNN encoder might capture that user Y and user Q work in the same location (both in Beijing), and the second-layer GNN encoder might capture that user Y and user Q work at the same institution (both at Beijing XX University). Since the node representations corresponding to users Y and Q in the two first-layer layers are similar, it indicates a connection between them. Alternatively, it could be assumed that the first-layer GNN encoder captures that user Y and user Q work in the same location (both in Beijing), and the nth (last) layer GNN encoder captures that user Y and user Q work at the same institution (both at Beijing XX University). By calculating the vector representations between the first and last layers using a cross-correlation decoder, it can also be calculated that user Y and user Q are similar, indicating a connection between them. Fine-grained information is obtained by multi-layer GNN encoders. The similarity between any two layers, i.e. the similarity between any two nodes, can be calculated by cross-correlation decoder. The vectorized representation of the nodes is obtained by GNN encoder. GNN decoder determines whether two nodes are connected by an edge based on the vectorized representation of the nodes output by GNN encoder. The reconstructed graph structure data is the result of whether any nodes are connected by an edge or not.

[0120] By fully considering the inherent similarity between any two nodes, the image structure data is reconstructed. The edge embedding is first obtained using the following formula:

[0121]

[0122] Here, ⊙ represents element-wise multiplication. This represents the final representation of the edges, K represents the number of layers, and || represents the join operator. This indicates the embedding of node u at layer j.

[0123] In this way, the common factors among neighboring nodes can be enhanced, while the differences between neighboring nodes are eliminated through element-wise multiplication. Therefore, only elements that are highly correlated among neighboring nodes are retained. Enhancing the common parts of edge representations can facilitate subsequent edge prediction. After obtaining the edge embeddings, a multilayer perceptron (MLP) is used to predict the existence probability of each edge embedding, using the following formula:

[0124] The edge representations generated by the above method play a crucial role in filtering out noise information, thus retaining only common features. This selective information retention helps guide the MLP to make accurate predictions, focusing more on the most relevant and shared features, effectively simplifying the decision-making process, and thus improving the overall efficiency and accuracy of edge prediction. The reconstructed graph structure data obtained by the GNN decoder can be represented as A′←DEC(H)∈R. N×N DNC(·) represents the GNN decoder.

[0125] Finally, the loss value between the sample image structure data and the reconstructed image structure data is calculated using a loss function, and training continues based on this loss value until the training termination condition is met, resulting in a trained mask autoencoder with structure awareness. The loss functions used to calculate the loss value include a first loss function, a second loss function, and a third loss function. The first loss function calculates the loss value between the vectorized representations of any two nodes. The second loss function selects edges with smaller residual errors in the sample image structure data to form the edge set constituting the curriculum. The third loss function optimizes the number of edges in the selected edge set.

[0126] Specifically, the formula for calculating the loss function is as follows:

[0127]

[0128] The formula for calculating the first loss function is as follows: H represents the perturbation graph obtained by masking the edge set selected based on the first difficulty score, H represents all node representations of the sample graph structure data, A represents the adjacency matrix of the sample graph structure data, ENC(·) represents the GNN encoder, and DEC(·) represents the GNN decoder.

[0129] The formula for calculating the second loss function is βΣ. i，j S i，j R ij β represents a balance hyperparameter, S ij This represents the selected set of edges. A represents the residual error of the edge. ij This represents the sample graph structure data. Represents the reconstructed graph structure data, where ||·|| is the l2 norm.

[0130] The third loss function is calculated as f(S; λ), where f(S; λ) = λ||SA||, λ represents the regularization term, A represents the adjacency matrix of the sample graph structure data, and S represents the mask matrix used to indicate the selection of the edge set.

[0131] In an optional embodiment of the present invention, the optimization process of the structure-aware mask autoencoder is as follows: The input data consists of node features X, adjacency matrix A, step size μ, and hyperparameter γ. X can be a description of a user in a chat application; for example, X can be an employee or teacher of a school. The adjacency matrix A indicates the relationships between nodes. Generally, an adjacency matrix is ​​used to represent a graph, and this adjacency matrix is ​​an n*n expression with n nodes. The output is the GNN model parameters w obtained after training. The optimization process is as follows: Step 1: Initialize w (0) =S (0) , λ (0) Step 2: Calculate A (0) =S (0) ⊙A; where A is calculated by randomly initializing a value. (0) S (0) It is an edge selection matrix, A (0) It is the input of a structure-aware mask autoencoder f, that is, the selected set of edges, which is processed by S. (0) Randomly select an edge and perform element-wise multiplication with the adjacency matrix A to obtain A. (0) .

[0132] When the result of the structure-aware mask autoencoder f fails to converge, a loss function is introduced. The residual map calculation is optimized to minimize the difference between the optimized and previous values. The input A of the graph autoencoder is then encoded into an embedding Z using the optimized structure-aware mask autoencoder f. (t) For all pairings of node i and node j, predict edge A. ij The possibility of its existence is Introducing loss function Optimize by selecting the edge corresponding to the minimum residual and updating A. (t) =S (t) ⊙A, where, The residual is The corresponding value, when 0, indicates that the prediction for that edge is accurate, but when the value is not 0, it indicates that the prediction for that edge is inaccurate. Here, S... (t) This indicates which edges were selected after the first round of training, when A (t) When λ ≠ A, update λ according to the step size μ, ending the first round of training and optimization. The optimization process of the structure-aware mask autoencoder described above constitutes one round of optimization, and the above optimization process is repeated.

[0133] In an optional embodiment of the present invention, the step of continuing training based on the loss value includes: firstly, inputting the sample graph structure data into the previously trained structure-aware mask autoencoder through the first mask and the second mask to obtain a new vectorized representation of each node in the sample graph structure data; then, based on the new vectorized representation of each node in the sample graph structure data, obtaining a second residual error; and based on the second difficulty score represented by the second residual error, obtaining a set of edges selected based on the second difficulty score, wherein the second difficulty score is greater than the first difficulty score; the structure-aware mask autoencoder uses the set of edges selected based on the second difficulty score to mask and obtain a new perturbation graph, and uses the new perturbation graph as input to reconstruct new reconstructed graph structure data; finally, based on the sample graph structure data and the new reconstructed graph structure data, calculating a new loss value, and continuing training based on the new loss value. Repeat the above training process until the training termination condition is met, that is, the loss value between the reconstructed graph structure data and the input sample graph structure data is less than the preset loss value, so that the obtained structure-aware mask autoencoder can reconstruct the graph structure data.

[0134] In an optional embodiment of the present invention, the above-described self-supervised pre-training system based on course learning can be used for edge prediction after training. For example, assuming a graph structure data set, after hiding some of its edges, the system can recover the hidden edges. For instance, if the graph structure data is data C from December 11, 2023 to December 20, 2023, then the data from December 21, 2023 to December 23, 2023 is equivalent to the hidden data D. The system can then predict or recover data D based on data C.

[0135] Specifically, in the context of a social network system, if user Y adds three new basketball team members today, the system can predict that user Y might add a fourth or fifth new basketball team member tomorrow. The connections corresponding to the fourth or fifth new basketball team member are the potential added edges. Similarly, in the context of a shopping platform system, if user Y buys some items today, even if some edges are hidden, the system can still predict what items user Y might buy tomorrow. For example, if user Y bought a phone today, the system can predict that user Y might buy phone cases, phone screen savers, or similar items tomorrow, and can add corresponding edges accordingly.

[0136] In an optional embodiment of the present invention, the self-supervised pre-training system based on course learning described above can be used for node prediction after training. For example, in a system corresponding to a purchasing platform, when nodes correspond to different products and different users, the edge between the product and the user represents whether the user will purchase the product. When it is necessary to predict whether a user will purchase all products on the platform, all products are predicted, and it is predicted which products are most likely to form an edge with the user. The products corresponding to the nodes connected by these edges are the products that the user is most likely to purchase. When a user needs to purchase a certain product, for example, a mobile phone, it is only necessary to predict the brand and model of the mobile phone, predicting which brands and models are most likely to form an edge with the user. The mobile phones corresponding to the nodes connected by these edges are the mobile phones that the user is most likely to purchase. For example, in a system corresponding to a dating website, when nodes represent users (male or female) and edges represent attributes, when it is necessary to predict the attribute corresponding to a certain node v, the system predicts whether the node most likely to be connected to node v is male or female. When the node most likely to be connected to node v is female, then the user corresponding to node v can be classified as male.

[0137] This invention proposes a self-supervised pre-training system based on course learning. A first masker inputs sample graph structure data into a structure-aware masking autoencoder during training to obtain a vectorized representation of each node in the sample graph structure data. A first residual error is calculated. Based on a first difficulty score represented by the first residual error, edges to be masked are selected from the sample graph structure data to determine the edge set constituting the course. A second masker selects edges to be masked based on the edge set constituting the course and the sample graph structure data, obtaining a selected edge set based on the first difficulty score. The structure-aware masking autoencoder performs masking based on the selected edge set with the first difficulty score to obtain a perturbation graph. The perturbation graph is used as input for reconstruction to obtain reconstructed graph structure data. Based on the sample graph structure data and the reconstructed graph structure data, a loss value is calculated, and training continues based on the loss value until the training termination condition is met, resulting in a trained structure-aware masking autoencoder. The self-supervised training method proposed in this invention does not require training labels. Instead, it integrates training data samples for reconstruction tasks in a meaningful order from easy to difficult, learns information-based graph neural network representations, and enables the pre-trained system to learn step by step from easy to difficult. It selects appropriate data samples from previous tasks whose difficulty matches the training state of the GNN encoder, achieves more powerful node representations, and exhibits better performance in downstream tasks.

[0138] In a second aspect of this invention, a self-supervised pre-training method based on course learning is proposed, see [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart illustrating the steps of a self-supervised pre-training method based on course learning proposed in an embodiment of the present invention. The method includes the following steps:

[0139] Step 301: Input the sample graph structure data into the structure-aware mask autoencoder during the training process through the first mask to obtain the vectorized representation of each node in the sample graph structure data, calculate the first residual error, and select the edges to be masked from the sample graph structure data based on the first difficulty score represented by the first residual error to determine the edge set that constitutes the course.

[0140] Step 302: Using the second masker, select the edges to be masked based on the set of edges constituting the course and the sample graph structure data, to obtain the set of edges selected based on the first difficulty score;

[0141] Step 303: A perturbation map is obtained by masking the selected edge set based on the first difficulty score using a structure-aware masking autoencoder. The perturbation map is then used as input to reconstruct the reconstructed graph structure data.

[0142] Step 304: Calculate the loss value based on the sample graph structure data and the reconstructed graph structure data, and continue training based on the loss value until the training termination condition is met, thus obtaining a trained mask autoencoder with structure awareness capability.

[0143] Specifically, the sample graph structure data is first input into the structure-aware masking autoencoder during training. By encoding the nodes in the sample graph structure data, a vectorized representation of each node is obtained. Then, the similarity between any two nodes is calculated, and the first error residual of the edge between any two nodes is calculated based on the similarity. Based on the first difficulty score represented by the first residual error, edges to be masked are selected from the sample graph structure data to determine the edge set constituting the course. The difficulty of the selected edges to be masked matches the training state of the structure-aware masking autoencoder during the current training process. That is, when the structure-aware masking autoencoder is first trained, its training state should be relatively low in difficulty. At this time, the difficulty score of the selected edges to be masked should be relatively low, allowing the structure-aware masking autoencoder to learn from simple training data samples to difficult training data samples, thus making the training and optimization process smoother.

[0144] Furthermore, to progressively and autonomously increase the number of edges requiring masking during training, a second mask is used to select edges from the edge set constituting the course and the sample graph structure data, resulting in a selected edge set. Since the edges in the edge set constituting the course are selected based on the first difficulty score, selecting edges solely based on difficulty would inevitably lead to continuous masking of easily processed edges during training. Therefore, a segmentation ratio is introduced within the second mask to increase the number of randomly selected edges from the sample graph structure data.

[0145] Then, a perturbation map is obtained by masking the selected edge set based on the first difficulty score using a structure-aware mask autoencoder. This perturbation map is then used as input for reconstruction, resulting in reconstructed graph structure data. Based on the sample graph structure data and the reconstructed graph structure data, a loss value is calculated, and training continues based on this loss value until the training termination condition is met, resulting in a trained structure-aware mask autoencoder.

[0146] The method further includes:

[0147] The first mask and the second mask input the sample graph structure data into the structure-aware mask autoencoder obtained from the previous training, and obtain a new vectorized representation of each node in the sample graph structure data.

[0148] Based on the new vectorized representation of each node in the sample graph structure data, a second residual error is obtained. Based on the second difficulty score represented by the second residual error, a set of edges selected based on the second difficulty score is obtained, wherein the second difficulty score is greater than the first difficulty score.

[0149] A structure-aware mask autoencoder uses a set of edges selected based on the second difficulty score to mask and obtain a new perturbation map. The new perturbation map is then used as input to reconstruct the new reconstructed map structure data.

[0150] Based on the sample graph structure data and the new reconstructed graph structure data, a new loss value is calculated, and training continues based on the new loss value.

[0151] The calculation yields a first residual error. Based on the first difficulty score represented by the first residual error, edges to be masked are selected from the sample graph structure data to determine the set of edges constituting the course, including:

[0152] Based on the vectorized representation of each node, the similarity between the vectorized representations of every two nodes is calculated to obtain multiple similarity scores.

[0153] Based on multiple similarities, the first residual error of the edge connecting every two nodes is calculated, and the first residual error is used to characterize the first difficulty score for reconstructing the sample graph structure data;

[0154] Based on the first residual error and the sample graph structure data, a residual graph is obtained;

[0155] The first difficulty scores are sorted, and the edges to be masked are selected from the residual graph to determine the set of edges that constitute the course.

[0156] The step of selecting edges to be masked based on the edge set constituting the course and the sample graph structure data, to obtain the selected edge set based on the first difficulty score, includes:

[0157] Based on the sample graph structure data, the edges to be masked in the sample graph structure data are randomly selected as the first edge set. The randomly selected edges to be masked do not include the edges in the edge set that constitutes the course.

[0158] The selected edge set based on the first edge set and the edge set constituting the course is determined.

[0159] The step of determining the selected edge set based on the first difficulty score according to the first edge set and the edge set constituting the course further includes:

[0160] The upper bound of the mask ratio and the random selection ratio are determined by the segmentation ratio. The upper bound of the mask ratio is used to determine the number of edges selected from the set of edges constituting the course, and the random selection ratio is used to determine the number of edges selected from the sample image structure data.

[0161] According to the upper bound of the masking ratio, select the edges to be masked from the set of edges constituting the course as the second set of edges;

[0162] Based on the sample graph structure data and the random selection ratio, the edges to be masked in the sample graph structure data are randomly selected as the third edge set, and the randomly selected edges to be masked do not include the edges in the second edge set.

[0163] Based on the edges in the second edge set and the edges in the third edge set, the edge set that constitutes the selection based on the first difficulty score is determined.

[0164] The structure-aware mask autoencoder includes a GNN encoder and a GNN decoder. The perturbation map is obtained by masking the edge set selected based on the first difficulty score. The perturbation map is then used as input for reconstruction to obtain reconstructed graph structure data, including:

[0165] The nodes in the perturbation graph are encoded by the GNN encoder to obtain the node representation for each layer.

[0166] The GNN decoder decodes the node representation of each layer and reconstructs it based on the inherent similarity between nodes to obtain the reconstructed graph structure data.

[0167] The GNN decoder includes a cross-correlation decoder and a multilayer perceptron. The GNN decoder decodes the node representations of each layer and reconstructs the data based on the inherent similarity between nodes to obtain the reconstructed graph structure data, including:

[0168] Based on the inherent similarity between any two nodes, the edge embedding of any two nodes is calculated by the cross-correlation decoder;

[0169] A multilayer perceptron is used to predict the existence probability of each edge embedding.

[0170] The reconstruction is performed based on the existence probability to obtain the reconstructed graph structure data.

[0171] The loss function used to calculate the loss value includes a first loss function, a second loss function, and a third loss function;

[0172] The loss value between the vectorized representations of any two nodes is calculated using the first loss function;

[0173] The edge set constituting the course is constructed by selecting edges with smaller residual errors in the sample image structure data using a second loss function.

[0174] The number of edges in the selected edge set is optimized using a third loss function.

[0175] The formula for calculating the loss function is as follows:

[0176]

[0177] The formula for calculating the first loss function is as follows: H represents the perturbation graph obtained by masking the edge set selected based on the first difficulty score, H represents all node representations of the sample graph structure data, A represents the adjacency matrix of the sample graph structure data, ENC(·) represents the GNN encoder, and DEC(·) represents the GNN decoder.

[0178] The formula for calculating the second loss function is β∑ i，j s ij R ij β represents a balance hyperparameter, S ij This represents the selected set of edges. A represents the residual error of the edge. ij This represents the sample graph structure data. This represents the reconstructed graph structure data, where ||·|| represents the l2 norm.

[0179] The third loss function is calculated as f(S; λ), where f(S; λ) = λ||SA||, λ represents the regularization term, A represents the adjacency matrix of the sample graph structure data, and S represents the mask matrix used to indicate the selection of the edge set.

[0180] In an optional embodiment of the present invention, the inventors applied the self-supervised pre-training method based on curriculum learning proposed in this invention to a node classification and link prediction dataset to verify its performance. The verification results are shown below:

[0181] Part 1: The self-supervised pre-training method based on curriculum learning proposed in this invention is compared with world-leading methods in related fields for node classification, and the corresponding node classification accuracy (%) is obtained (the results are shown in Table 1). In each column, the bold score represents the best result among all methods, and the corresponding ranking is the average ranking. It can be seen that the self-supervised pre-training method based on curriculum learning proposed in this invention achieves the best performance in terms of average ranking and has the best node prediction accuracy.

[0182] Table 1 Node Classification Accuracy Results

[0183]

[0184] Part Two: The self-supervised pre-training method based on curriculum learning proposed in this invention is compared with world-leading methods in related fields for link prediction, and the corresponding link prediction results (%) are obtained (see Table 2). In each column, the bold score represents the best result among all methods, and the corresponding ranking is the average ranking. It can be seen that the self-supervised pre-training method based on curriculum learning proposed in this invention achieves significant performance improvements compared with other state-of-the-art methods in small-scale datasets or more challenging large-scale benchmark tests. ("-" in Table 2 indicates insufficient memory in the experiment.)

[0185] Table 2 Link Prediction Results

[0186]

[0187] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0188] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods and apparatus according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0189] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0191] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0192] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0193] The above provides a detailed description of the self-supervised pre-training system and method based on course learning provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A self-supervised pre-training system based on curriculum learning, characterized in that, The system includes: The first masker inputs the sample graph structure data into a structure-aware masking autoencoder during the training process to obtain a vectorized representation of each node in the sample graph structure data, calculates the first residual error, and selects the edges to be masked from the sample graph structure data based on the first difficulty score represented by the first residual error to determine the set of edges that constitute the course. The second masker, based on the set of edges constituting the course and the sample graph structure data, selects the edges to be masked, thus obtaining the set of edges selected based on the first difficulty score. A structure-aware mask autoencoder performs masking based on the edge set selected by the first difficulty score to obtain a perturbation map, and uses the perturbation map as input to reconstruct the reconstructed map structure data. Based on the sample graph structure data and the reconstructed graph structure data, the loss value is calculated, and training continues based on the loss value until the training termination condition is met, resulting in a trained mask autoencoder with structure awareness. The calculation yields a first residual error. Based on the first difficulty score represented by the first residual error, edges to be masked are selected from the sample graph structure data to determine the set of edges constituting the course, including: Based on the vectorized representation of each node, the similarity between the vectorized representations of every two nodes is calculated to obtain multiple similarity scores. Based on multiple similarities, the first residual error of the edge connecting every two nodes is calculated, and the first residual error is used to characterize the first difficulty score for reconstructing the sample graph structure data; Based on the first residual error and the sample graph structure data, a residual graph is obtained; Sort the first difficulty scores, select the edges to be masked from the residual graph, and determine the set of edges that constitute the course; The structure-aware mask autoencoder includes a GNN encoder and a GNN decoder. The perturbation map is obtained by masking the edge set selected based on the first difficulty score. The perturbation map is then used as input for reconstruction to obtain reconstructed graph structure data, including: The nodes in the perturbation graph are encoded by the GNN encoder to obtain the node representation for each layer. The GNN decoder decodes the node representation of each layer and reconstructs it based on the inherent similarity between nodes to obtain the reconstructed graph structure data. The GNN decoder includes: a cross-correlation decoder and a multilayer perceptron; the GNN decoder decodes the node representation of each layer and reconstructs the graph based on the inherent similarity between nodes to obtain the reconstructed graph structure data, including: Based on the inherent similarity between any two nodes, the edge embedding of any two nodes is calculated by the cross-correlation decoder; A multilayer perceptron is used to predict the existence probability of each edge embedding. The reconstruction is performed based on the existence probability to obtain the reconstructed graph structure data.

2. The self-supervised pre-training system based on course learning according to claim 1, characterized in that, The step of continuing training based on the loss value includes: The first mask and the second mask input the sample graph structure data into the structure-aware mask autoencoder obtained from the previous training, and obtain a new vectorized representation of each node in the sample graph structure data. Based on the new vectorized representation of each node in the sample graph structure data, a second residual error is obtained. Based on the second difficulty score represented by the second residual error, a set of edges selected based on the second difficulty score is obtained, wherein the second difficulty score is greater than the first difficulty score. A structure-aware mask autoencoder uses a set of edges selected based on the second difficulty score to mask and obtain a new perturbation map. The new perturbation map is then used as input to reconstruct the new reconstructed map structure data. Based on the sample graph structure data and the new reconstructed graph structure data, a new loss value is calculated, and training continues based on the new loss value.

3. The self-supervised pre-training system based on course learning according to claim 1, characterized in that, Based on the edge set constituting the course and the sample graph structure data, the edges to be masked are selected to obtain the selected edge set based on the first difficulty score, including: Based on the sample graph structure data, the edges to be masked in the sample graph structure data are randomly selected as the first edge set. The randomly selected edges to be masked do not include the edges in the edge set that constitutes the course. The selected edge set based on the first edge set and the edge set constituting the course is determined.

4. The self-supervised pre-training system based on course learning according to claim 3, characterized in that, The step of determining the selected edge set based on the first difficulty score according to the first edge set and the edge set constituting the course further includes: The upper bound of the mask ratio and the random selection ratio are determined by the segmentation ratio. The upper bound of the mask ratio is used to determine the number of edges selected from the set of edges constituting the course, and the random selection ratio is used to determine the number of edges selected from the sample graph structure data. According to the upper bound of the masking ratio, select the edges to be masked from the set of edges constituting the course as the second set of edges; Based on the sample graph structure data and the random selection ratio, the edges to be masked in the sample graph structure data are randomly selected as the third edge set, and the randomly selected edges to be masked do not include the edges in the second edge set. Based on the edges in the second edge set and the edges in the third edge set, the edge set that constitutes the selection based on the first difficulty score is determined.

5. The self-supervised pre-training system based on course learning according to claim 1, characterized in that, The loss functions used to calculate the loss value include a first loss function, a second loss function, and a third loss function; The loss value between the vectorized representations of any two nodes is calculated using the first loss function; The edge set constituting the course is constructed by selecting edges with smaller residual errors in the sample graph structure data using a second loss function. The number of edges in the selected edge set is optimized using a third loss function.

6. The self-supervised pre-training system based on course learning according to claim 5, characterized in that, The formula for calculating the loss function is as follows: ， The formula for calculating the first loss function is as follows: , H represents the perturbation graph obtained by masking the edge set selected based on the first difficulty score, H represents all node representations of the sample graph structure data, A represents the adjacency matrix of the sample graph structure data, ENC(·) represents the GNN encoder, and DEC(·) represents the GNN decoder. The formula for calculating the second loss function is as follows: , Indicates a balanced hyperparameter, This represents the selected set of edges. This represents the residual error of the edge. This represents the sample graph structure data. Represents the reconstructed graph structure data. express Norm; The formula for calculating the third loss function is as follows: ,in, λ represents the regularization term, A represents the adjacency matrix of the sample graph structure data, and S represents the mask matrix used to indicate the selection set of edges.

7. A self-supervised pre-training method based on curriculum learning, characterized in that, The method is implemented using the self-supervised pre-training system based on course learning as described in any one of claims 1-6, comprising: The sample graph structure data is input into the structure-aware mask autoencoder during the training process through the first mask, and the vectorized representation of each node in the sample graph structure data is obtained. The first residual error is calculated. Based on the first difficulty score represented by the first residual error, the edges to be masked are selected from the sample graph structure data to determine the set of edges that constitute the course. The second masker selects the edges to be masked based on the set of edges constituting the course and the sample graph structure data, thereby obtaining the set of edges selected based on the first difficulty score. A perturbation map is obtained by masking the selected edge set based on the first difficulty score using a structure-aware mask autoencoder. The perturbation map is then used as input to reconstruct the reconstructed graph structure data. Based on the sample graph structure data and the reconstructed graph structure data, a loss value is calculated, and training continues based on the loss value until the training termination condition is met, resulting in a trained mask autoencoder with structure awareness.

Images