Graph representation

Abstract

To relate to extracting representations from graphs.SOLUTION: A computer-implemented method comprises performing a training process, the training process comprising: using a student graph neural network (GNN), to extract a pair of first representations from a pair of training graphs, respectively; computing a disagreement between the first representations; using at least one teacher GNN to extract a pair of second representations from the pair of training graphs, respectively; computing a perceptual distance between the second representations; comparing the disagreement between the first representations with a target disagreement which is based at least in part on the perceptual distance between the second representations; and adjusting at least one weight of the student GNN based on the comparison.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to extracting representations from graphs, and in particular to computer-implemented methods, computer programs and information programming apparatus. [Background technology]

[0002] Annotating graphical input (i.e., graphs) is often expensive, time-consuming, and may be practically impossible at scale. Instead of annotating / labeling graphs, it may be useful to extract information representations from unlabeled graphs. For example, extracting information representations from unlabeled graphs can be a cheap and time-saving solution to the problem of needing labeled data to train systems on tasks involving graphs.

[0003] Neural networks are commonly used to learn representations using self-supervised learning techniques. For example, graph neural networks can be used to learn representations from graphs. Such representations may be used for downstream tasks in various real-world domains, such as any of the following non-exhaustive list: (1) in the medical domain, for (a) drug discovery, and (b) to detect disease outbreaks; (2) in the chemical domain, to predict chemical properties of unknown compounds; (3) in social networks, for (a) controlling user privacy, and (b) to detect spam emails; and (4) in navigation systems, to find efficient routes based on current traffic patterns.

[0004] In light of the above, it is desirable to efficiently extract representations from graphs. Summary of the Invention

[0005] According to an embodiment of the first aspect, there is provided a computer-implemented method comprising the step of performing a training process, said training process comprising: extracting a pair of first representations from a pair of training graphs using a student / first graph neural network (GNN); - calculating / measuring / determining a discrepancy / difference between said first representations (of a pair of first representations); extracting a pair of second representations from each pair of training graphs using at least one (trained) teacher / second GNN; - calculating a perceptual distance between said second representations (of a pair of second representations); comparing a target disparity / difference between the first representations based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student / first GNN based on the comparison (between the discrepancy / difference and the target discrepancy / difference) (to bring the discrepancy / difference to or closer to the target discrepancy / difference); A computer-implemented method is disclosed herein, comprising:

[0006] Features associated with any aspect / embodiment may be applied to any other aspect / embodiment. [Brief description of the drawings]

[0007] By way of example, reference is made to the accompanying drawings in which: [Figure 1] FIG. 13 is a diagram showing a comparison method. [Diagram 2] FIG. 13 is a diagram showing a comparison method. [Diagram 3] FIG. 13 is a diagram showing a comparison method. [Figure 4] FIG. 2 illustrates the training process. [Diagram 5] FIG. 2 illustrates the training process. [Figure 6] FIG. 2 illustrates the training process. [Figure 7] FIG. 2 illustrates the training process. [Figure 8] FIG. 2 illustrates the training process. [Figure 9] FIG. 2 illustrates the training process. [Figure 10] 1 is a diagram useful for understanding the invention. [Figure 11] 1 is a table of results. [Figure 12] FIG. 1 illustrates an exemplary implementation. [Figure 13] FIG. 1 illustrates an exemplary implementation. [Figure 14] FIG. 1 illustrates an exemplary implementation. [Figure 15] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] FIG. 1 illustrates a first comparison method. The first comparison method is a predictive or prediction-based method for training a graph neural network (GNN) to extract representations from a graph. The first comparison method is illustrated in "1. Predictive Pre-Training." Attribute masking is performed on the input graph. In other words, at least one node and / or at least one edge is "masked" or removed from the input graph. The GNN is then used to predict the masked attributes. The GNN is updated based on the difference between the predicted attributes and the actual masked attributes. In this way, the GNN is trained. Illustrated in "2. Inference Phase" is the subsequent use of the trained GNN to extract graph representations from the input graph.

[0009] The problem with the first comparison method is that GNNs trained this way tend not to correctly predict the global structure of the graph.

[0010] FIG. 2 illustrates a second comparison method. The second comparison method is a contrastive learning-based method for training a graph neural network (GNN) to extract representations from a graph. In the second comparison method, the input graph is augmented (e.g., by removing nodes and deleting / transforming edges). The GNN is used to extract representations from two augmentations of the input graph (as shown, two instances of the GNN with common parameters can be used). The GNN is then tuned to maximize the match between the two extracted representations.

[0011] A problem with the second comparison method is that perturbations / extensions of parts of the graph (e.g., editing nodes / edges) may significantly change the properties of the graph: for example, two extensions of an input graph may have very different properties (e.g., in the domain of interest) even if they are considered similar by the second comparison method.

[0012] FIG. 3 illustrates a third comparison method. The third comparison method is a discrepancy-based self-supervised learning (D-SLA) method that trains a graph neural network (GNN) to extract representations from a graph. In the third comparison method, a first input graph is augmented to generate two augmentations (e.g., in the same manner as the input graph is augmented in the second comparison method). The GNN is used to extract representations from the two augmentations of the first input graph. The GNN is tuned such that the discrepancy between the representations is at or approaches an amount of loss proportional to the edit distance between the augmentations. The edit distance can, for example, include / be based on the minimum number of augmentations required to obtain one augmented version of the input graph starting from other augmentations of the input graph.

[0013] The third comparison method further includes using the GNN to extract a representation from a second input graph (not a simple extension of the first input graph). Then, tuning the GNN also aims to bring the mismatch between the representation of the second input graph and the representation of the extension of the first input graph to or close to a value greater than the mismatch between the representations of the extensions of the first input graph.

[0014] In other words, the GNN is trained to make the first disparity between the representations of the augmentations of the first input graph proportional to the edit distance between the augmentations, and to make the representations of the second input graph and the augmentations of the first input graph disparate "more" (by at least a predetermined margin) than the representations of the augmentations of the first input graph.

[0015] A problem with the third comparison method is that the edit distance between two extensions of an input graph (or between a graph and its extended / perturbed variation) may not capture the exact semantic differences between these graphs. Another problem is that computing the edit distance between any two graphs is NP-hard.

[0016] An example implementation of the training process is shown in Figure 4 for training a GNN to extract representations from a graph.

[0017] In circle 1, an input graph (input graph 1) is augmented, for example by any of the following: removing at least one node, adding at least one node, removing at least one edge, adding at least one edge. In this way, two augmentations of input graph 1 are generated, which can be called the first and second input graphs. The augmentations can be called augmented versions or perturbed versions / changes. Instead of performing two separate augmentations, one can perform a single augmentation and use the original input graph 1 for further steps instead of the second augmented version of input graph 1. That is, instead of using two augmented versions of input graph 1 as the first and second input graphs, the augmented version and the original input graph 1 can be used as the first and second input graphs.

[0018] In circle 2, GNN 10 is used to extract representations from the first and second input graphs, respectively. GNN 10 may be referred to as the first GNN or the student GNN. As shown, two instances 11 and 12 of student GNN 10 are used for the first and second input graphs, respectively. Student GNNs 11 and 12 have common parameters. In other words, student GNNs 11 and 12 have the same parameters as each other. Thus, they can be considered as the same GNN. Student GNNs 11 and 12 may be collectively referred to as student GNN 10.

[0019] In circle 3, a different GNN 20 is used to extract representations from the first and second input graphs, respectively. The GNN 20 can be referred to as a second GNN or a teacher GNN. As shown, two instances 21 and 22 of the teacher GNN 20 are used for the first and second input graphs, respectively. The teacher GNNs 21 and 22 have common parameters. In other words, the teacher GNNs 21 and 22 have the same parameters as each other. Thus, they can be considered as the same GNN. The teacher GNNs 21 and 22 can be collectively referred to as the teacher GNN 20. The teacher GNN 20 is a trained GNN. That is, the teacher GNN 20 has already undergone a training process.

[0020] In circle 4, a perceptual distance between the representations of the first and second input graphs extracted by the teacher GNN 20 is calculated. The perceptual distance may be or may include an L2 distance, as will be explained in more detail later. A discrepancy between the representations of the first and second input graphs extracted by the student GNN 10 is calculated, which may be or may include an L2 distance. This discrepancy is compared to a target discrepancy based at least in part on the perceptual distance. The student GNN 10 adjusts the discrepancy to or close to the target discrepancy (in other words, to minimize the error / loss calculated between the discrepancy and the target discrepancy). The first and second GNNs 10 and 20 may be referred to by the labels "student" and "teacher", since the first GNN 10 can be considered to be "learning" from the second GNN 20 by the comparison of discrepancies by perceptual distance. The labels "student" and "teacher" may be replaced by "first" and "second".

[0021] In circle 5, student GNN10 is used to extract a representation from a different input graph, input graph2.

[0022] In circle 6, a teaching GNN 20 is used to extract a representation from the input graph 2.

[0023] In circle 7, a perceptual distance is calculated between the representation of the first input graph extracted by the teacher GNN 20 and the representation of input graph 2 extracted by the teacher GNN. The perceptual distance may be or may include an L2 distance. A discrepancy between the representations of the first input graph and input graph 2 extracted by the student GNN 10 is calculated, which may be or may include an L2 distance. This discrepancy is compared to another target discrepancy based at least in part on the perceptual distance between the representations of the first input graph and input graph 2 extracted by the teacher GNN 20. The student GNN 10 adjusts the discrepancy to or close to this target discrepancy (in other words, to minimize the error / loss calculated between the discrepancy and the target discrepancy).

[0024] As shown in Figure 4, the processing of circles 5-7 may be performed together with the processing of circles 1-4 (i.e., in the same training process). In this case, for example, another instance 13 of student GNN 10 and another instance 23 of teacher GNN 20 may be used as shown in the figure. Parameters are shared (i.e., identical) between instances 11-13 of student GNN, and these instances may be collectively referred to as student GNN 10. Parameters are shared (i.e., identical) between instances 21-23 of teacher GNN, and these instances may be collectively referred to as teacher GNN 20.

[0025] The training process may include processing of circles 1-4 rather than processing of circles 5-7. Alternatively, the training process may include processing of circles 5-7 rather than processing of circles 1-4. It will be appreciated that the processing of circles 2-4 is the same as the processing of circles 5-7, except that the input graph used is different. In this way, the training process can be said to include processing of circles 2-4 based on two input images. These input images may be two different and separate input images, one input image may be an augmented or expanded version of the other, or both input images may be different augmented or expanded versions of another input image.

[0026] In an example implementation, the method includes iterating the training process to train the student GNN 10 (to bring the mismatch up to or close to the target mismatch). Different iterations include using different pairs of input images. The method can include iterations using the processes in circles 1-4, iterations using the processes in circles 5-7, and iterations using the processes in circles 1-7.

[0027] In FIG. 4, a further teacher GNN is shown behind an instance of teacher GNN 20. Multiple different teacher GNNs can be used, not just teacher GNN 20, which extracts representations from the two input graphs involved. That is, multiple teacher GNNs can be used to extract multiple representation pairs from the pair of input images. Multiple perceptual distances can then be calculated corresponding to each teacher GNN. Target mismatch in such a training process can be based, at least in part, on the average (i.e., mean) of these perceptual distances.

[0028] FIG. 5 illustrates a training phase and an inference or testing phase according to an implementation example. The training phase includes phases 1, 2, and 3. The training phase and the inference or testing phase are described in more detail in FIGS. 6-8. The training phase and the testing phase can be considered as a training process. The training phase or only phase 2, only phases 2 and 3, or only phases 1 and 2 can be considered as a training process.

[0029] Training Phase 1 involves training at least one teacher GNN. This training can be performed using Comparison Method 3, or any other comparison method, or any other training method. In Training Phase 1, multiple teacher GNNs can be trained.

[0030] Training phase 2 involves training a student GNN. For example, the training process described with respect to FIG. 4 can be repeated using the teacher GNN trained in training phase 1.

[0031] Training phase 3 involves further training for a specific task, i.e., the task for which the student GNN is intended to be used. This task can be called the downstream task. This training phase can be thought of as involving fine-tuning the student GNN trained in training phase 2.

[0032] The inference or testing phase involves using the test input graph to extract representations for testing / validation using the trained student GNN.

[0033] FIG. 6 illustrates the method according to training phases 1 and 2, respectively.

[0034] The method according to training phase 1 includes steps S11 to S17. Step S11 includes loading training data in the form of an input graph or "training graph". The training graphs are unlabeled. One or two or more training graphs can be loaded. In an implementation, multiple training graphs are loaded in the form of a batch of training graphs.

[0035] Step S12 involves encoding the input samples, i.e. encoding at least one training graph. In an implementation, a batch of training graphs is encoded.

[0036] Step S13 includes embedding the training graph in at least one of the at least one teacher GNN to be trained, or in other words, extracting a representation from the training graph in the at least one teacher GNN.

[0037] Step S14 includes performing a self-supervised task on the embedding / representation. That is, step S14 includes performing a process corresponding to a training method, for example, any one of the first to third comparison methods. For example, step S14 may include using at least one teacher GNN to extract a representation from another training graph and calculating the discrepancy between the two representations.

[0038] Step S15 includes calculating an error. The error is calculated according to the training method used (e.g., any of the first to third comparison methods). For example, step S15 can include determining an edit distance between two training graphs from which the teacher GNN extracted expressions, and calculating an error based on the discrepancy between the expressions and the edit distance between the graphs.

[0039] Step S15 may involve a comparison rather than a direct error calculation, for example, step S15 may involve comparing an edit distance to the discrepancy.

[0040] Step S16 includes adjusting at least one weight of the teacher GNN, i.e., to reduce the calculated error, for example, the weight can be adjusted to scale the discrepancy to the edit distance or proportional to the edit distance.

[0041] Step S17 involves determining whether the maximum number of epochs has been reached. If Yes, the method ends. If No, the method proceeds to step S11 where another at least one training graph is loaded. The method iterates in this manner. Alternatively, error convergence may be determined to determine whether to perform further iterations or to end the method. For example, it may be determined whether the error is below a threshold for a predetermined number of iterations. This may be determined in addition to the maximum epoch determination. That is, the method may end if either determination is true ("Yes"), i.e., if the maximum number of epochs has been reached or if it is determined that the error is / has converged.

[0042] As described above, the teacher GNN may be trained using any training technique, for example, any of the first to third comparison methods. The training phase 1 method may be considered to not include step S11 and / or step S12, and one or both of these steps may be performed in a pre-method process.

[0043] Training phase 2 includes steps S20 to S29. Step S20 includes loading training data in the form of an input graph or "training graph". The training graphs are unlabeled. One or two or more training graphs can be loaded. In an implementation, multiple training graphs are loaded in the form of a batch of training graphs.

[0044] Step S21 involves encoding the input sample, i.e. encoding the loaded training graph (or one of the loaded training graphs). In an implementation, a batch of training graphs is encoded.

[0045] Step S22 includes generating an augmented view of the training graph. The augmentation may include at least one of the following: removing at least one node, adding at least one node, removing at least one edge, adding at least one edge. The augmentation may be referred to as an augmented version or a perturbed view / version.

[0046] Step S23 includes embedding the training graph and the perturbed views into a student GNN to be trained to generate / extract two representations, which may be referred to herein as a first representation pair. Step S23 further includes embedding the training graph and the perturbed views into a trained teacher GNN (i.e., trained in training phase 1) to generate / extract two representations, which may be referred to herein as a second representation pair.

[0047] Step S24 involves calculating a perceptual loss or distance between the second representations (or "teacher representations"). As will be explained below, the perceptual distance may be or include an L2 distance.

[0048] Step S25 involves calculating a mismatch between the first representations (or "student representations"). As described below, the mismatch may be or may include an L2 distance. This mismatch may be referred to as the student perceptual distance, and the perceptual distance between the second representations may be referred to as the teacher perceptual distance.

[0049] Step S26 includes computing at least one other loss term, which may include, for example, a task-specific loss term and / or, for example, an edit-distance based loss term. Step S26 is optional.

[0050] Step S27 includes calculating an error, for example between the mismatch calculated in step S25 and a target mismatch based at least in part on the perceptual distance calculated in step S24 and, if applicable, at least one loss term calculated in step S26.

[0051] Step S27 may involve a comparison rather than a direct error calculation, for example, step S27 may involve comparing the discrepancy to a target discrepancy.

[0052] Step S28 includes adjusting at least one weight of the student GNN to reduce the error, i.e., to bring the mismatch calculated in step S25 to or closer to the target mismatch. In an implementation, all weights of the student GNN are adjusted.

[0053] Step S29 involves determining whether the maximum number of epochs has been reached. If Yes, the method ends. If No, the method proceeds to step S20 where another training graph is loaded. The method thus repeats. Alternatively, error convergence may be determined to determine whether to perform further iterations or to end the method. For example, it may be determined whether the error is below a threshold for a predetermined number of iterations. This may be determined in addition to the maximum epoch determination. That is, the method may end if either determination is true ("Yes"), i.e., if the maximum number of epochs has been reached or if it is determined that the error is / has converged.

[0054] The training phase 2 method may be considered to not include step S20 and / or step S21. One or both of these steps may be considered to be performed in a pre-method process. For example, all of the training graph may be loaded (and encoded) prior to the training phase 2 method.

[0055] For a given iteration, the training graph and the perturbed version may be referred to as a pair of training graphs (or a pair of input graphs). In at least one iteration, two different training graphs may be used as the pair of training graphs instead of the training graph and its perturbed version, and the method may not include step S22 for the at least one iteration. In an implementation, an iteration (e.g., some or all of the iterations) may include using a first pair of training graphs including the reference graph and its perturbed variation, and a second pair of training graphs including the reference graph and another graph that is not the perturbed variation. That is, the reference graph may be shared / common between the two pairs. In this case, the discrepancy and perceptual distance may be calculated corresponding to each pair, and the comparison and weight adjustment (comparison and weight adjustment described above) may be performed for each pair. The weight adjustment may be performed for both pairs of training graphs at once. In this implementation, the reference graph and its perturbed variation may be considered as a positive training example or positive pair, and the reference graph and the different graph may be considered as a negative training example or negative pair.

[0056] At the end of the training phase 2 method, a trained student GNN is provided.

[0057] Figure 7 shows an example of another implementation of the training process. It will be understood that the method of Figure 7 can correspond to the method of Figure 6 (training phase 2) with some adjustments.

[0058] The method of FIG. 7 includes steps S51 to S57.

[0059] Step S51 involves extracting a pair of first representations from a pair of training graphs, respectively, using the student / first GNN.

[0060] Step S52 involves calculating / measuring / determining the discrepancy / difference between the first representations (of the pair of first representations).

[0061] Step S53 involves extracting a pair of second representations from a pair of training graphs respectively using at least one (trained) teacher / second GNN.

[0062] Step S54 involves calculating a perceptual distance between the second representations (of the pair of second representations).

[0063] Step S55 involves comparing the disparity / difference between the first representations to a target disparity / difference that is based at least in part on the perceptual distance between the second representations.

[0064] Step S56 includes adjusting at least one weight of the student / first GNN based on the comparison, e.g., to bring the discrepancy / difference closer to a target discrepancy / difference. In an implementation, all weights of the student GNN are updated.

[0065] Step S57 involves deciding whether to perform another iteration using a different pair of training graphs (returning to step S51) or to end the method; the considerations discussed above with reference to steps S17 and S29 may be considered to apply to step S57.

[0066] Steps S51 to S56 or steps S51 to S57 can be called a training process.

[0067] In the above description of Figures 6 and 7, adjusting at least one weight may be referred to as updating at least one network parameter.

[0068] FIG. 8 shows two methods that respectively correspond to the training phase 3 and the testing phase shown in FIG.

[0069] The method for training phase 3 includes steps S31-S37 and is for training a student GNN for a particular task (downstream task).

[0070] Step S31 involves loading training data, for example at least one training graph. In an implementation, multiple training graphs are loaded, for example in the form of a (training) batch.

[0071] Step S32 involves encoding the input samples (at least one training graph). The considerations discussed above with reference to steps S12 and S21 may be applied to step S32.

[0072] Step S33 involves adding task-specific blocks to the trained student GNN. For example, a classification head (as a task-specific block) can be added to the student GNN so that the student GNN can be used for the task (and trained / fine-tuned specifically for the task). This step can be considered to be performed as a preliminary step before the training phase 3 method.

[0073] Step S34 includes predicting at least one desired label. This is an example of a task. The at least one label may be of a node and / or an edge and / or the entire graph. The at least one label may be referred to as at least one class label. Step S34 may alternatively or additionally include predicting a node or an edge in the graph. A number of example tasks are described below. Step S34 includes making a prediction using the student GNN trained in training phase 2.

[0074] Step S35 includes calculating an error between the predicted label and the ground truth (actual label). For example, if an edge or node is being predicted, step S35 may include calculating an error between the prediction and the ground truth (actual) node or edge. Step S35 generally includes comparing the prediction to the ground truth.

[0075] Step S36 involves updating the network parameters of the student GNN, i.e. adjusting at least one weight, to reduce the error, i.e. bring the prediction up to or closer to the ground truth. In an implementation, all weights of the student GNN are updated.

[0076] Step S37 involves deciding whether to perform another iteration (return to step S31) or to end the method. The considerations discussed above with reference to steps S17 and S29 may apply to step S37.

[0077] The method according to the test phase comprises steps S41 to S43. Step S41 comprises loading the test data, i.e. the test graph. Step S42 comprises encoding the input samples (the test graph). The considerations mentioned above with respect to the loading and encoding steps can be applied to steps S41 and S42. Step S43 comprises generating predictions using the student GNN that has been trained (according to training phase 2) and / or further trained or fine-tuned according to training phase 3.

[0078] Training phases 1-3 and the testing phase can be considered as part of an overall method. With respect to training a student GNN for a specific (downstream) task, it will be understood that such training can be performed as part of training phase 2 and / or as a separate training phase 3. Training for a specific task is not required; that is, aspects of the methods disclosed herein can be or include the methods described with respect to training phase 2 (FIG. 6 or FIG. 7) (and the target mismatch in the training phase 2 method can be independent of task-specific loss / mismatch).

[0079] The method of training phase 1 is not required, and aspects of the methods disclosed herein can be or include the methods described with respect to training phase 2 (FIG. 6 or FIG. 7) and can utilize an already trained teacher GNN.

[0080] A testing phase is also not required, i.e., aspects of the methods disclosed herein may be or include the methods described with respect to training phase 2 (FIG. 6 or FIG. 7).

[0081] Fig. 9 is a schematic diagram illustrating a training process according to an example implementation. The training process may correspond to the training process described with respect to Fig. 4 and / or Fig. 6 (training phase 2) and / or Fig. 7. Some example dimensions are shown in Fig. 9. These dimensions are for ease of understanding and are not exhaustive or limiting.

[0082] As shown, a student GNN 10 (to be trained) is used to extract a first pair of representations from a pair of training graphs (Input Graph 1 and Input Graph 2), and a teacher GNN is used to extract a second pair of representations from the pair of training graphs.

[0083] In this implementation example, the L2 distance between the second representations is calculated. Given a second representation corresponding to input graph 1, which may be referred to herein as representation 1, a pooling operation is performed for each hidden layer of representation 1. For example, max pooling may be used (as well as average pooling or hierarchical pooling). In this example, there are five layers, but there may be any number of layers. The number of layers does not have to be the same as the number of layers of the corresponding teacher GNN.

[0084] As shown in Figure 9 (top right), the dimension of each node embedded in each layer is 1 × 256 in this example. After the pooling operation, a vector of dimension 1 × 256 is generated for each hidden layer of Representation 1. Based on the vector generated by the pooling operation for the hidden layers of Representation 1, concatenation is performed to generate a vector of dimension 5 × 256 (since there are five layers in this example).

[0085] As shown in FIG. 9, corresponding processing is performed based on a second representation of input graph 2 (which can be called representation 2), and a concatenated representation of dimensions 5×256 is generated corresponding to representation 2.

[0086] The perceptual distance between these concatenated representations is calculated as the perceptual distance between the second representations. To calculate the perceptual distance between the concatenated representations, the L2 distance can be calculated. Similarly, the L1 distance or the inverse of the dot product of the two vectors can be calculated. The L2 distance is sometimes called the squared Euclidean distance or the L2-squared distance and involves calculating the distance between two vectors by taking the sum of the squared vector values. This can be thought of as calculating the L2 distance between the second representations (indeed, as mentioned above).

[0087] The discrepancy between the first representations is calculated (referred to in FIG. 9 as “Student Network Representation Level Distance”). Although not shown in FIG. 9, the discrepancy may be or may include L2 distance, and may be calculated along the lines of the above description of perceptual distance calculation. The discrepancy is compared to a loss based at least in part on the perceptual distance between the second representations. As shown in FIG. 9, the first representation is also compared to at least one other loss term (referred to in FIG. 9 as “Other Unsupervised Loss (General or Task-Specific)”). This comparison may utilize the discrepancy between the first representations, or the comparison may be a direct comparison of one or both of the first representations with a target representation corresponding to the task for which the student GNN is trained, for example.

[0088] Rather than performing the comparison with the two at least one loss term separately, the discrepancy between the first representations may be compared to a loss that includes a term based on the perceptual distance between the second representations, and / or at least one term based on a task-specific loss, and / or at least one term based on a general loss.

[0089] As indicated by the dots in Figure 9, multiple teacher GNNs may be used to generate multiple pairs of second representations, respectively. The above-described process for calculating the perceptual distance may be performed for each pair of second representations. The overall perceptual distance on which the loss term is based (at least in part) may include an average (i.e., mean) of the perceptual distances corresponding to each pair of second representations.

[0090] Figure 10 is a schematic diagram showing two examples illustrating the difference and relative usefulness of edit distance and perceptual distance. In these examples, perceptual distance was calculated using a teacher network trained using the D-SLA method.

[0091] In Example 1, three molecules a-c are shown. The differences between the three molecules are highlighted in each case with dotted boxes. The edit distance between molecules a and b is 2, and the perceptual distance is 7.87. The edit distance between molecules b and c is 2, and the perceptual distance is 3.23. All three molecules a-c belong to the same class (positive class). Example 1 helps to show that the perceptual distance between the first pair of molecules can be significantly different from the perceptual distance between the second pair of molecules, even though the edit distance between the first pair of molecules is the same as the edit distance between the second pair of molecules. Thus, it will be appreciated that edit distance may not effectively capture semantic differences between graphs.

[0092] In Example 2, two molecules a and b are shown. The difference between the molecules is highlighted with a dashed box. Molecules a and b belong to different classes and the perceptual distance between them is 10.68, but the edit distance between them is 1. Example 2 helps to illustrate that even when molecules have a large perceptual distance and belong to different classes, the edit distance can be small.

[0093] Furthermore, when considering Example 1 together with Example 2, it will be seen that the edit distance of Example 1 is greater than the edit distance of Example 2, even though the molecules in Example 1 belong to the same class and the molecules in Example 2 belong to a different class, and the perceptual distance of Example 1 is less than the perceptual distance of Example 2. Together, these examples help to illustrate that edit distance may not capture semantic differences between graphs.

[0094] The loss / error in the above training process for training the student GNN (including training phase 2) can include a term based on the edit distance between the pair of training graphs. In this way, the training of the student GNN is based on both structural and semantic differences between the training graphs of the pair of training graphs. The perceptual distance in such a training method can be considered to act as a regularization, for example, to provide a better notion of the semantic differences between the training graphs. For example, a loss term based on the perceptual distance can include a regularization term that penalizes (increases the loss) based on the size of the perceptual distance. In this way, it can be considered that if the teacher GNN is "incorrect" (i.e., the perceptual distance is very different from the edit distance), the student GNN can "choose" to learn from the raw data instead, which may result in a better graph representation-extraction.

[0095] In an implementation where the training graphs for an iteration include a first pair of training graphs that include a reference graph and its perturbed variation, and a second pair of training graphs that include the reference graph and another graph that is not the perturbed variation, the loss can be calculated as follows:

[0096] The perceptual distance between the second representations of the first pair of graphs is called positive perceptual distance, and the perceptual distance between the second representations of the second pair of graphs is called negative perceptual distance. Similarly, the discrepancy between the first representations of the first pair of graphs is called positive discrepancy, and the discrepancy between the first representations of the second pair of graphs is called negative discrepancy.

[0097] The difference between the positive and negative discrepancies is compared to the difference between the positive and negative perceptual distances. There is a user-defined threshold for the difference between the positive and negative discrepancies, and if the difference between the perceptual distances is below this threshold, the student network can be thought of as using the raw data and learning from it instead of the second (teacher) representation. The lower bound defined by the perceptual distance can be thought of as creating a dynamic error / loss margin. Furthermore, the difference between the positive and negative perceptual distances acts as a lower bound, and the weight adjustment step forces the difference between the positive and negative discrepancies to at least exceed this lower bound.

[0098] It will be appreciated that this implementation is somewhat in line with symmetric or triplet loss schemes in ML training. Triplet loss is a loss function in which a reference input (called an anchor) is compared to a matching input (called a positive) and a mismatching input (called a negative). The distance from the anchor to the positive is minimized, and the distance from the anchor to the negative input is maximized. Alternatively, a modified version of the Normalized Temperature-Scaled Cross-Entropy (NT-Xent) loss can be applied as part of the comparison step by updating the similarity measures of the positive and negative pairs using a threshold margin.

[0099] It will be appreciated that the comparison of the discrepancy to the loss / error can be performed in many ways as disclosed herein.

[0100] FIG. 11 shows a table. The table includes results (in the form of prediction accuracy) of applying three known methods and a method according to an embodiment of the present invention (the "proposed method") to eight datasets. The known methods are ContextPred (Hu, Weihua, et al. "Strategies for pre-training graph neural networks." arXiv preprint arXiv:1905.12265 (2019)), GraphCL (You, Yuning, et al. "Graph contrastive learning with augmentations." Advances in neural information processing systems 33 (2020): 5812-5823), and D-SLA (Kim, Dongki, Jinheon Baek, and Sung Ju Hwang. "Graph self-supervised learning with accurate discrepancy learning." arXiv preprint arXiv:2202.02989 (2022)). The datasets are BBBP, ClinTox, MUV, HIV, BACE, SIDER, Tox21, and ToxCast (Wu, Zhenqin, et al. "MoleculeNet: a benchmark for molecular machine learning." Chemical science 9.2 (2018): 513-530).

[0101] The proposed method involves training a teacher model using the D-SLA method on the ZINC dataset (Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11):2324-2337, 2015), a large-scale dataset containing 2 million unannotated data points. Then, a student GNN was trained following the above training process using the trained teacher model as the teacher GNN (i.e., as shown in Figures 4 / 6 (Phase 2) / 7 / 9). The trained student GNN was then fine-tuned individually using eight datasets: BBBP, ClinTox, MUV, HIV, BACE, SIDER, Tox21, and ToxCast.

[0102] The trained and fine-tuned student GNNs were then used to perform a classification task (a molecular property prediction task - Wu, Zhenqin, et al. "MoleculeNet: a benchmark for molecular machine learning." Chemical science 9.2 (2018): 513-530)) and an AUROC score was calculated in each case.

[0103] Other methods (ContextPred, GraphCL, and D-SLA) were used to train / fine-tune GNNs using the ZINC dataset and the eight other datasets, and these trained GNNs were then used to perform the same classification task. The corresponding AUROC scores were calculated.

[0104] The AUROC scores of the known and proposed methods are shown in the table of FIG. 11. The average score is also shown for each method. It can be seen that the average score of the proposed method is higher than any of the known methods, and the scores for each dataset are also higher than any of the known methods. This shows the advantages of the method proposed herein (e.g., improved representations of graphs trained according to the method proposed herein) compared to known methods of training GNNs for representation extraction. That is, the graph representations extracted by the GNNs trained according to the method disclosed herein can capture more information (semantically and / or structurally) from the graph than other methods. The method proposed herein is believed to be advantageous, for example, for the process of knowledge distillation.

[0105] FIG. 12 shows two implementation examples A and B of the method for training a GNN disclosed herein. In A, a student GNN 10 trained according to the method disclosed herein is used for graph classification and prediction of chemical properties of molecules (e.g., unknown molecules). That is, the student GNN 10 extracts a representation from a graph representing the unknown molecule, and based on the representation (e.g., based on a Dense layer), a prediction is made as to whether a certain property is likely to be possessed by the (unknown) molecule. It will be understood that using a GNN trained according to the method disclosed herein that can extract more (useful) information from a graph will lead to an improvement in property prediction tasks (including extracting a representation from a graph representing a molecule).

[0106] In B, the student GNN 10 is used in a link prediction task. Depending on what the graph represents, it will be understood that link prediction can be implemented, for example, in disease outbreak detection, in social network systems for controlling user privacy or for detecting spam emails, or in navigation systems for finding efficient routes based on current traffic patterns. The student GNN 10 extracts a representation of the graph, and then, based on that representation (e.g., based on a Dense layer), a prediction is made as to whether a particular further link or links exist in the graph. It will be understood that using a GNN trained according to the methods disclosed herein that can extract more (useful) information from the graph will lead to an improvement in the link prediction task (including extracting a representation from the graph).

[0107] In A and B, the student GNN 10 has been trained for a particular task, for example during training according to the methods disclosed herein and / or in a separate training step subsequent to training according to the methods disclosed herein.

[0108] Figure 13 shows another example of the method disclosed herein, which may be called drug discovery. This may be considered as an extension of A in Figure 12. Here, a student GNN 10 trained as disclosed herein is used to predict properties of an (unknown) molecule based on a graph showing the molecule, and then based on these predicted properties it is decided which candidates will be selected for wet-lab testing. The decision will of course depend on the specific requirements and objectives of the method (i.e. what properties are desired). The decision may be considered to be based on the similarity of the (unknown) molecule with a target molecule or group of target molecules, or based on a target property or group of target properties. The unknown molecule may be AI-generated or human (expert) generated.

[0109] FIG. 14 shows another embodiment of the method disclosed herein, which may be called a recommendation system. This may be considered as an extension / use of B in FIG. 12. Here, the graph from which the representation is extracted by the student GNN 10 (trained as disclosed herein) is a graph representing users and products, as well as the users' interests in the products. As shown, the connections between users are based on social networks, and the connections between products are based on the similarity between the products. The predicted links are between nodes representing users and nodes representing products, and are used to indicate the likelihood that a user (user 3) is interested in a product (product 2). The predictions may be taken into account based on the similarity of the user to other users, and the similarity of product 2 to other products, as well as which users are interested in which products. It will be appreciated that the prediction of the links may be more accurate if the student GNN 10 is able to extract better representations (i.e., representations with better (semantic and structural) information).

[0110] A "task" as used herein can include a spam detection task. That is, a GNN trained as disclosed herein can be used in a spam detection task. In a spam detection task, a graph (input graph) is obtained by considering available emails / messages as nodes of the graph. Here, edges of the graph are generated by obtaining similarity measures between various emails / messages. The task is to predict the labels of unknown nodes. Thus, during inference, this can be seen as a node classification task. For example, emails / messages for which the "spam" label is predicted can be flagged / detected. A GNN / model that can extract a rich representation of nodes is useful for node classification to detect spam.

[0111] A "task" as used herein can include a disease outbreak detection task. That is, a GNN trained as disclosed herein can be used in a disease outbreak detection task. In disease outbreak detection, cities / regions / areas are considered as nodes in an input graph, and edges represent the indicators of connectivity between cities / regions / areas. The task is to predict whether a given city has a disease outbreak (i.e., whether the disease will spread to that city) based on the links and the spread of the disease in other cities. This task can be seen as a node classification task.

[0112] A "task" as used herein may include, for example, a pathfinding task in which a trained student GNN is used to predict links between nodes in a graph representing a node network.

[0113] Extracting a representation of a graph can be referred to as generating a graph embedding, or graph embedding (or node embedding), or representation learning. Extracting a representation of a graph can be thought of as including extracting a representation for each node in the graph, and combining these node representations (or node-level representations) to obtain a representation of the graph. A node embedding or node representation can include information about a node about some or all of the other nodes in the graph and the connections to (and between) them.

[0114] The GNN may include layers that may be referred to as message passing layers, responsible for combining node and edge information into node embeddings. This process may be referred to as graph convolution. For example, a node embedding may include information about the node and other nodes (and edges) in its neighborhood. It will be understood that there are many specific ways in which a representation may be extracted from a graph, and the specific method used is not critical to the methods disclosed herein. The methods disclosed herein relate to training a student GNN as described above, and the specific embedding method may vary.

[0115] As mentioned above, considering the conventional methods, prediction-based methods (e.g., comparison method 1) generate representations by learning contextual relationships by predicting local structures such as nodes, edges, and sub-graphical features including subgraphs, contrastive learning-based methods (e.g., comparison method 2) generate representations that attract (i.e., make more similar) the representations of the original graph and its perturbed changes and reject (i.e., make less similar) the representations of other graph inputs (different graphs), and existing disagreement-based self-supervised learning methods (e.g., comparison method 3) incorporate the edit distance between the original graph and its perturbations and introduce the concept of disagreement into the learned representations.

[0116] In summary, some of the problems with the traditional approach are: Comparison method 1: Does not capture the overall structure of the graph. Comparison method 2: Perturbations in the discrete space, such as editing edges / nodes, can significantly change the properties of the graph. Comparison Method 3: The edit distance between the original graph and its perturbed variation cannot capture the exact semantic differences between different graphs.

[0117] Aspects disclosed herein can include that given two unlabeled graphs, calculate their perceptual distance using a hidden representation of a pre-trained (teacher) network, and use those perceptual distances along with other loss functions to train another network (the student GNN) to obtain a final graph representation for downstream tasks.

[0118] The embodiments disclosed herein can capture the appropriate mismatch between different changes of the perturbed graph to obtain better discriminative representations for downstream tasks. The embodiments disclosed herein can achieve better performance in various downstream tasks with limited labeled data (graph classification, feature prediction, link prediction, etc.), as shown, for example, in FIG.

[0119] In other words, in conventional methods, hand-chosen loss functions are merely proxies for the desired objectives for training existing graph representation learning models. These losses often enforce only limited semantic information on local graph structures (e.g., predictive learning) without quantifying the notion of distance (e.g., contrastive learning) or focus only on structural information with noisy semantic signals (e.g., D-SLA). In some implementations disclosed herein, the student GNN can be considered to learn not only from edit distance but also from other loss functions to capture both semantic and structural differences of different graphs. Disclosed herein is a sample-dependent perceptual loss that captures both semantic and structural differences between graphs from the perspective of the teacher network. As mentioned above, perceptual distance can be considered to act as a regularizer to provide a better notion of semantic distance between graphs. The aspects disclosed herein can be thought of as using a better "ground truth" for the notion of semantic and structural distance between graphs by incorporating perceptual distance in the loss function along with other losses for training, so that the student GNN can learn a regularized notion of distance between graphs to generate better representations.

[0120] The objective of the aspects disclosed herein can be considered to improve the performance of downstream tasks (e.g., graph classification, node classification, link prediction) by learning a representation of the graph with accurate discrepancies using self-supervised perceptual distances obtained from a more discriminatory teacher network as a proxy for human intelligence. The proposed training methodology is general in that it can be applied to graphs regardless of the number of nodes or edges, other structural details, or domain / use case.

[0121] Aspects disclosed herein may include a method for learning an effective representation of a graph with accurate discrepancies by introducing a self-supervised perceptual loss using a teacher network to improve downstream tasks with limited supervision of labeled data. The teacher network's recognition provides the semantic difference between two different graphs.

[0122] The graphs disclosed herein may be stored (in memory) in the form of linked data nodes.

[0123] 15 is a block diagram of an information processing appliance 10 or computing device 10, such as a data storage server, that may be used to embody the present invention, to perform some or all of the operations of the methods embodying the present invention, and to perform some or all of the tasks of the apparatus of the embodiments. The computing device 10 may be used to implement any of the method steps described above, for example any of steps S11-S17, S20-S29, S51-S57, S31-S37, and S41-S43.

[0124] Computing device 10 includes a processor 993 and a memory 994. Optionally, the computing device also includes a network interface 997 for communicating with other such computing devices, such as other computing devices of embodiments of the present invention. Optionally, the computing device also includes one or more input mechanisms, such as a keyboard and mouse 996, and a display unit, such as one or more monitors 995. These elements may facilitate user operation. The components may be connected to each other via a bus 992.

[0125] The memory 994 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data that are accessible by a computer (e.g., one or more processors) to cause it to perform one or more functions or operations. For example, computer-executable instructions may include instructions for implementing the methods disclosed herein, or any method disclosed herein, such as steps S11-S17, S20-S29, S51-S57, S31-S37, and S41-S43. The term "computer-readable storage medium" may include any medium capable of storing, encoding, or retaining a set of instructions for execution by a machine and causing a machine to perform any one or more of the method steps of the present disclosure, and thus the term "computer-readable storage medium" includes, but is not limited to, solid-state memory, optical media, and magnetic media. For example, but not by way of limitation, the term "computer-readable storage medium" may include solid-state memory, optical media, and magnetic media. By way of example, and without limitation, such computer-readable media may include non-transitory computer-readable media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices).

[0126] The processor 993 is configured to control the computing device and perform processing operations such as executing computer program code stored in the memory 994 to perform any of the method steps described herein. The memory 994 can store data read and written by the processor 993, and can store the GNN and / or graph and / or image and / or weights and / or error amounts and / or target discrepancies and / or other data described above, and / or programs for performing any of the method steps described above. As referred to herein, a processor may include one or more general-purpose processing devices such as a microprocessor, a central processing unit, etc. The processor may include a processor that implements a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or other instruction set or processor that implements a combination of instruction sets. The processor may also include one or more special-purpose processing devices such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), a network processor, etc. In one or more embodiments, the processor is configured to execute the operations and instructions for performing the operations discussed herein. The processor 993 may be considered to include any of the modules described above. Any operation described to be performed by a module may be implemented as a method by a computer, such as the processor 993.

[0127] The display unit 995 may display representations of data stored by the computing device, such as graphs and / or predictions based thereon and / or GUI windows and / or interactive representations that allow a user to interact with the device 10, for example by drag-and-drop or selection interactions, and / or any other output described above, and may display cursors and dialog boxes and screens that allow interaction between a user and programs and data stored on the computing device. The input mechanism 996 may enable a user to input data and instructions into the computing device, such as allowing a user to enter any of the user inputs described above.

[0128] The network interface (network I / F) 997 can be connected to a network such as the Internet and can be connected to other such computing devices via the network. The network I / F 997 can control input and output data to and from other devices via the network. Other peripheral devices such as microphones, speakers, printers, power supply units, fans, cases, scanners, tracker balls, etc. may also be included in the computing device.

[0129] A method embodying the present invention may be performed on a computing device / appliance 10 as shown in FIG. 15. Such a computing device need not have all the components shown in FIG. 15, but may be comprised of a subset of these components. For example, the device 10 includes a processor 993 and a memory 994 coupled to the processor 993. Alternatively, the device 10 includes a processor 993, a memory 994 coupled to the processor 993, and a display 995. A method embodying the present invention may be performed by a single computing device in communication with one or more data storage servers over a network. The computing device may be a data storage device that stores at least a portion of the data.

[0130] Methods embodying the invention may be performed by a plurality of computing devices operating in cooperation with one another, one or more of which may be data storage servers that store at least a portion of the data.

[0131] The invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can be implemented as a computer program or computer program product, i.e. a computer program embodied in a non-transitory information carrier, for example a machine-readable storage device or a propagated signal, for execution by or controlling the operation of one or more hardware modules.

[0132] A computer program may be in the form of a stand-alone program, computer program portions, or multiple computer programs, may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0133] The steps of the methods of the present invention may be performed by one or more programmable processors executing computer programs to perform the functions of the present invention by operating on input data and generating output. The apparatus of the present invention may be implemented as programmed hardware or as special purpose logic circuitry including Field Programmable Gate Arrays (FPGAs) or Application-Specific Integrated Circuits (ASICs), etc.

[0134] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memories that store instructions and data.

[0135] The above-described embodiments of the invention may be advantageously used independently of any other embodiment or in any feasible combination with one or more other embodiments of the embodiments.

[0136] The present disclosure extends to the following statements:

[0137] (S1) 1. A computer-implemented method comprising: executing a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student / first graph neural network (GNN); - calculating / measuring / determining a discrepancy / difference between said first representations (of a pair of first representations); extracting a pair of second representations from each pair of training graphs using at least one (trained) teacher / second GNN; - calculating a perceptual distance between said second representations (of a pair of second representations); comparing a target disparity / difference between the first representations based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student / first GNN based on the comparison (between the discrepancy / difference and the target discrepancy / difference) (to bring the discrepancy / difference to or closer to the target discrepancy / difference); 4. A computer-implemented method comprising:

[0138] (S2) The computer-implemented method of S1, wherein the training process includes generating one training graph of the pair of training graphs by performing at least one augmentation on the other training graph of the pair of training graphs.

[0139] (S3) The computer-implemented method of S1, wherein the training process includes generating one of the pair of training graphs by performing a first at least one augmentation on an original graph and generating the other of the pair of training graphs by performing a second at least one augmentation on the original graph, where the first and second at least one augmentations are different.

[0140] (S4) The computer-implemented method of S2 or S3, wherein the expansion includes at least one of the following: deleting at least one node, adding at least one node, deleting at least one edge, and adding at least one edge.

[0141] (S5) A computer-implemented method as described in any of S1 to S4, wherein the step of using the student GNN to extract each of the pairs of first representations from the pair of training graphs includes using two instances of the student GNN having common parameters.

[0142] (S6) The computer-implemented method of any of S1 to S5, wherein the step of using the teacher GNN to extract each of the pairs of second representations from the pair of training graphs includes using two instances of the teacher GNN having common parameters.

[0143] (S7) The computer-implemented method of any of S1 to S6, wherein calculating the disparity between the first representations includes calculating a Euclidean distance (L2 distance) between the first representations.

[0144] (S8) The training process includes: extracting, using each of the plurality of (said) teacher GNNs, a pair of second representations from each of the pair of training graphs; calculating, for each pair of said second representations, a perceptual distance between said second representations of the pair; calculating an average of said plurality of perceptual distances; Including, The computer-implemented method of any of S1-S7, wherein the target mismatch is based at least in part on an average of the plurality of perceptual distances.

[0145] (S9) The computer-implemented method of any of S1 to S8, wherein the step of using the teacher GNN to extract each of the pairs of second representations from the pair of training graphs includes using two instances of the teacher GNN having common parameters.

[0146] (S10) The computer-implemented method of any of S1 to S9, wherein the training process includes training the at least one teacher GNN before using the at least one teacher GNN to extract the second representation pairs.

[0147] (S11) The computer-implemented method of any of S1 to S10, wherein the step of training the at least one teacher GNN comprises using a training method based on contrastive learning and / or a training method based on discrepancy-based self-supervised learning and / or a training method based on prediction or prediction.

[0148] (S12) The computer-implemented method of any of S1 to S11, wherein calculating the perceptual distance between the (said / each pair of) second representations comprises calculating a Euclidean distance (L2 distance) between the second representations.

[0149] (S13) A computer-implemented method according to any of S1 to S12, wherein the step of calculating the perceptual distance between the second representations and / or the disparity between the first representations (of the / each pair) comprises the step of calculating the Euclidean distance (L2 distance) between (normalized) hidden representation vectors of at least one layer in each of the first / second representations.

[0150] (S14) A computer-implemented method as described in any of S7, S12, and S13, wherein the step of calculating the L2 distance includes a step of performing a pooling operation for the at least one layer of the first / second representation of interest for each of the first / second representations to generate a concatenated representation of the first / second representation or the at least one layer.

[0151] (S15) A computer-implemented method according to any of S1 to S14, wherein the step of calculating the perceptual distance between the ( / each pair of) second representations and / or the disparity between the first representations comprises the step of performing, for each of the first / second representations, a pooling operation on each / at least one layer of the first / second representation of interest to generate a concatenated representation of the first / second representations.

[0152] (S16) The computer-implemented method of any of S1 to S15, wherein the step of calculating the perceptual distance between the (pair of) second representations and / or the disparity between the first representations comprises the steps of: for each of the first / second representations, performing a pooling operation on each / at least one layer of the first / second representations of interest to generate a concatenated representation of the first / second representations; and calculating an L2 distance between the generated concatenated representations.

[0153] (S17) the pair being a first pair including a reference training graph and an extended version of the reference training graph, and the training process includes: performing the steps of extracting the first and second representations and calculating the disparity and perceptual distance based on a second pair of training graphs, the second pair of training graphs including the reference graph and another reference graph (which is not an extended version of the reference graph); comparing the mismatch to the target mismatch comprises comparing a first (absolute) difference between the mismatch corresponding to the first and second pairs with a second (absolute) difference between the perceptual distance corresponding to the first and second pairs; The computer-implemented method of any of S1-S16, wherein adjusting the at least one weight comprises adjusting the at least one weight based on a comparison between the first and second differences.

[0154] (S18) The computer-implemented method of S17, wherein the adjustment is to make the first difference at least larger than the second difference.

[0155] (S19) The computer-implemented method of S17 or S18, wherein if the second difference is less than a user-defined threshold, the adjustment is to make the first difference at least greater than the user-defined threshold.

[0156] (S20) The computer-implemented method of any of S1-S19, wherein the goal mismatch is based at least in part on (a graph representation-extraction) task (a difference between training graphs according to use of the student GNN in the task or a task-based goal mismatch between the first representations).

[0157] (S21) The computer-implemented method of any of S1 to S20, wherein the goal mismatch is based on task-based terms based on a task-based goal mismatch between the first representations according to (the use of the student GNN in) a (graph representation-extraction) task.

[0158] (S22) The training process includes comparing the mismatch to a further target mismatch based on a graph representation extraction task; The computer-implemented method of any of S1 to S21, wherein adjusting at least one weight of the student GNN includes adjusting the at least one weight based on a comparison of the mismatch and the further target mismatch.

[0159] (S23) The training process includes a step of comparing the mismatch with a task-based goal mismatch between the first representations according to a graph representation-extraction task; A computer-implemented method as described in any of S1 to S21, wherein the step of adjusting at least one weight of the student GNN includes a step of adjusting the at least one weight based on a comparison of the mismatch and the further target mismatch.

[0160] (S24) the training process includes comparing the discrepancy with a further target discrepancy based on differences between the training graphs according to the graph representation-extraction task; The computer-implemented method of any of S1 to S21, wherein adjusting at least one weight of the student GNN includes adjusting the at least one weight based on a comparison of the mismatch and the further target mismatch.

[0161] (S25) The computer-implemented method of any of S1 to S24, wherein comparing the mismatch between the first representations to a target mismatch comprises calculating an error / loss between the mismatch and the target mismatch.

[0162] (S26) The computer-implemented method of any of S1-S25, wherein the training process includes comparing at least one of the first representations with a target representation based on a (graph representation-extraction) task, and adjusting at least one weight of the student GNN based on the comparison.

[0163] (S27) The computer-implemented method of any of S1 to S26, comprising, after the training process, performing a task-based training process (of graph representation-extraction) to train the student GNN on the (graph representation-extraction) task.

[0164] (S28) The (Graph Representation-Extraction) task is as follows: Extracting graph representations from graphs that represent molecules (predicting molecular properties), Extracting a graphical representation from the graph that represents the user's interactions with the product (predicting (at least one link / relationship indicating) the user's level of interest in the product); Extracting graph representations from disease location graphs (to predict / detect disease occurrence), Extracting a graph representation from a graph representing (road) traffic patterns (finding a route), Extracting a graphical representation from the graph representing the emails and messages (classifying / detecting at least one spam email or message); 20. The computer-implemented method of claim 27, further comprising:

[0165] (S29) The tasks include: Predicting links in a graph representing users, products and the interactions between them to predict whether a user will be interested in a product; Classifying the graph representing the molecule (indicating whether it is useful for further experimental (wet-lab) testing), Classifying nodes in a graph, where the nodes represent emails and / or messages and the edges represent similarities between the emails and / or messages, the classification including classification as being spam or not spam; classifying nodes in the graph, the nodes representing geographic regions and the links representing measures of connectivity between associated geographic regions, the classification including as a geographic region having an outbreak of disease or a geographic region not having an outbreak of disease; The computer-implemented method of S27, comprising any one of:

[0166] (S30) The computer-implemented method of any of S1-S29, wherein the target mismatch is based in part on the difference (edit distance) between pairs of the training graphs of interest.

[0167] (S31) The computer-implemented method includes, after the training process, performing a (graph representation-extraction) task-based training process to train the student GNN on a (graph representation-extraction) task, the (graph representation-extraction) task being: Extracting graph representations from graphs that represent molecules (predicting molecular properties), Extracting a graphical representation from the graph that represents the user's interactions with the product (predicting (at least one link / relationship indicating) the user's level of interest in the product); Extracting graph representations from disease location graphs (to predict / detect disease occurrence), Extracting a graph representation from a graph representing (road) traffic patterns (finding a route), Extracting a graphical representation from the graph representing the emails and messages (classifying / detecting at least one spam email or message); The computer-implemented method according to any one of S1 to S30, comprising:

[0168] (S32) The training process includes: extracting a group of first representations from a group of training graphs (including pairs of training graphs) using the student GNN; calculating a disparity between the first representations for each pair of first representations in the group of first representations (including the pair of first representations); extracting a group of second representations from the group of training graphs (including the pair of training graphs) respectively using at least one teacher GNN; calculating a perceptual distance between each pair of second representations in the group of second representations (including the pair of second representations); For each pair of training graphs, comparing a disparity between the first representations of the corresponding pair of the first representations to a target disparity based at least in part on a perceptual distance between the second representations of the corresponding pair of the second representations; adjusting at least one weight of the student GNN based on the comparison; The computer-implemented method of any one of S1 to S31, comprising:

[0169] (S33) The computer-implemented method of any of S1 to S32, comprising repeating / iterating the training process multiple times, each time using a different pair / group of training graphs.

[0170] (S34) The computer-implemented method of any of S1 to S33, comprising iterating / repeat- ing the training process until the error converges or falls below an error threshold (an iteration).

[0171] (S35) The computer-implemented method of any of S1 to S33, comprising repeating / iterating the training process until the error falls below an error threshold (for a predetermined number of consecutive iterations).

[0172] (S36) The computer-implemented method according to any of S1 to S33, comprising repeating / iterating the training process a predetermined number of times.

[0173] (S37) The computer-implemented method includes (after training) using the (trained) student graph to perform a (graph representation-extraction) task, the (graph representation-extraction) task being: Extracting graph representations from graphs that represent molecules (predicting molecular properties), Extracting a graphical representation from the graph that represents the user's interactions with the product (predicting (at least one link / relationship indicating) the user's level of interest in the product); Extracting graph representations from disease location graphs (to predict / detect disease occurrence), Extracting a graph representation from a graph representing (road) traffic patterns (finding a route), Extracting a graphical representation from the graph representing the emails and messages (classifying / detecting at least one spam email or message); The computer-implemented method of any one of S1 to S36,

[0174] (S38) A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising the step of carrying out a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student / first graph neural network (GNN); - calculating / measuring / determining a discrepancy / difference between said first representations (of a pair of first representations); extracting a pair of second representations from each pair of training graphs using at least one (trained) teacher / second GNN; - calculating a perceptual distance between said second representations (of a pair of second representations); comparing a target disparity / difference between the first representations based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student / first GNN based on the comparison (between the discrepancy / difference and the target discrepancy / difference) (to bring the discrepancy / difference to or closer to the target discrepancy / difference); A computer program comprising:

[0175] (S39) 1. A computer processing device comprising: a memory; and a processor coupled to the memory, the processor configured to execute a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student / first graph neural network (GNN); - calculating / measuring / determining a discrepancy / difference between said first representations (of a pair of first representations); extracting a pair of second representations from each pair of training graphs using at least one (trained) teacher / second GNN; - calculating a perceptual distance between said second representations (of a pair of second representations); comparing a target disparity / difference between the first representations based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student / first GNN based on the comparison (between the discrepancy / difference and the target discrepancy / difference) (to bring the discrepancy / difference to or closer to the target discrepancy / difference); (c) a computer processing device;

[0176] In addition to the above-described embodiment, the following supplementary notes are further disclosed. (Supplementary Note 1) A computer-implemented method comprising the steps of: executing a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student graph neural network (GNN); calculating a disparity between said first representations; extracting a pair of second representations from each pair of training graphs using at least one teacher GNN; calculating a perceptual distance between said second representations; comparing a disparity between the first representations to a target disparity based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student GNN based on the comparison; 4. A computer-implemented method comprising: (Appendix 2) The training process includes: extracting a pair of second representations from the pair of training graphs using each of a plurality of teacher GNNs; calculating, for each pair of said second representations, a perceptual distance between said second representations of the pair; calculating an average of a plurality of said perceptual distances; Including, 2. The computer-implemented method of claim 1, wherein the target mismatch is based at least in part on an average of a plurality of the perceptual distances. (Supplementary Note 3) The computer-implemented method of Supplementary Note 1 or 2, wherein the training process further includes a step of training the at least one teacher GNN before using the at least one teacher GNN to extract the second representation pair. (Supplementary Note 4) The computer-implemented method of any one of Supplementary Notes 1 to 2, wherein calculating a perceptual distance between the second representations comprises calculating a Euclidean distance between the second representations. (Supplementary Note 5) The computer-implemented method of any one of Supplementary Notes 1 to 2, wherein the goal mismatch is based in part on a difference between a pair of the training graphs of interest. (Supplementary Note 6) The goal mismatch includes a task-based term based on a task-based goal mismatch between the first representations according to a task; and / or 3. The computer-implemented method of claim 1 or 2, wherein the training process further includes comparing at least one of the first representations to a task-based target representation and adjusting at least one weight of the student GNN based on the comparison. (Supplementary Note 7) The computer-implemented method of any one of Supplementary Notes 1 to 2, further comprising, after the training process, performing a task using the student graph. (Appendix 8) The tasks are as follows: extracting a graph representation from a graph representing the molecule; Extracting a graphical representation from the graph representing the interaction between the user and the product; extracting a graph representation from the graph representing the locations of disease cases; extracting a graphical representation from the graph representing the traffic patterns; Extracting a graphical representation from the graph representing the emails and messages; 7. The computer-implemented method of claim 6, (Supplementary Note 9) A computer program, which when executed on a computer, causes the computer to execute a method including the step of executing a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student graph neural network (GNN); calculating a disparity between said first representations; extracting a pair of second representations from each pair of training graphs using at least one teacher GNN; calculating a perceptual distance between said second representations; comparing a disparity between the first representations to a target disparity based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student GNN based on the comparison; A computer program comprising: (Supplementary Note 10) An information processing device, comprising: a memory; and a processor coupled to the memory, the processor configured to execute a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student graph neural network (GNN); calculating a disparity between said first representations; extracting a pair of second representations from each pair of training graphs using at least one teacher GNN; calculating a perceptual distance between said second representations; comparing a disparity between the first representations to a target disparity based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student GNN based on the comparison; Including, information processing equipment. [Explanation of symbols]

[0177] 11~13 Student GNN 20~23 Teacher GNN

Claims

1. 1. A computer-implemented method comprising: executing a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student graph neural network (GNN); calculating a disparity between the first representations; extracting pairs of second representations from the pairs of training graphs using at least one teacher GNN; calculating a perceptual distance between said second representations; comparing a disparity between the first representations to a target disparity based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student GNN based on the comparison; 4. A computer-implemented method comprising:

2. The training process includes: extracting a pair of second representations from the pair of training graphs using each of a plurality of teacher GNNs; - for each pair of said second representations, calculating a perceptual distance between said second representations of said pair; calculating an average of a plurality of said perceptual distances; Including, The computer-implemented method of claim 1 , wherein the target mismatch is based at least in part on an average of a plurality of the perceptual distances.

3. 3. The computer-implemented method of claim 1, wherein the training process further comprises training the at least one teacher GNN before using the at least one teacher GNN to extract the second representation pairs.

4. 3. The computer-implemented method of claim 1 or 2, wherein calculating a perceptual distance between the second representations comprises calculating a Euclidean distance between the second representations.

5. The computer-implemented method of claim 1 or 2, wherein the target mismatch is based in part on a difference between the pair of training graphs of interest.

6. The goal mismatch includes a task-based term based on a task-based goal mismatch between the first representations according to a task; and / or 3. The computer-implemented method of claim 1, wherein the training process further comprises comparing at least one of the first representations to a task-based target representation and adjusting at least one weight of the student GNN based on the comparison.

7. The computer-implemented method of claim 1 or 2, further comprising, after the training process, performing a task using the student graph.

8. The tasks include: extracting a graph representation from a graph representing the molecule; Extracting a graphical representation from the graph representing the interaction between the user and the product; extracting a graph representation from the graph representing the locations of disease cases; Extracting a graphical representation from the graph representing the traffic patterns; Extracting a graphical representation from the graph representing the emails and messages; The computer-implemented method of claim 6 , comprising:

9. A computer program, which when executed on a computer, causes the computer to carry out a method comprising the step of carrying out a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student graph neural network (GNN); calculating a disparity between the first representations; extracting pairs of second representations from the pairs of training graphs using at least one teacher GNN; calculating a perceptual distance between said second representations; comparing a disparity between the first representations to a target disparity based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student GNN based on the comparison; A computer program comprising:

10. An information processing device comprising a memory and a processor coupled to the memory, the processor configured to execute a training process, the training process comprising: extracting a pair of first representations from a pair of training graphs using a student graph neural network (GNN); calculating a disparity between the first representations; extracting pairs of second representations from the pairs of training graphs using at least one teacher GNN; calculating a perceptual distance between said second representations; comparing a disparity between the first representations to a target disparity based at least in part on the perceptual distance between the second representations; adjusting at least one weight of the student GNN based on the comparison; Information processing equipment.

Images