Cross-domain graph model pre-training method and system based on learnable graph patch

Abstract

The invention discloses a cross-domain graph model pre-training method and system based on learnable graph patches, and belongs to the technical field of graph model training. Obtaining a pre-training graph data set of a multi-source domain; splitting the node feature of each graph data in the pre-training graph data set into a node token set of each channel, wherein the node token set of each channel and the graph structure of the corresponding channel serve as key graph patches of the channel; key graph patches of all channels in the graph data are coded and aggregated to serve as graph data representation; and based on a graph data representation result, introducing a feature shielding recovery task and a graph neighbor context prediction task to pre-train each module, and combining the pre-trained modules to serve as a cross-domain graph model for extracting target domain graph data representation so as to realize a downstream task of target domain graph data. According to the method, the migratable information is extracted from different source domain graph data from the decomposed key graph patch under the condition of no auxiliary information, and the migration performance of a graph model in a multi-domain task is improved.

Description

Technical Field

[0001] The present invention relates to the field of graph model training technology, and in particular to a cross-domain graph model pre-training method and system based on learnable graph patches. Background Art

[0002] In the era of data-driven artificial intelligence, graph data, as a ubiquitous form of unstructured data, is widely used in social networks, paper citations, molecular modeling and other fields. However, graph data in different fields have significant differences in feature distribution, graph structure, etc., resulting in limited performance of existing graph neural networks (GNNs) in cross-domain tasks. Therefore, how to design an efficient and transferable graph representation learning method to enhance the cross-domain adaptability of graph models is an important research direction in the current field of graph learning.

[0003] Existing research mainly relies on domain alignment or generative methods, but these methods usually have too strong assumptions about data distribution and are difficult to adapt to complex and changing practical scenarios. In addition, how to achieve efficient representation transfer while retaining the structured characteristics of graph data remains a technical difficulty. Therefore, there is an urgent need for a new cross-domain graph learning method that can extract domain-agnostic general graph representations in a flexible way and support efficient transfer of multi-domain tasks. Summary of the invention

[0004] In order to solve the problem of limited performance of graph models in cross-domain migration tasks in the prior art, the present invention proposes a pre-training method and system for cross-domain graph models based on learnable graph patches. The cross-domain graph model of the present invention is composed of a graph patch extraction module, a single-channel graph patch encoding module and a graph patch aggregation module. The graph patch partitioning mechanism, the node feature decomposition method and the graph structure learning method based on the attention mechanism are introduced through the learnable graph patch extraction module to construct a universal key graph patch that can adapt to multiple fields, and the key graph patches are encoded and aggregated in combination with the learnable single-channel graph patch encoding module and the graph patch aggregation module to obtain graph data representation, thereby improving the migration performance of the graph model in multi-domain tasks.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention proposes a cross-domain graph model pre-training method based on learnable graph patches, comprising:

[0007] (1) Obtain pre-trained graph datasets from multiple source domains, including two or more of citation graph data, social network graph data, molecular graph data, financial network graph data, power network graph data, recommendation system graph data, and transportation network graph data. Each graph data consists of a node feature matrix and an original adjacency matrix.

[0008] (2) Using the graph patch extraction module, the node features of each graph data in the pre-trained graph dataset are split into node token sets of each channel. A graph structure is established based on the node token sets of each channel. The node token sets of each channel and the graph structure of the corresponding channel are used as the key graph patches of the channel.

[0009] (3) Using the single-channel image patch encoding module to encode the key image patches of each channel in the image data, and then using the image patch aggregation module to aggregate the encoding results of the key image patches of all channels as the image data representation;

[0010] (4) Based on the graph data representation results, the feature masking restoration task and the graph neighbor context prediction task are introduced to pre-train the graph patch extraction module, the single-channel graph patch encoding module and the graph patch aggregation module. The pre-trained module combination is used as a cross-domain graph model for extracting the representation of the target domain graph data to achieve downstream tasks of the target domain graph data.

[0011] As a preferred embodiment of the present invention, the calculation process of the image patch extraction module includes:

[0012] (2-1) Split the node features of each graph data into sliding window-like segments, where the node features are split into node tokens. The node token is used as the The node tokens of each channel have the same dimension;

[0013] (2-2) Construct the graph structure of the corresponding channel based on the node token set of each channel disassembled:

[0014] ;

[0015] in, represents a neural network model for learning graph structures, represents the original adjacency matrix of the graph data, Represents the first The node token set of the channel, Represents the number of nodes in the graph data, Indicated by The generated The node token set of each channel corresponds to the adjacency matrix of the graph structure, which is used to represent a graph structure;

[0016] (2-3) According to the node token set of each channel and the graph structure of its corresponding channel, the key graph patch of each channel is obtained:

[0017] ;

[0018] in, Indicates Keymap patches for each channel.

[0019] As a preferred embodiment of the present invention, the neural network model for learning graph structure is a neural network model based on an attention mechanism, and the calculation process includes:

[0020] The first The attention scores between the node tokens of each channel;

[0021] The attention scores are sparsely distributed through nonlinear functions, and the sparse attention scores between all nodes in the graph data constitute the first The sparse attention adjacency matrix of channels;

[0022] The data of the figure The sparse attention adjacency matrix of the channels is residually connected with the original adjacency matrix to form the The node token set of each channel corresponds to the adjacency matrix of the graph structure.

[0023] As a preferred embodiment of the present invention, the calculation process of the single-channel image patch encoding module includes:

[0024] (3-1) The node token set of the channel and the original adjacency matrix are used as the input of the graph neural network to generate the first Channel-wise patch representation ;

[0025] And, The node token set of the channel is the same as the The adjacency matrix of the graph structure corresponding to the node token set of the channel is used as the input of the graph neural network to generate the first Channel-wise patch representation ;

[0026] (3-2) The two image patch representations are fused to obtain a single-channel image patch encoding. After traversing all channels, the image patch encoding of all channels is obtained.

[0027] As a preferred embodiment of the present invention, the process of fusing two image patch representations is as follows:

[0028] ;

[0029] in, represents a multi-layer perceptron, Indicates splicing, represents a nonlinear function with a range of (0,1), Indicates Channel-wise image patch encoding.

[0030] As a preferred embodiment of the present invention, the graph patch aggregation module adopts a Transformer encoder to obtain a graph data representation by aggregating graph patch encodings of all channels.

[0031] As a preferred embodiment of the present invention, the feature masking restoration task includes:

[0032] Masking some node features of the graph data, and generating a graph data representation based on the masked graph data;

[0033] Using the first decoder to restore the obscured node features according to the graph data representation;

[0034] The feature occlusion restoration loss is calculated by combining the true occlusion result and the restoration result.

[0035] As a preferred embodiment of the present invention, the graph neighbor context prediction task includes:

[0036] Generate graph data representation based on original graph data;

[0037] Using the second decoder to predict the feature mean vector of the p-order neighbor nodes of each node of the graph data according to the graph data representation, p=1,2,…,P; where P represents the set maximum order;

[0038] The graph neighbor context prediction loss is calculated by combining the true results and the predicted results.

[0039] As a preferred embodiment of the present invention, the target domain graph data is molecular graph data, and the downstream task of the target domain graph data is a molecular graph classification task.

[0040] In a second aspect, the present invention proposes a cross-domain graph model pre-training system based on learnable graph patches, which is used for the above-mentioned cross-domain graph model pre-training method based on learnable graph patches.

[0041] The beneficial effects of the present invention are:

[0042] (1) The present invention implements a pre-trained graph model for cross-domain graph structure data. A learnable key graph patch is obtained through a graph patch extraction module, and the original graph data is split into small-scale and domain-agnostic graph patches. The key graph patches are encoded and aggregated by combining a single-channel graph patch encoding module and a graph patch aggregation module to obtain a graph data representation, thereby effectively utilizing graph structure data in multiple source domains.

[0043] (2) The graph patch extraction module of the present invention efficiently realizes the mining of common information between graph data in different domains by splitting node features into node tokens, so that downstream tasks of the target domain graph data can be completed more accurately and reasonably.

[0044] (3) Based on the results of graph data characterization, the present invention introduces the feature masking restoration task and the graph neighbor context prediction task to pre-train the graph patch extraction module, the single-channel graph patch encoding module and the graph patch aggregation module, thereby realizing the extraction of transferable information from different source domain graph data after decomposition of key graph patches without auxiliary information, thereby improving the migration performance of the graph model in multi-domain tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is an overall block diagram of a cross-domain graph model pre-training method based on learnable graph patches shown in an embodiment of the present invention.

[0046] Figure 2 It is a flowchart of a cross-domain graph model pre-training method based on learnable graph patches shown in an embodiment of the present invention.

[0047] Figure 3 It is a schematic diagram of the calculation process of the image patch extraction module shown in an embodiment of the present invention.

[0048] Figure 4 It is a schematic diagram of a method for dividing image patches according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0050] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0051] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0052] The core idea of ​​this invention is to divide the graph data into multiple small-scale and domain-agnostic key graph patches, and optimize the representation ability of the graph patches through joint pre-training. Figure 1The overall block diagram of the cross-domain graph model pre-training method based on learnable graph patches is shown. The present invention collects pre-trained graph datasets from multiple source domains, each of which consists of a node feature matrix and an original adjacency matrix; the node features are split into node tokens of a unified dimension through a graph patch extraction module, and the corresponding graph structure is constructed based on the disassembled node tokens to capture the efficient interaction relationship between nodes and obtain the key graph patches of each channel; the key graph patches of each channel in the graph data are encoded using a single-channel graph patch encoding module, and the encoding results of the key graph patches of all channels are aggregated using a graph patch aggregation module as a graph data representation; the learnable parameters in the model are jointly trained through a multi-task learning method (feature masking restoration task and graph neighbor context prediction task) to enable it to adapt to tasks in different fields.

[0053] like Figure 2 As shown, the cross-domain graph model pre-training method based on learnable graph patches proposed in the present invention mainly includes the following steps:

[0054] S1, obtain pre-trained graph datasets from multiple source domains. Each graph data consists of a node feature matrix and an original adjacency matrix, where the node feature represents the attribute information of each node, and the original adjacency matrix represents the connection relationship between nodes.

[0055] The pre-trained graph dataset of multiple source domains contains at least two domains, for example, a graph dataset consisting of a citation graph network and a social graph network, where:

[0056] In the graph data of the citation graph network, nodes represent papers, node features are paper information, and edges represent citation relationships between papers. Here, the node feature matrix is ​​the matrix composed of paper information, and the original adjacency matrix is ​​the citation relationship matrix between papers.

[0057] In the graph data of the social graph network, nodes represent users, node features are user information, and edges represent the interaction relationship between users. Here, the node feature matrix is ​​the matrix composed of user information, and the original adjacency matrix is ​​the interaction relationship matrix between users.

[0058] In addition, the source domain can also include other areas, such as:

[0059] In the graph data of the molecular graph network, nodes represent atoms, node features are the elements of atoms, and edges represent the connection relationship between atoms; here, the node feature matrix is ​​the matrix composed of atomic elements, and the original adjacency matrix is ​​the connection relationship matrix between atoms.

[0060] In the graph data of financial networks, nodes represent accounts, node features are account information, and edges represent the fund flow relationship between accounts. Here, the node feature matrix is ​​the matrix composed of account information, and the original adjacency matrix is ​​the fund flow relationship matrix between accounts.

[0061] In the graph data of the power network, the nodes represent substations, the node features are the substation specifications, and the edges represent the transmission lines; here, the node feature matrix is ​​the matrix composed of substation specifications, and the original adjacency matrix is ​​the transmission line connection matrix.

[0062] In the graph data of the recommendation system, nodes represent users and products, node features are user information and product information, and edges represent the interaction relationship between users and products (such as purchase and evaluation). Here, the node feature matrix is ​​the matrix composed of user information and product information, and the original adjacency matrix is ​​the interaction relationship matrix between users and products.

[0063] In the graph data of the transportation network, nodes represent transportation nodes (such as stations and intersections), node features are the geographical information of transportation nodes, and edges represent roads; here, the node feature matrix is ​​the matrix composed of the geographical information of transportation nodes, and the original adjacency matrix is ​​the road connection matrix.

[0064] S2, using the graph patch extraction module to split the node features of each graph data in the pre-trained graph dataset into node token sets of each channel, and establish a graph structure based on the node token sets of each channel. The node token sets of each channel and the graph structures of their corresponding channels serve as the key graph patches of the channel.

[0065] In this step, if Figure 3 As shown, an optional implementation is as follows:

[0066] S21, firstly adopt the graph patch partitioning method to divide each graph data in the pre-trained graph dataset into graph patches; split the node features of each graph data into node tokens in a sliding window manner, and split the node features into node tokens in order. The node token is the The node tokens of each channel ensure that the dimensions of the node tokens of each graph data (M and K below) are the same. Therefore, the original node features of different graph data will be decomposed into different node tokens, but all node tokens are in the same latent space, expressed as:

[0067]

[0068]

[0069]

[0070] in, Represents the first nodes, Indicates channels; is the length size of the node token, is the sliding window step size, here , Indicates rounding down. Represents the dimension of the original node features; since the node feature dimensions of graph data in different fields may be different, the dimensions of the node tokens of each graph data can be ensured to be consistent by adjusting the sliding window step size. This decomposition method can effectively avoid the distribution inconsistency problem caused by the direct participation of the original features in cross-domain learning, and lay the foundation for subsequent unified processing. Indicates the data in the figure The node in The node token in a channel is a smaller unit of information in the node token. The node feature matrix representing the graph data, represents the number of node tokens for each node in the data graph, , They are The starting and ending feature positions of the node tokens in each channel. Figure 4 Take node 1 in as an example, the node characteristics are , set the sliding window length to 3 and the step size to 1, and divide node 1 into three node tokens, where , , .

[0071] Traverse all nodes in the graph data and get the first The node token set of a channel is recorded as .

[0072] S22, then build the graph structure of the corresponding channel based on the node tokens of different channels that have been disassembled. The node tokens of each channel will correspond to its own independent graph structure, which can be expressed as:

[0073]

[0074] in, It is a neural network model used to learn graph structures. represents the adjacency matrix of the graph data, Is The corresponding generated The adjacency matrix of the corresponding graph structure of the node tokens of the channels, is the number of nodes in the graph data. In this embodiment, It can be implemented using a variety of neural network models, such as CNN or neural networks based on attention mechanisms.

[0075] The present invention adaptively learns the dynamic graph structure according to the similarity between node tokens. By calculating the interaction relationship between each pair of node tokens in the latent space, a sparse dynamic adjacency matrix is ​​constructed. Each node token is connected only with the most relevant tokens, which not only improves the computational efficiency but also enhances the generalization ability of the model.

[0076] In a specific implementation of the present invention, taking a neural network based on an attention mechanism as an example, the graph structure The calculation process includes:

[0077] First, calculate the first The attention score between the node tokens of each channel :

[0078]

[0079] in, represents the learnable parameters, represents dot product, Represents a multi-layer perceptron used to learn the attention scores of node tokens of the same channel at different nodes , Respectively represent the i-th and j-th nodes The node token for each channel.

[0080] Secondly, the attention score is sparsed through a nonlinear function:

[0081]

[0082] in, represents the sparse attention score, represents the softmax function, express The top k largest node set, Represents the i-th node in the graph data The node token of the channel is the same as the node token of the other nodes. The attention scores between node tokens of channels.

[0083] Combine the original adjacency matrix with residual linking to form an updated adjacency matrix:

[0084]

[0085] Among them, the final generated That is, the collection of node tokens The corresponding graph structure.

[0086] S23, the node token set of all nodes in each channel after the disassembly And its corresponding learned adjacency matrix They are combined together to form a learnable key graph patch, expressed as:

[0087]

[0088] in, Indicates The key graph patch of each channel. The key graph patch is not only suitable for tasks in a single domain, but also can achieve migration optimization in a multi-domain environment. By combining each group of node tokens with its corresponding graph structure, the representation ability of the graph patch is enhanced, and each graph patch contains information about a certain part of the node features and its contextual relationship in the graph structure. At this stage, the present invention pays special attention to the migration adaptability of graph patches between domains, that is, how to make the graph patches generated in different domains perform consistently in the same latent space, significantly reducing the distribution offset between domains. S3, uses a single-channel graph patch encoding module to encode the key graph patches of each channel in the graph data, and then uses a graph patch aggregation module to aggregate the encoding results of the key graph patches of all channels as graph data representation.

[0089] In this step, an optional implementation method is as follows:

[0090] S31, based on and , code Image patch for channels:

[0091]

[0092]

[0093] in, is based on the original adjacency matrix A channel-wise representation of image patches, which takes into account the impact of the original image structure on the encoding of image patches; is based on the updated adjacency matrix Channel-wise image patch representation; It is a graph neural network;

[0094] S32, merge the two parts of the constructed image patches:

[0095]

[0096] in, is a multi-layer perceptron, Indicates splicing, represents a nonlinear function with a range of (0,1), Represents a single-channel image patch encoding. The image patch encoding of all channels is composed of .

[0097] S33, using Transformer-based encoder to aggregate image patches of different channels:

[0098]

[0099]

[0100]

[0101] in, It is a multi-head attention mechanism; It is layer normalization, which is used to provide training stability; It is a feed-forward network; is the pooling layer, which performs pooling in the direction of K image patches; finally, we get As the final graph data representation; They are the intermediate results calculated by layer normalization.

[0102] In order to deal with the possible differences in channel characteristics between data in different fields, a cross-channel information aggregation mechanism is introduced through the Transformer encoder to unify the representation of graph patches from different channels. This method can capture the interaction relationship between cross-channels, thereby improving the applicability of the final graph representation.

[0103] S4, based on the results of graph data characterization, introduce the feature masking restoration task and the graph neighbor context prediction task to pre-train the graph patch extraction module, the single-channel graph patch encoding module and the graph patch aggregation module. In this step, the pre-training method is used to simultaneously train the learnable graph patch extraction module, the single-channel graph patch encoding module and the graph patch aggregation module. Specifically, two loss functions are used for the feature masking restoration task and the graph neighbor context prediction task: and Among them, the reconstruction loss of the feature masking restoration task is used to ensure that the model can retain the important information in the original node features; the loss of the graph neighbor context prediction task is used to strengthen the model's learning ability of local structural relationships. In the process of optimizing the loss function, the present invention pays special attention to the consistency between domains to ensure that the model can adapt to the data of the source domain and the target domain at the same time during the training process.

[0104] Feature masking restoration task: mask some node features of the graph data, generate graph data representation based on the masked graph data; use the first decoder to restore the masked node features based on the graph data representation; combine the actual masking result and the restoration result to calculate the feature masking restoration loss:

[0105]

[0106] in, represents the mask of the j-th dimension feature of the i-th node, The dimension of the original node features, represents the number of masked features, Represents the number of nodes in the graph data, They are the true masking result and the restoration result of the j-th dimension feature of the i-th node. In this task, the model is predicted by masking certain node features, and then the difference between the true value and the predicted value is measured to achieve the effect of training the model. Here, the first decoder can use a multi-layer perceptron.

[0107] Graph neighbor context prediction task: Generate graph data representation based on original graph data; Use the second decoder to predict the feature mean vector of the p-order neighbor nodes of each node in the graph data based on the graph data representation, p=1,2,…,P; where P represents the set maximum order; Calculate the graph neighbor context prediction loss by combining the actual result and the predicted result:

[0108]

[0109] in, They respectively represent the true result and the predicted result of the feature mean vector of the p-order neighboring node of node i. Here, the second decoder can adopt the same structure as the first decoder, such as a multilayer perceptron, and the two decoders are independent of each other.

[0110] The weighted result of the two losses is used as the final loss to update the trainable parameters of the model. The pre-trained modules are combined in practical applications as a cross-domain graph model for extracting representations of target domain graph data to achieve downstream tasks of the target domain graph data. The present invention can effectively improve the migration performance of the graph model and significantly reduce the learning barriers caused by domain differences.

[0111] The target domain referred to in the present invention may be the same as the source domain or different. For example, the target domain may be molecular graph data, and the downstream task of the target domain graph data is a molecular graph classification task, which divides the molecular graph into corresponding categories according to chemical properties. In order to obtain a better characterization result, the pre-trained model can also use the target domain graph data to perform fine-tuning on the corresponding downstream task.

[0112] In order to verify the effect of the present invention, this embodiment uses graph datasets of various source domains for pre-training, and tests on molecular graph datasets and citation graph network datasets to demonstrate the versatility of the proposed method. The molecular graph dataset includes a dataset of 2 million molecules extracted from the ZINC commercial compound database as a pre-training dataset; the molecular graph dataset also includes Tox21, Toxcast, Sider, ClinTox, MUV, HIV, BBBP, BACE, etc. as downstream datasets. The citation graph network dataset includes Arxiv and DBLP, which contain 169,343 and 28,702 articles respectively, of which DBLP is used as a downstream test dataset. The social network dataset uses the Flickr dataset, which contains 105,938 user nodes.

[0113] Evaluation index: The evaluation index method of the present invention is to evaluate according to the effect of the pre-trained model in the downstream. Compared with the method without pre-training, the higher the effect of pre-training, the better the performance. Since the present invention is to construct a cross-domain model, it is necessary to use not only the upstream and downstream data of the same field, but also the upstream and downstream data of different fields for verification. For graph classification problems, the commonly used ROC-AUC index is used.

[0114] Comparison method: The experiment uses the existing commonly used model GIN as the basic model for comparison, and compares different degrees of migration. The test results are shown in Table 1.

[0115] Table 1 Migration effect

[0116]

[0117] As shown in Table 1, the proposed method shows excellent ability in handling feature heterogeneity, and can significantly improve model performance both between pre-training datasets and in the migration process between pre-training and downstream datasets. This heterogeneity is mainly reflected in the distribution differences of multi-domain data and the diversity of task objectives. The proposed method effectively narrows the performance gap caused by these distribution differences by combining pre-training with transfer learning. Without pre-training, the traditional GIN model and the proposed method perform similarly in downstream tasks, reflecting the basic capabilities of basic graph neural networks in end-to-end training scenarios. However, the proposed method shows consistent and significant performance improvement after combining pre-training. For example, on the Sider dataset, after joint pre-training (ZINC and Arxiv datasets), the proposed method achieves an accuracy of 53.71±0.48, which is better than 52.14±0.56 of the GIN model without pre-training; in the HIV dataset, joint pre-training increases the accuracy from 56.58±2.57 without pre-training to 60.23±1.15. The same trend is also reflected in other datasets, which will not be repeated here. These results fully demonstrate that pre-training can provide strong support for transfer learning, especially in tasks with significant differences in data distribution.

[0118] It can also be observed from Table 1 that joint pre-training is more effective than pre-training on a single dataset. For example, on the BACE dataset, the accuracy achieved by the pre-training method based only on the ZINC dataset is 59.74±0.54, while the joint pre-training combined with ZINC and Arxiv further improves the accuracy to 59.75±1.02. This shows that pre-training data from different sources can play a complementary role in the information integration process and significantly enhance the generalization ability of the model. In addition, for some domain data (such as the DBLP dataset), even if there are significant differences in the background between the pre-training datasets, the joint pre-training strategy can still achieve excellent performance, reaching an accuracy of 79.12±1.93, which is better than 74.86±2.26 of the model without pre-training. This performance further demonstrates the applicability and advantages of the method of the present invention in cross-domain migration tasks.

[0119] It is worth noting that while achieving the above-mentioned performance improvement, the method of the present invention also exhibits good robustness. For example, in some extreme scenarios, when there is a significant inconsistency between the feature distribution of pre-training data and downstream data, the present invention can still provide consistent performance improvements. This shows that by optimizing the feature division and structural learning of graph data, the method of the present invention can efficiently capture the domain-agnostic characteristics between data, thereby achieving effective migration in multi-task scenarios. The test results in Table 1 comprehensively demonstrate the wide applicability and superior performance of the cross-domain graph model proposed in the present invention in multiple tasks and multiple fields. Through the joint pre-training and optimization design of data from different fields, the method of the present invention significantly improves the migration ability of the model, lays a solid foundation for the realization of a general graph model, and also provides an important reference for future research on graph model infrastructure.

[0120] Based on the same inventive concept, a cross-domain graph model pre-training system based on learnable graph patches is also provided in this embodiment, and the system includes:

[0121] A pre-trained graph dataset acquisition module, which is used to acquire pre-trained graph datasets of multiple source domains, including two or more of reference graph data, social network graph data, and molecular graph data, each of which consists of a node feature matrix and an original adjacency matrix;

[0122] A graph patch extraction module is used to split the node features of each graph data in the pre-trained graph dataset into node token sets of each channel, and establish a graph structure based on the node token sets of each channel. The node token sets of each channel and the graph structure of the corresponding channel are used as the key graph patches of the channel.

[0123] A single-channel image patch encoding module, which is used to encode the key image patches of each channel in the image data of the encoding module;

[0124] A graph patch aggregation module, which is used to aggregate the encoding results of key graph patches of all channels and use the aggregation results as graph data representation;

[0125] A pre-training module is used to pre-train the graph patch extraction module, the single-channel graph patch encoding module and the graph patch aggregation module based on the graph data representation results. The pre-trained module combination is used as a cross-domain graph model for extracting the representation of the target domain graph data to achieve downstream tasks of the target domain graph data.

[0126] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0127] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0128] The above-mentioned embodiments only express several implementation modes of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A cross-domain graph model pre-training method based on learnable graph patches, characterized in that: include: (1) Obtain pre-trained graph datasets from multiple source domains, including two or more of citation graph data, social network graph data, molecular graph data, financial network graph data, power network graph data, recommendation system graph data, and transportation network graph data. Each graph data consists of a node feature matrix and an original adjacency matrix. (2) Using the graph patch extraction module, the node features of each graph data in the pre-trained graph dataset are split into node token sets of each channel. A graph structure is established based on the node token sets of each channel. The node token sets of each channel and the graph structure of the corresponding channel are used as the key graph patches of the channel. (3) Using the single-channel image patch encoding module to encode the key image patches of each channel in the image data, and then using the image patch aggregation module to aggregate the encoding results of the key image patches of all channels as the image data representation; (4) Based on the graph data representation results, the feature masking restoration task and the graph neighbor context prediction task are introduced to pre-train the graph patch extraction module, the single-channel graph patch encoding module and the graph patch aggregation module. The pre-trained module combination is used as a cross-domain graph model for extracting the representation of the target domain graph data to achieve downstream tasks of the target domain graph data.

2. The cross-domain graph model pre-training method based on learnable graph patches according to claim 1, characterized in that: The calculation process of the image patch extraction module includes: (2-1) Split the node features of each graph data into sliding window-like segments, where the node features are split into node tokens. The node token is used as the The node tokens of each channel have the same dimension; (2-2) Construct the graph structure of the corresponding channel based on the node token set of each channel disassembled: ； in, represents a neural network model for learning graph structures, represents the original adjacency matrix of the graph data, Represents the first The node token set of each channel, Represents the number of nodes in the graph data, Indicated by The generated The node token set of each channel corresponds to the adjacency matrix of the graph structure, which is used to represent a graph structure; (2-3) According to the node token set of each channel and the graph structure of its corresponding channel, the key graph patch of each channel is obtained: ； in, Indicates Keymap patches for each channel.

3. The cross-domain graph model pre-training method based on learnable graph patches according to claim 2, characterized in that: The neural network model for learning graph structure is a neural network model based on the attention mechanism, and the calculation process includes: The first The attention scores between the node tokens of each channel; The attention scores are sparsely distributed through nonlinear functions, and the sparse attention scores between all nodes in the graph data constitute the first The sparse attention adjacency matrix of channels; The data of the figure The sparse attention adjacency matrix of the channels is residually connected with the original adjacency matrix to form the The node token set of each channel corresponds to the adjacency matrix of the graph structure.

4. The cross-domain graph model pre-training method based on learnable graph patches according to claim 1, characterized in that: The calculation process of the single-channel image patch encoding module includes: (3-1) The node token set of the channel and the original adjacency matrix are used as the input of the graph neural network to generate the first Channel-wise patch representation ; And, the The node token set of the channel is the same as the The adjacency matrix of the graph structure corresponding to the node token set of the channel is used as the input of the graph neural network to generate the first Channel-wise patch representation ; (3-2) The two image patch representations are fused to obtain a single-channel image patch encoding. After traversing all channels, the image patch encoding of all channels is obtained.

5. The cross-domain graph model pre-training method based on learnable graph patches according to claim 4, characterized in that: The process of fusing two image patch representations is as follows: ； in, represents a multi-layer perceptron, Indicates splicing, represents a nonlinear function with a range of (0,1), Indicates Channel-wise image patch encoding.

6. The cross-domain graph model pre-training method based on learnable graph patches according to claim 1, characterized in that: The graph patch aggregation module adopts a Transformer encoder to obtain a graph data representation by aggregating graph patch encodings of all channels.

7. The cross-domain graph model pre-training method based on learnable graph patches according to claim 1, characterized in that: The feature masking restoration task includes: Masking some node features of the graph data, and generating a graph data representation based on the masked graph data; Using the first decoder to restore the obscured node features according to the graph data representation; The feature occlusion restoration loss is calculated by combining the true occlusion result and the restoration result.

8. The cross-domain graph model pre-training method based on learnable graph patches according to claim 1, characterized in that: The graph neighbor context prediction task includes: Generate graph data representation based on original graph data; Using the second decoder to predict the feature mean vector of the p-order neighbor nodes of each node of the graph data according to the graph data representation, p=1,2,…,P; where P represents the set maximum order; The graph neighbor context prediction loss is calculated by combining the true results and the predicted results.

9. The cross-domain graph model pre-training method based on learnable graph patches according to claim 1, characterized in that: The target domain graph data is molecular graph data, and the downstream task of the target domain graph data is a molecular graph classification task.

10. A cross-domain graph model pre-training system based on learnable graph patches, used to implement the method of claim 1; characterized in that: The system comprises: A pre-trained graph dataset acquisition module, which is used to acquire pre-trained graph datasets of multiple source domains, including two or more of reference graph data, social network graph data, and molecular graph data, each of which consists of a node feature matrix and an original adjacency matrix; A graph patch extraction module is used to split the node features of each graph data in the pre-trained graph dataset into node token sets of each channel, and establish a graph structure based on the node token sets of each channel. The node token sets of each channel and the graph structure of the corresponding channel are used as the key graph patches of the channel. A single-channel image patch encoding module, which is used to encode the key image patches of each channel in the image data of the encoding module; A graph patch aggregation module, which is used to aggregate the encoding results of key graph patches of all channels and use the aggregation results as graph data representation; A pre-training module is used to pre-train the graph patch extraction module, the single-channel graph patch encoding module and the graph patch aggregation module based on the graph data representation results. The pre-trained module combination is used as a cross-domain graph model for extracting the representation of the target domain graph data to achieve downstream tasks of the target domain graph data.

Images