A method and system for constructing a general pre-training graph structure large model

Abstract

The application discloses a kind of general pre-training graph structure large model construction method and system, belong to graph computing technical field, including: from the sample set of text attribute subgraph and node text attribute contained in multiple source text attribute graph data is constructed;Graph structure encoder is pre-trained using the reversible serialization mode based on Euler path improvement mode, and graph structure representation is obtained;Text feature encoder is used to encode node text attribute into text semantic representation, after graph structure representation and text semantic representation are mapped to the same alignment space by mapping network, based on text semantic representation and graph structure representation, construct joint global representation alignment task, substructure-phrase level local semantic alignment task, and graph to text reconstruction task, train mapping network, obtain general pre-training graph structure large model, so it can effectively bridge the modal gap between graph structure and text semantic, significantly improve the generalization ability and reasoning reliability of model in downstream task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of combining graph computing and artificial intelligence, and particularly relates to a construction method and system of a general pre-training graph structure large model. BACKGROUND

[0002] Graph structure data widely exists in the real world, such as user interaction relationships in social networks, molecular structures and protein interactions in biological medicine, and commodity association networks on e-commerce platforms. These data usually exist in the form of text-attributed graphs (TAGs), that is, the nodes and edges in the graph are accompanied by rich natural language descriptions (such as the title and abstract of a paper, the detailed introduction of a commodity). In recent years, with the breakthrough progress of large language models (LLMs) in the field of natural language processing, graph-LLMs that combine the powerful semantic understanding ability of LLMs with the structure modeling ability of graph neural networks (GNNs) have become the current research hotspot. Inspired by the success of foundation models in natural language processing and computer vision, graph learning has recently entered the era of graph foundation models (GFMs). These models are pre-trained on large-scale graphs using self-supervised objectives, enabling them to learn general representations that can be transferred across tasks and domains. GFMs integrate structural dependencies and semantic content, showing strong zero-shot and few-shot generalization. By decoupling model training from specific tasks, GFMs reduce the dependence on labeled data and specific domain heuristic methods, making the field closer to general graph intelligence.

[0003] However, existing Graph-LLMs still face significant technical bottlenecks in terms of technical implementation and application effect, mainly in the following two core problems:

[0004] First, there is a serious information loss and lack of topological fidelity in graph structure encoding. Existing mainstream methods (such as LLaGA, GraphGPT, etc.) often rely on random walk, neighborhood sampling (such as Neighbor Sampling) or simplified linearization strategy (such as DFS / BFS traversal) to convert the graph structure into a sequence form that can be processed by LLM. These processes often introduce start point bias and randomness of traversal path. More seriously, in order to adapt to the limited context window of LLM, these methods have to truncate the graph or only keep part of the neighbors, resulting in the loss of key global topological structure information such as cycles, bridges and long links in the graph. The sequence received by the model is incomplete, and it is difficult to restore the complete connectivity of the graph based on it, thus performing poorly in tasks that require accurate structure perception (such as judging connectivity, shortest path search, complex subgraph matching), resulting in systematic performance degradation.

[0005] Second, the graph-text representation space is fragmented, and the cross-modal alignment granularity is coarse. After converting the graph into a sequence input for LLM, existing alignment strategies usually only perform coarse-grained contrastive learning (Contrastive Learning) at the full graph / subgraph level and the text paragraph level. This approach ignores the fine-grained correspondence between local substructures in the graph (such as benzene rings in chemical molecules, fully connected subgroups in social networks) and specific phrases in the text (such as aromatic, tight community). Language models naturally lack graph induction bias, resulting in inconsistent semantic mapping of graph features after projection into the language space. This manifests as the model generating semantically coherent but structurally incorrect answers, i.e., incorrectly describing the connection between nodes, or failing to accurately anchor the topological features of the graph to the corresponding text description, severely limiting the model's generalization ability and explainability in complex graph tasks.

[0006] In view of the above technical problems, there is an urgent need for a general graph structure large model construction scheme that can realize lossless serialization of graph structure and implement deep semantic alignment between graph and text at multiple levels, to support more accurate node classification and more reliable graph reasoning applications. SUMMARY

[0007] In view of the above, the purpose of the present application is to provide a general pre-training graph structure large model construction method and system, by introducing a reversible serialization mechanism based on Euler path, providing a high-quality serialization method for graph structure representation, and designing a multi-level alignment strategy including global, local and generative dimensions, effectively bridging the modal gap between graph structure and text semantics, and improving the generalization ability and reasoning reliability of the graph structure large model in downstream tasks.

[0008] To achieve the above invention purpose, the general pre-training graph structure large model construction method provided by the embodiment comprises the following steps:

[0009] After the pre-processing of the multi-source text attribute graph data, a sample set containing text attribute sub-graphs and node text attributes is constructed;

[0010] Each text attribute sub-graph is converted into a continuous one-dimensional graph Token sequence by using a reversible serialization method based on Euler paths; the one-dimensional graph Token sequence is taken as input, and a dispatch mask Token prediction is used as a self-supervised training target to pre-train the graph structure encoder, thereby obtaining a graph structure representation with general topological understanding capability;

[0011] The node text attributes corresponding to the text attribute sub-graphs are encoded into text semantic representations by using a text feature encoder; after the graph structure representation and the text semantic representation are mapped to the same alignment space by a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and the graph structure representation in the same alignment space, and the mapping network is trained by jointly performing a global representation alignment task, a local semantic alignment task at the sub-structure-phrase level, and a graph-to-text reconstruction task, thereby obtaining a general pre-trained graph structure large model composed of the graph structure encoder, the text feature encoder, the mapping network, and a language model introduced for decoding.

[0012] Preferably, each text attribute sub-graph is converted into a continuous one-dimensional graph Token sequence by using a reversible serialization method based on Euler paths, which includes:

[0013] First, connectivity enhancement and Euler characteristic preprocessing and path generation are performed;

[0014] Then, node re-indexing and cycle re-indexing are performed to construct node sequences;

[0015] Finally, attribute attachment is performed based on the node sequences, and node indices, edge types, edge attribute features, and node attribute features are converted into discrete Tokens to form the final one-dimensional graph Token sequence.

[0016] Preferably, the graph structure encoder adopts a bidirectional Transformer structure, and a dispatch mask Token prediction is used as a self-supervised training target to pre-train the graph structure encoder, which includes:

[0017] For the input token sequence, a set of mask positions is first sampled according to a dispatch strategy ， and the tokens at these mask positions are replaced with a special mark [MASK] to obtain a post-mask input sequence; after the graph structure encoder encodes the post-mask input sequence, a prediction head is introduced to predict the mask tokens, thereby obtaining a prediction probability distribution; and a loss function is defined based on the average cross-entropy of the prediction probability distribution at all mask positions to optimize the parameters of the graph structure encoder.

[0018] Preferably, the mapping network comprises a graph feature alignment network and a text feature alignment network, both of which have the same structure and employ a mapping function to map input features to a fixed scale, so as to map the graph structure representation and the text semantic representation to the same alignment space.

[0019] Preferably, the multi-level cross-modal loss comprises a global representation alignment loss corresponding to a global representation alignment task, a local semantic alignment loss corresponding to a substructure-phrase level local semantic alignment task, and a reconstruction loss corresponding to a graph-to-text reconstruction task.

[0020] Preferably, for the global representation alignment task, the reference contrast learning employs an InfoNCE loss to construct the global representation alignment loss which comprises an InfoNCE loss defined in the direction from the graph to the text based on the normalized graph structure representation and the normalized text semantic representation and an InfoNCE loss defined in the direction from the text to the graph .

[0021] Preferably, for the substructure-phrase level local semantic alignment task, the constructed local semantic alignment loss includes a token level matching loss constructed based on the matching degree between the structure token and the text token and further includes an InfoNCE loss defined in the direction from the graph to the text based on the normalized substructure representation and the normalized local text semantic representation and an InfoNCE loss defined in the direction from the text to the graph wherein the substructure representation is the representation of the substructure obtained by dividing the text attribute subgraph and passing through the graph structure encoder and the mapping network, and the local text semantic representation is the representation of the text segment obtained by extracting the node text attribute and passing through the text feature encoder and the mapping network.

[0022] Preferably, for the graph-to-text reconstruction task, the constructed reconstruction loss is the cross entropy of the autoregressive conditional distribution of the true text attribute token sequence output by the text decoder;

[0023] wherein the text decoder takes the graph structure representation mapped by the graph feature alignment network as input, takes the local true text attribute sequence as input, and generates the autoregressive conditional distribution of the true text attribute token sequence based on the input.

[0024] To achieve the above-mentioned purposes of the application, the embodiment further provides a construction system of a general pre-training graph structure large model, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the construction method of the general pre-training graph structure large model.

[0025] To achieve the above-mentioned purposes of the application, the embodiment further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the construction method of the general pre-training graph structure large model.

[0026] Compared with the prior art, the application has at least the following beneficial effects:

[0027] 1. High structural fidelity: To solve the problem that local connection patterns and structural details are easily lost in the serialization process of graph data, the graph serialization method based on Euler path is used to realize full-fidelity encoding of the topology and local connection patterns within the window. Within the limited context window, the connectivity and structural semantics within the subgraph are maximized, effectively alleviating the dependence capture problem within the local receptive field, and providing a better structural perception basis for the graph model.

[0028] 2. Accurate semantic alignment: The multi-level alignment strategy breaks the limitation of single global alignment, and through joint optimization in three dimensions of global, local fine granularity and semantic generation integrity, the model deeply aligns the graph-text representation in the latent space, enhancing the deep understanding and semantic association ability of the graph model for graph-text heterogeneous data. The test accuracy of the graph structure large model combined with graph representation improvement and cross-modal loss alignment on the node classification tasks of Cora, PubMed and OGB-Arxiv is 2.71%, 1.30% and 3.34% higher than that of the existing similar methods.

[0029] 3. Strong generalization ability: Through pre-training with mixed multi-source heterogeneous graph data and combined with high-fidelity graph structure serialization encoding method, the model can be seamlessly transferred to various downstream tasks such as node classification and graph reasoning, and exhibits excellent zero-shot and few-shot performance. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0031] Figure 1is a flowchart of the construction method of the general pre-training graph structure large model provided by the embodiment;

[0032] Figure 2 is a structural flowchart of the construction method of the general pre-training graph structure large model provided by the embodiment;

[0033] Figure 3 is a schematic diagram of graph structure representation learning based on Euler paths provided by the embodiment;

[0034] Figure 4 is a schematic diagram of multi-level cross-modal learning provided by the embodiment. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0036] As shown in Figure 1 and Figure 2 , the construction method of a general pre-training graph structure large model provided by the embodiment includes the following steps:

[0037] S1, after pre-processing the multi-source text attribute graph data, a sample set containing graph topology structure and node text attributes is constructed.

[0038] In the embodiment, public graph datasets from different fields are integrated, specifically including academic citation networks (such as Cora, PubMed, OGB-Arxiv), e-commerce co-purchase networks (such as Amazon-Computers, Amazon-Photo), and molecular graph data (such as PCQM4Mv2). These graph datasets cover various topological patterns from social relationships to chemical structures. Then, the graph datasets are pre-processed, including node ID remapping, feature normalization, etc., and all the graph data are combined into a large-scale mixed graph corpus, which is represented in the form of an entire text attribute graph, containing nodes and link relationships between nodes, as well as text attributes of each node.

[0039] Considering that the size of the entire graph is too large to be directly input into the model, and in order to capture the local structural features of the nodes, a subgraph sampling strategy based on the center node is used to sample the entire graph. Specifically, a ShaDowKHop sampler is used to extract a text attribute subgraph containing k-hop (for example, 2-hop or 3-hop) neighborhood around the target node , wherein, represents a set of nodes, represents a set of edges, represents a set of node text attributes, For super large graphs, random walk strategy can be combined with down-sampling to balance the calculation efficiency and structure coverage. Thus, a sample set of subgraphs and their node text attributes is obtained.

[0040] S2, each text attribute subgraph is converted into a continuous one-dimensional graph Token sequence by using an Euler path-based reversible serialization method; the one-dimensional graph Token sequence is taken as input, and scheduling mask Token prediction is taken as a self-supervised training target to pre-train the graph structure encoder, so that a graph structure representation with general topological understanding ability is obtained.

[0041] Existing graph models (such as GraphGPT, LLaGA) often lose structure due to truncation or sampling during serialization. Therefore, the present application serializes each text attribute subgraph by using an Euler path-based reversible serialization method to obtain a lossless graph structure sequence represented by one-dimensional graph Token. As shown in Figure 3 , the specific process includes:

[0042] To ensure that an Euler path covering all edges exists, first, connectivity enhancement and Euler characteristic preprocessing and path generation are performed: for the disconnected part in the text attribute subgraph , a synthetic edge is added by adding a special token (such as [EDGE_JUMP]) to connect all connected components to obtain a connected graph ; the connected graph is subjected to Eulerization determination to count the number of odd-degree nodes; if the number of odd-degree nodes is not 0 or 2, a virtual edge is added by using a minimum edge duplication strategy to construct an Euler multi-graph that meets the existence condition of the Euler path; Hierholzer algorithm or its variants are executed on the Euler multi-graph to search for an Euler path that covers all edges and visits each edge only once .

[0043] The original node number usually has an implementation bias, and if directly used for tokenization, it will cause a high frequency imbalance of the structure token. Therefore, node re-indexing is also performed: a re-indexing mapping is defined based on the order of first occurrence of the path π ( v ), and further to alleviate the index distribution offset between different graph scales, a cyclic re-indexing mechanism π r ( v ) is introduced, which specifically uses the following formula: π r ( v ) = ( π ( v ) +r ) mod N wherein, r is a random integer uniformly sampled in a preset interval, N is a hyper-parameter greater than the maximum node number, which on the one hand activates different index tokens uniformly in the training process, and on the other hand avoids the OOD situation that large-numbered nodes never appear in pre-training. Based on the above re-indexing, the node sequence s containing only structural information can be obtained struct , and the formula is:

[0044]

[0045] Finally, the attribute attachment is performed: on the basis of the structure sequence, the node and edge attributes need to be integrated. It is recorded that each edge carries d e dimensional or quantized attributes, and each node carries d dimensional attributes. In the extended format, each pair of adjacent nodes is expanded to contain node index, edge type, edge attribute, and node attribute. Finally, the node index, edge type, edge attribute features, and node attribute features in the extended format are converted into discrete tokens to form the final one-dimensional graph token sequence.

[0046] In the embodiment, the graph structure encoder maps the structured one-dimensional graph token sequence generated by the Eulerization path into a node-level structure vector containing rich topological semantics, i.e., a graph structure representation.

[0047] As shown in Figure 3 , the graph structure encoder adopts a bidirectional Transformer structure. Taking the one-dimensional graph token sequence as input, before inputting the one-dimensional graph token sequence, in order to avoid token semantic ambiguity, this paper divides the tokens in the sequence into node / edge / attribute / special marker categories according to the type function, and introduces type embedding to explicitly distinguish different syntax components.

[0048] When pre-training the graph structure encoder, schedule mask token prediction (SMTP) is used as the self-supervised training target, which essentially classifies and predicts the masked tokens in the graph token sequence, so that the graph structure encoder recovers the masked structure and semantic information from the context. Formally, for the input token sequence, first sample a set of mask positions ，and the token at the masked position is replaced by a special token [MASK] to obtain a masked input sequence. After the graph structure encoder encodes the masked input sequence, a prediction head is introduced to predict the masked token to obtain a prediction probability distribution. A loss function is defined based on the average cross-entropy of the prediction probability distributions of all masked positions to optimize the parameters of the graph structure encoder, and a graph structure representation with general topological understanding capability is obtained.

[0049] S3, the text attribute subgraph corresponding node text attribute is encoded into text semantic representation by using the text feature encoder. After the graph structure representation and the text semantic representation are mapped to the same alignment space by the mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and the graph structure representation in the same alignment space. The mapping network is trained by combining the global representation alignment task, the local semantic alignment task at the substructure-phrase level, and the graph-to-text reconstruction task, and a general pre-training graph structure large model composed of the graph structure encoder, the text feature encoder, the mapping network, and the language model introduced for decoding is obtained.

[0050] In the embodiment, the text attribute subgraph corresponding node text attribute is encoded into text semantic representation by using the text feature encoder. Although the graph structure encoder and the text feature encoder can learn effective graph structure representation and text semantic representation respectively, there is still a deviation in the geometric structure and distribution form of the representation space between the two modalities. In this case, if the graph vector is only connected to the pre-training language model through a shallow linear projection, the model can only obtain weakly related cross-modal coupling and it is difficult to stably establish a traceable corresponding relationship between the structure information and the text semantic. Therefore, a multi-level alignment target is constructed for the graph-text cross-modal alignment process to impose stronger constraints on the shared representation space.

[0051] Specifically, as shown in Figure 4 First, the graph structure representation and the text semantic representation are mapped to the same alignment space by the mapping network. The mapping network includes a graph feature alignment network and a text feature alignment network, and the two network structures are the same. The mapping function is used to map the input features to a fixed scale to map the graph structure representation and the text semantic representation to the same alignment space to obtain the mapped graph structure representation and the text semantic representation. The mapping function can use MLP.

[0052] To learn the alignment capability of the mapping network for the two modalities, a multi-level cross-modal loss is constructed by introducing the joint global representation alignment task, the local semantic alignment task at the substructure-phrase level, and the graph-to-text reconstruction task to realize the parameter optimization of the mapping network.

[0053] Specifically, for the global representation alignment task, the InfoNCE loss is used to construct the global representation alignment loss which contains InfoNCE loss defined from graph to text direction and InfoNCE loss defined from text to graph direction i.e.

[0054]

[0055]

[0056]

[0057] wherein, i denotes sample index, denotes the number of samples in a batch, and denote the normalized graph structure representation and the normalized text semantic representation in sample i and denote the normalized graph structure representation and the normalized text semantic representation in sample denotes the similarity between the two, j denotes temperature parameter. Only relying on the previous InfoNCE-based global representation alignment, although it can narrow the semantic distance between the graph structure representation and the text representation as a whole, its constraint mainly acts on the node-level global vector, and it is difficult to explicitly depict the consistency between the local substructure and the fine-grained language segment on the Euler path. To further improve the model's ability to capture edge neighborhood patterns and relationship semantics, a substructure-sentence level local semantic alignment mechanism is introduced on the basis of Euler path serialization, which imposes stronger corresponding constraints on local subgraphs and local text descriptions in a unified alignment space.

[0058] Specifically, for the substructure-sentence level local semantic alignment task, first, construct local substructures and text segments. Specifically, for each target node, extract its local neighborhood subgraph, and divide it into several substructures of controllable size by a substructure sampling operator. For each substructure, the reversible serialization method of the Euler path is also used to obtain the substructure token sequence, and the graph structure encoder is used to encode the substructure token sequence to obtain the substructure feature representation. For the text segment, a local description template is designed for the substructure, which integrates the center node attribute, neighborhood attribute and local relationship pattern into a natural language segment, and is mapped to the semantic space by the text feature encoder to obtain the local text semantic representation.

[0059] After the substructure feature representation and the local text semantic representation are mapped to the same alignment space by the mapping network, the InfoNCE loss is used to construct the substructure alignment loss

[0060] After the substructure feature representation and the local text semantic representation are mapped to the same alignment space by the mapping network, the InfoNCE loss is used to construct the substructure alignment loss ​which contains local InfoNCE loss defined from graph to text direction and local InfoNCE loss defined from text to graph direction i.e.

[0061]

[0062]

[0063]

[0064] where, denotes the number of substructures in the i th sample, and denote the normalized substructure representation and the normalized local text semantic representation of the k th substructure in the th sample, denote the normalized substructure representation and the normalized local text semantic representation of the j th substructure in the l th sample, denotes the similarity between them, denotes the temperature parameter.

[0065] In addition to the substructure-level alignment, we further introduce token-level matching loss to characterize the fine-grained correspondence between the tokens in the subgraph and the text. Specifically, we first compute the similarity between each structure token and its most similar text token as the matching score of each structure During training, we assign label i, k = 1 to the positive sample pair consisting of the same substructure and text, and label = 0 to the negative sample pair consisting of different substructures or different nodes, and construct binary cross-entropy loss to obtain the token-level matching loss :

[0066]

[0067] where, is the Sigmoid function, is the number of samples in a batch.

[0068] The local semantic alignment loss of the substructure-phrase-level local semantic alignment task is :

[0069]

[0070] where, and denote weight parameters.

[0071] After the alignment of the graph structure and the semantic text is completed, a graph-to-text reconstruction task based on cross-entropy is further introduced to strengthen the ability of the graph structure representation to retain and restore the semantic information of the nodes. Specifically, a text decoder is introduced, which takes the graph structure representation mapped by the graph feature alignment network as input, and simultaneously takes the local real text attribute sequence as input. The text decoder generates an autoregressive conditional distribution of the real text attribute token sequence based on the input in turn. The reconstruction loss corresponding to the graph-to-text reconstruction task is .

[0072]

[0073] wherein, denotes a node, is the length of the real text attribute sequence, denotes all word pieces before the t th position, denotes the mapped representation of the node v with the text attribute, denotes the real text attribute sequence, denotes the autoregressive conditional distribution of the text decoder with parameters modeling the node text.

[0074] In summary, the multi-level cross-modal loss used in the training of the mapping network is represented as:

[0075]

[0076] wherein, , , are weight parameters for balancing the relative contributions of the global alignment task, the local alignment task, and the reconstruction task in the training process.

[0077] The intuitive role of joint optimization can be summarized as follows: (i) aligning the node-level graph-text representation in the global semantic space through the InfoNCE constraint, ensuring the consistency of the graph structure vector and the corresponding node text in the embedding space; (ii) further refining the constraint through substructure-phrase-level alignment to avoid excessive smoothing of semantics caused by relying solely on global pooling, and to establish a more fine-grained correspondence between key subgraph patterns and text phrases; (iii) The semantic integrity of the graph representation is verified from the perspective of generative modeling, which requires the graph encoding to be able to restore high-quality node description text when sent into the decoder. The three work together to make the final graph structure representation suitable for downstream node classification tasks and have good cross-modal semantic alignment and generation capabilities.

[0078] After the joint optimization of all the losses mentioned above, a general pre-training graph structure large model is obtained, which is composed of a graph structure encoder, a text feature encoder, a mapping network, and a language model introduced as a decoder. The general pre-training graph structure large model can be used for node classification tasks and graph reasoning tasks.

[0079] When used for node classification tasks: keep the graph structure encoder and the mapping network and the main part of the language model frozen, only fine-tune the classification head in the language model, specifically input the subgraph and text description of the node to be classified, get the structure graph representation through Euler serialization and graph structure encoder, get the text semantic representation through the text feature encoder, and then concatenate the two modal representations after the mapping network and input them into the language model LLM used for decoding, while inputting the language prompt of the node classification task (such as please classify the node) into the language model, then the language model outputs the node class label.

[0080] When used for graph reasoning tasks: based on the general pre-training graph structure large model, cold start fine-tuning is performed using Chain-of-Thought data, and then based on the verifiable reward function and the group strategy optimization (GRPO) algorithm, the model is trained by reinforcement learning to optimize the logicality and correctness of the reasoning path.

[0081] The embodiments also provide performance evaluation of the pre-training graph structure large model in node classification tasks, which specifically constructs the following three schemes:

[0082] Scheme 1: only replace the graph structure input representation and the structure pre-training target. Specifically, under the premise of keeping the overall framework of LLaGA unchanged, replace the neighborhood template with the Euler path structure serialization method as the graph structure input, and introduce the SMTP self-supervised target for structure pre-training on mixed graph data, only optimize the graph structure encoding related modules. This configuration is used to verify the effect of high-fidelity structure serialization and large-scale mixed graph pre-training on downstream tasks. Investigate the influence of different cross-modal training target combinations on model performance, and depict the specific contribution of multi-level cross-modal alignment strategy in node classification tasks.

[0083] Scheme 2: Only introduce multi-level graph-text alignment mechanism, do not change the linearization strategy of the original LLaGA graph structure. In the case of keeping the ND / HO template input and the basic training process unchanged, introduce global contrast, local matching and generative reconstruction and other multi-level graph-text alignment losses. This configuration is used to verify the independent contribution of the alignment loss design.

[0084] Scheme 3: Integrate scheme 1 and scheme 2 to form a complete graph structure large model. That is, the Euler path structure sequence is taken as the graph structure input, the SMTP structure pre-training is carried out first, then the multi-level graph-text alignment is carried out, and finally the supervised fine-tuning is carried out on the downstream task. This configuration is used to evaluate the superimposed synergistic effect of the two types of core designs, and is used as the final method of this paper.

[0085] The performance evaluation results of the three schemes are shown in Table 1:

[0086] Table 1

[0087]

[0088] From Table 1, it can be seen that scheme 3 is better than any single improvement on three data sets: on the PubMed data set, it is improved by 2.21 and 2.74 percentage points respectively compared with schemes 1 and 2; on the Cora data set, it is improved by 1.30 and 2.93 percentage points respectively; on the OGB-Arxiv data set, it is improved by 1.24 and 1.78 percentage points respectively. At the same time, scheme 3 reaches 77.36% on the OGB-Arxiv data. These results show that the high-fidelity serialized representation on the structure side and the multi-level alignment constraint on the semantic side have a complementary effect in the node classification setting, and the combination of the two can stably improve the accuracy of the test set under the unified evaluation configuration.

[0089] Based on the same inventive concept, the embodiment also provides a construction system of a general pre-training graph structure large model, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the construction method of the general pre-training graph structure large model, specifically comprising the following steps:

[0090] S1, obtaining a sample set comprising text attribute subgraphs and node text attributes after preprocessing of multi-source text attribute graph data;

[0091] S2, converting each text attribute subgraph into a continuous one-dimensional graph Token sequence using a reversible serialization method based on the Euler path; using the one-dimensional graph Token sequence as input, using dispatch mask Token prediction as a self-supervised training target, pre-training the graph structure encoder to obtain a graph structure representation with general topological understanding ability;

[0092] S3, a text feature encoder is used to encode the node text attributes corresponding to the text attribute subgraph into text semantic representations, the graph structure representations and the text semantic representations are mapped to the same alignment space through a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representations and the graph structure representations in the same alignment space, the mapping network is trained based on a global representation alignment task, a substructure-phrase level local semantic alignment task, and a graph-to-text reconstruction task, and a general pre-training graph structure large model composed of the graph structure encoder, the text feature encoder, the mapping network, and a language model introduced for decoding is obtained.

[0093] At the hardware level, in addition to the processor and the memory, the system also includes internal buses, network interfaces, memories, and other hardware required by the business. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the construction method of the general pre-training graph structure large model described in S1-S3. Of course, in addition to the software implementation, the present application does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc. That is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0094] Based on the same inventive concept, the embodiment also provides a computer readable storage medium having a program stored thereon, which, when executed by a processor, implements the construction method of the general pre-training graph structure large model described above, and specifically includes the following steps:

[0095] S1, after pre-processing the multi-source text attribute graph data, a sample set containing text attribute subgraphs and node text attributes is constructed;

[0096] S2, a reversible serialization method based on Euler path is used to convert each text attribute subgraph into a continuous one-dimensional graph Token sequence; the one-dimensional graph Token sequence is used as input, and scheduling mask Token prediction is used as a self-supervised training target to pre-train the graph structure encoder, and a graph structure representation with general topological understanding capability is obtained;

[0097] S3, a text feature encoder is used to encode the node text attributes corresponding to the text attribute subgraph into text semantic representations, the graph structure representations and the text semantic representations are mapped to the same alignment space through a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representations and the graph structure representations in the same alignment space, the mapping network is trained based on a global representation alignment task, a substructure-phrase level local semantic alignment task, and a graph-to-text reconstruction task, and a general pre-training graph structure large model composed of the graph structure encoder, the text feature encoder, the mapping network, and a language model introduced for decoding is obtained.

[0098] In embodiments, computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data.

[0099] The above detailed description sets forth the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modification, supplement, and equivalent replacement within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a general pre-training graph structure large model, characterized in that, The method comprises the following steps: After preprocessing the multi-source text attribute graph data, a sample set containing text attribute subgraphs and node text attributes is constructed; Each text attribute subgraph is converted into a continuous one-dimensional graph Token sequence by using an Euler path-based reversible serialization method; the one-dimensional graph Token sequence is taken as input, a scheduling mask Token prediction is taken as a self-supervised training target, a graph structure encoder is pre-trained, and a graph structure representation with general topological understanding capability is obtained; The node text attributes corresponding to the text attribute subgraphs are encoded into text semantic representations by using a text feature encoder; after the graph structure representation and the text semantic representation are mapped to the same alignment space by a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and the graph structure representation in the same alignment space; a global representation alignment task, a substructure-phrase-level local semantic alignment task, and a graph-to-text reconstruction task are combined to train the mapping network, and a general pre-trained graph structure large model composed of the graph structure encoder, the text feature encoder, the mapping network, and a language model introduced for decoding is obtained; The multi-level cross-modal loss comprises a global representation alignment loss corresponding to the global representation alignment task, a local semantic alignment loss corresponding to the substructure-phrase-level local semantic alignment task, and a reconstruction loss corresponding to the graph-to-text reconstruction task; For the global representation alignment task, reference contrastive learning adopts InfoNCE loss to construct global representation alignment loss which includes InfoNCE loss defined from the graph to the text direction based on the normalized graph structure representation and the normalized text semantic representation and the InfoNCE loss defined from the text to the graph direction ; For the local semantic alignment task of substructure-phrase level, the local semantic alignment loss is constructed including token-level matching loss constructed based on the matching degree between structure tokens and text tokens , also including InfoNCE loss defined from the graph to the text direction based on the normalized substructure representation and the normalized local text semantic representation and the InfoNCE loss defined from the text to the graph direction , wherein the substructure representation is the representation of the substructure obtained by dividing the text attribute subgraph through the graph structure encoder and the mapping network, and the local text semantic representation is the representation of the text segment extracted from the node text attribute through the text feature encoder and the mapping network.

2. The method of claim 1, wherein the method further comprises: Each text attribute subgraph is converted into a continuous one-dimensional graph Token sequence by using an Euler path-based reversible serialization method, which comprises the following steps: First, connectivity enhancement and Euler characteristic preprocessing and path generation are performed; Then, node reindexing and cycle reindexing are performed to construct a node sequence; Finally, attribute attachment is performed based on the node sequence, and node index, edge type, edge attribute feature, and node attribute feature are converted into discrete Token to form a final one-dimensional graph Token sequence.

3. The method of claim 1, wherein the method further comprises: The graph structure encoder adopts a bidirectional Transformer structure, and a scheduling mask Token prediction is taken as a self-supervised training target to pre-train the graph structure encoder, which comprises the following steps: For the input token sequence, a set of mask position sets is first sampled according to the scheduling strategy ， And replace the tokens of these mask positions with special marks [MASK] to obtain the masked input sequence. After the graph structure encoder encodes based on the masked input sequence, the prediction head is introduced to predict the mask tokens to obtain the prediction probability distribution. The average cross-entropy of the prediction probability distribution of all mask positions is used to define the loss function to optimize the parameters of the graph structure encoder.

4. The method of claim 1, wherein the method further comprises: The mapping network comprises a graph feature alignment network and a text feature alignment network, and the two network structures are the same, both of which adopt a mapping function to map input features to a fixed scale to map the graph structure representation and the text semantic representation to the same alignment space.

5. The method of claim 1, wherein the method further comprises: For the image-to-text reconstruction task, a reconstruction loss is constructed cross-entropy of an autoregressive conditional distribution that outputs a sequence of real text attributes tokens for the text decoder; The text decoder takes the graph structure representation mapped by the graph feature alignment network as input, simultaneously takes a local real text attribute sequence as input, and generates an autoregressive conditional distribution of a real text attribute Token sequence based on the input.

6. A system for constructing a general pre-trained graph structure large model, comprising a memory and one or more processors, the memory having stored therein executable code, the system being characterized by: The one or more processors execute the executable code to implement the construction method of the general pre-trained graph structure large model in any one of claims 1-5.

7. A computer readable storage medium characterized in that, The program is stored on the storage medium and is executed by the processor to implement the construction method of the general pre-trained graph structure large model in any one of claims 1-5.

Images