Construction method and system of general pre-training graph structure large model

Abstract

The invention discloses a method and system for constructing a general pre-training graph structure large model, and belongs to the technical field of graph computing, and the method comprises the steps: constructing a sample set containing text attribute sub-graphs and node text attributes from multi-source text attribute graph data; pre-training the graph structure encoder by adopting a reversible serialization improvement mode based on an Euler path to obtain graph structure representation; a text feature encoder is adopted to encode node text attributes into text semantic representations, and graph structure representations and text semantic representations are mapped to the same alignment space through mapping network representations; a joint global representation alignment task, a substructure-phrase level local semantic alignment task and a graph-to-text reconstruction task are constructed based on text semantic representation and graph structure representation, a mapping network is trained, and a general pre-training graph structure large model is obtained, so that a modal gap between a graph structure and text semantics can be effectively bridged; and the generalization ability and the reasoning credibility of the model in downstream tasks are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of combining graph computing and artificial intelligence, and particularly relates to a construction method and system of a general pre-training graph structure large model. BACKGROUND

[0002] Graph structure data widely exists in the real world, such as user interaction relationships in social networks, molecular structures and protein interactions in biological medicine, and commodity association networks on e-commerce platforms. These data usually exist in the form of text-attributed graphs (TAGs), that is, the nodes and edges in the graph are accompanied by rich natural language descriptions (such as the title and abstract of a paper, the detailed introduction of a commodity). In recent years, with the breakthrough progress of large language models (LLMs) in the field of natural language processing, graph-LLMs that combine the powerful semantic understanding ability of LLMs with the structure modeling ability of graph neural networks (GNNs) have become the current research hotspot. Inspired by the success of foundation models in natural language processing and computer vision, graph learning has recently entered the era of graph foundation models (GFMs). These models are pre-trained on large-scale graphs using self-supervised objectives, enabling them to learn general representations that can be transferred across tasks and domains. GFMs integrate structural dependencies and semantic content, showing strong zero-shot and few-shot generalization. By decoupling model training from specific tasks, GFMs reduce the dependence on labeled data and specific domain heuristic methods, making the field closer to general graph intelligence.

[0003] However, existing Graph-LLMs still face significant technical bottlenecks in terms of technical implementation and application effect, mainly in the following two core problems: First, there is a serious information loss and lack of topological fidelity in graph structure encoding. Existing mainstream methods (such as LLaGA, GraphGPT, etc.) often rely on random walk, neighborhood sampling (such as Neighbor Sampling) or simplified linearization strategy (such as DFS / BFS traversal) to convert the graph structure into a sequence form that can be processed by LLM. These processes often introduce start point bias and randomness of traversal path. More seriously, in order to adapt to the limited context window of LLM, these methods have to truncate the graph or only keep part of the neighbors, resulting in the loss of key global topological structure information such as cycle, bridge and long link in the graph. The sequence received by the model is incomplete, and it is difficult to restore the complete connectivity of the graph based on it, thus performing poorly in tasks that require accurate structure perception (such as judging connectivity, shortest path search, complex subgraph matching), resulting in systematic performance degradation.

[0004] Second, the graph-text representation space is fragmented, and the cross-modal alignment granularity is coarse. After converting the graph into a sequence input for LLM, existing alignment strategies usually only perform coarse-grained contrastive learning (Contrastive Learning) at the full graph / subgraph level and the text paragraph level. This approach ignores the fine-grained correspondence between local substructures in the graph (such as benzene rings in chemical molecules, fully connected subgroups in social networks) and specific phrases in the text (such as aromatic, tight community). Language models naturally lack graph induction bias, resulting in inconsistent semantic mapping when graph features are projected into language space. This manifests as the model generating semantically coherent but structurally incorrect answers, i.e., incorrectly describing the connection between nodes, or failing to accurately anchor the topological features of the graph to the corresponding text description, severely limiting the model's generalization ability and explainability in complex graph tasks.

[0005] In view of the above technical problems, there is an urgent need for a general graph structure large model construction scheme that can realize lossless serialization of graph structure and implement deep semantic alignment between graph and text at multiple levels, to support more accurate node classification and more reliable graph reasoning applications. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a general pre-training graph structure large model construction method and system, by introducing a reversible serialization mechanism based on Euler path, providing a high-quality serialization method for graph structure representation, and designing a multi-level alignment strategy including global, local and generative dimensions, effectively bridging the modal gap between graph structure and text semantics, and improving the generalization ability and reasoning reliability of the graph structure large model in downstream tasks.

[0007] To achieve the above invention purpose, the general pre-training graph structure large model construction method provided by the embodiment comprises the following steps: After preprocessing the multi-source text attribute graph data, a sample set containing text attribute subgraphs and node text attributes is constructed. Each text attribute subgraph is converted into a continuous one-dimensional graph token sequence using a reversible serialization method based on Eulerian paths. The graph structure encoder is pre-trained using the one-dimensional graph token sequence as input and scheduling mask token prediction as a self-supervised training objective to obtain a graph structure representation with general topological understanding capabilities. A text feature encoder is used to encode the text attributes of nodes corresponding to text attribute subgraphs into text semantic representations. After mapping the graph structure representation and text semantic representation to the same alignment space through a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and graph structure representation in the same alignment space. The mapping network is trained by combining the global representation alignment task, the substructure-phrase level local semantic alignment task, and the graph-to-text reconstruction task, resulting in a general pre-trained graph structure large model composed of a graph structure encoder, a text feature encoder, a mapping network, and a language model introduced for decoding.

[0008] Preferably, a reversible serialization method based on Eulerian paths is used to convert each text attribute subgraph into a continuous one-dimensional graph token sequence, including: First, connectivity enhancement, Euler property preprocessing, and path generation are performed; Then, node reindexing and cyclic reindexing are performed to construct the node sequence; Finally, attribute attachment is performed based on the node sequence, and the node index, edge type, edge attribute features and node attribute features are transformed into discrete tokens, which are then combined to form the final one-dimensional graph token sequence.

[0009] Preferably, the graph structure encoder adopts a bidirectional Transformer structure and uses scheduling mask token prediction as a self-supervised training objective to pre-train the graph structure encoder, including: For the input token sequence, first sample a set of mask positions according to the scheduling policy. ， The tokens at these mask positions are replaced with a special marker [MASK] to obtain the masked input sequence. The graph encoder is then used to encode the masked input sequence, and a prediction head is introduced to predict the mask tokens to obtain the prediction probability distribution. The loss function is defined based on the average cross-entropy of the prediction probability distributions at all mask positions to optimize the graph encoder parameters.

[0010] Preferably, the mapping network includes a graph feature alignment network and a text feature alignment network. The two networks have the same structure and both use a mapping function to map the input features to a fixed scale, so as to map the graph structure representation and the text semantic representation to the same alignment space.

[0011] Preferably, the multi-level cross-modal loss includes the global representation alignment loss corresponding to the joint global representation alignment task, the local semantic alignment loss corresponding to the substructure-phrase level local semantic alignment task, and the reconstruction loss corresponding to the graph-to-text reconstruction task.

[0012] Preferably, for the global representation alignment task, reference contrastive learning uses InfoNCE loss to construct the global representation alignment loss. It includes InfoNCE loss defined from the graph to text direction, based on normalized graph structure representation and normalized text semantic representation. InfoNCE loss defined from text to graph direction .

[0013] Preferably, for the substructure-phrase level local semantic alignment task, the constructed local semantic alignment loss is... This includes a token-level matching loss constructed based on the degree of matching between structured tokens and text tokens. It also includes InfoNCE loss defined from the graph to text direction based on normalized substructure representation and normalized local text semantic representation. InfoNCE loss defined from text to graph direction The substructure representation is obtained by dividing the text attribute subgraph into substructures and passing them through a graph structure encoder and a mapping network. The local text semantic representation is obtained by extracting text fragments from node text attributes and passing them through a text feature encoder and a mapping network.

[0014] Preferably, for the graph-to-text reconstruction task, the constructed reconstruction loss is... The cross-entropy of the autoregressive conditional distribution of the real text attribute token sequence output by the text decoder; The text decoder takes the graph structure representation mapped by the graph feature alignment network as input, and the local real text attribute sequence as input. Based on the input, it decodes and generates the autoregressive conditional distribution of the real text attribute token sequence in sequence.

[0015] To achieve the above-mentioned objectives, the embodiments also provide a system for constructing a general pre-trained graph structure large model, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned method for constructing a general pre-trained graph structure large model.

[0016] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-mentioned method for constructing a general pre-trained graph structure large model.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. High structural fidelity: To address the issue of easily losing local connectivity patterns and structural details during graph data serialization, this paper implements a graph serialization method based on Eulerian paths to achieve full-fidelity encoding of the topological structure and local connectivity patterns within the viewport. Within a limited context window, it maximizes the preservation of connectivity and structural semantics within subgraphs, effectively alleviating the problem of dependency capture within the local receptive field, and providing a superior foundation for structure awareness in graph models.

[0018] 2. Precise Semantic Alignment: A multi-level alignment strategy breaks through the limitations of single global alignment. Through joint optimization across three dimensions—global, local fine-grained, and semantic generation completeness—the model deeply aligns graph and text representations in the latent space, enhancing the graph model's ability to deeply understand and semantically associate heterogeneous graph and text data. The large graph structure model combining graph sequence representation improvement and cross-modal loss alignment outperforms existing similar methods by 2.71%, 1.30%, and 3.34% in test accuracy on node classification tasks such as Cora, PubMed, and OGB-Arxiv, respectively.

[0019] 3. Strong generalization ability: By pre-training with mixed multi-source heterogeneous graph data and combining high-fidelity graph structure serialization encoding, the model can be seamlessly transferred to various downstream tasks such as node classification and graph reasoning, showing excellent zero-shot and few-shot performance. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the method for constructing a general pre-trained graph structure large model provided in the embodiment; Figure 2 This is a flowchart illustrating the construction method of a general pre-trained graph structure large model provided in the embodiment. Figure 3 This is a schematic diagram of graph structure representation learning based on Eulerian paths provided in the embodiment; Figure 4This is a schematic diagram of multi-level cross-modal learning provided in the embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0023] like Figure 1 and Figure 2 As shown in the embodiment, a method for constructing a general pre-trained graph structure large model includes the following steps: S1. After preprocessing the multi-source text attribute graph data, a sample set containing the graph topology and node text attributes is constructed.

[0024] In this embodiment, publicly available graph datasets from different fields are integrated, specifically including academic citation networks (such as Cora, PubMed, OGB-Arxiv), e-commerce co-purchasing networks (such as Amazon-Computers, Amazon-Photo), and molecular graph data (such as PCQM4Mv2). These graph datasets cover a variety of topological patterns, ranging from social relationships to chemical structures. These graph datasets are then preprocessed, including node ID remapping and feature normalization, to merge all graph data into a large-scale hybrid graph corpus. This hybrid graph corpus is represented as a complete text attribute graph, containing nodes, the links between nodes, and the text attributes of each node.

[0025] Considering that the entire graph is too large to be directly input into the model, and in order to capture the local structural features of the nodes, a subgraph sampling strategy based on the center node is adopted to sample the entire graph. Specifically, the ShaDowKHop sampler is used to extract text attribute subgraphs containing k-hop (e.g., 2-hop or 3-hop) neighborhoods centered on the target node. ,in, Represents a set of nodes. Denotes the set of edges. Represents the collection of text attributes of a node. The adjacency matrix is ​​used. For very large graphs, a random walk strategy can be used for downsampling to balance computational efficiency and structural coverage. This yields a sample set consisting of the subgraph and its node text attributes.

[0026] S2 uses a reversible serialization method based on Eulerian paths to convert each text attribute subgraph into a continuous one-dimensional graph token sequence; using the one-dimensional graph token sequence as input and scheduling mask token prediction as a self-supervised training objective, the graph structure encoder is pre-trained to obtain a graph structure representation with general topological understanding capabilities. Existing graph models (such as GraphGPT and LLaGA) often suffer structural loss during serialization due to truncation or sampling. To address this, this invention employs a reversible serialization method based on Eulerian paths to serialize each text attribute subgraph, resulting in a lossless graph structure sequence represented by a one-dimensional graph token. For example... Figure 3 As shown, the specific process includes: To ensure the existence of an Eulerian path covering all edges, connectivity enhancement and Eulerian property preprocessing are first performed before path generation: for text attribute subgraphs In areas where there is no connection, composite edges are created by adding a special token (e.g., [EDGE_JUMP]). By connecting all connected components, a connected graph is obtained. For connected graphs Perform Eulerianization and count the number of odd-degree nodes; if the number of odd-degree nodes is not 0 or 2, add virtual edges using the minimum edge copying strategy. Construct an Eulerian multigraph that satisfies the existence condition of an Eulerian path. In Euler multigraphs Perform the Hierholzer algorithm or a variant thereof to search and generate an Eulerian path that covers all edges and visits each edge only once. .

[0027] The original node numbers usually have implementation biases, and if used directly for tokenization, it will lead to a highly uneven frequency of tokens in the structure. Therefore, node re-indexing is performed: the re-indexing mapping is defined based on the order of the first occurrence of the path. π ( v To further mitigate index distribution offsets across different map scales, a cyclic reindexing mechanism is introduced. π r ( v The specific formula is as follows: π r ( v ) = ( π ( v ) + r ) mod N ,in, r A random integer sampled uniformly within a preset interval. NFor hyperparameters greater than the maximum number of nodes, this design, on the one hand, uniformly activates different index tokens during training, and on the other hand, avoids Out-of-Depth (OOD) scenarios where large-numbered nodes never appeared in pre-training during the testing phase. Based on the above re-indexing, a node sequence s containing only structural information can be obtained. struct Its formula is: Finally, attribute attachment is performed: based on the structural sequence, node and edge attributes need to be integrated. Each edge carries... d e The attributes are discretized or quantized in dimensionality, and each node carries... d One-dimensional attributes. In the extended format, each pair of adjacent nodes is expanded to include node index, edge type, edge attribute, and node attribute. Finally, the node index, edge type, edge attribute features, and node attribute features in the extended format are converted into discrete tokens and combined to form the final one-dimensional graph token sequence.

[0028] In the embodiment, a graph structure encoder is constructed to map the structured one-dimensional graph token sequence generated by the Eulerian path into a node-level structure vector containing rich topological semantics, i.e., a graph structure representation.

[0029] Among them, such as Figure 3 As shown, the graph structure encoder adopts a bidirectional Transformer structure. Taking a one-dimensional graph token sequence as input, before the input, in order to avoid semantic ambiguity of the tokens, this paper classifies the tokens in the sequence into categories such as nodes / edges / attributes / special markers according to type functions, and introduces type embedding to explicitly distinguish different syntactic components.

[0030] When pre-training the graph structure encoder, Scheduled Masked Token Prediction (SMTP) is used as the self-supervised training objective. Essentially, SMTP classifies and predicts masked tokens in the graph token sequence, enabling the graph structure encoder to recover the masked structural and semantic information from the context. Formally, for the input token sequence, a set of mask positions is first sampled according to the scheduling strategy. ， The tokens at these mask positions are replaced with special markers [MASK] to obtain the masked input sequence. The graph structure encoder is used to encode the input sequence based on the masked sequence, and a prediction head is introduced to predict the mask tokens to obtain the prediction probability distribution. The loss function is defined based on the average cross-entropy of the prediction probability distributions at all mask positions to optimize the graph structure encoder parameters and obtain a graph structure representation with general topological understanding capabilities.

[0031] S3 employs a text feature encoder to encode the text attributes of nodes corresponding to text attribute subgraphs into text semantic representations. After mapping the graph structure representation and text semantic representation to the same alignment space through a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and graph structure representation in the same alignment space. The mapping network is trained by combining the global representation alignment task, the substructure-phrase level local semantic alignment task, and the graph-to-text reconstruction task, resulting in a general pre-trained graph structure large model composed of a graph structure encoder, a text feature encoder, a mapping network, and a language model introduced for decoding.

[0032] In this embodiment, a text feature encoder is used to encode the text attributes of nodes corresponding to the text attribute subgraph into text semantic representations. Although the graph structure encoder and the text feature encoder can learn effective graph structure representations and text semantic representations respectively, there are still offsets between the two modalities in the geometric structure and distribution of the representation space. In this case, if graph vectors are only connected to the pre-trained language model through shallow linear projection, the model often only obtains weakly correlated cross-modal coupling, making it difficult to stably establish a traceable correspondence between structural information and text semantics. To address this, a multi-level alignment objective is constructed for the graph-text cross-modal alignment process to impose stronger constraints on the shared representation space.

[0033] Specifically, such as Figure 4 As shown, the graph structure representation and text semantic representation are first mapped to the same alignment space through a mapping network. This mapping network includes a graph feature alignment network and a text feature alignment network. These two networks have the same structure and both use a mapping function to map the input features to a fixed scale, thus mapping the graph structure representation and text semantic representation to the same alignment space, resulting in the mapped graph structure representation and text semantic representation. The mapping function can be an MLP.

[0034] To learn the mapping network's ability to align features between two modalities, a joint global representation alignment task, a substructure-phrase level local semantic alignment task, and a graph-to-text reconstruction task are introduced to construct a multi-level cross-modal loss to optimize the parameters of the mapping network.

[0035] Specifically, for the global representation alignment task, the reference contrastive learning uses InfoNCE loss to construct the global representation alignment loss. It includes the InfoNCE loss defined from the graph to the text direction. InfoNCE loss defined from text to graph direction ,Right now: in, iIndicates the sample index. This indicates the number of samples in a batch. and Representing samples respectively i Normalized graph structure representation and normalized text semantic representation and Representing samples respectively j Normalized graph structure representation and normalized text semantic representation Indicates the similarity between the two. This represents the temperature parameter.

[0036] While relying solely on the global representation alignment based on InfoNCE discussed earlier can narrow the semantic distance between graph structure representations and text representations overall, its constraints primarily apply to node-level global vectors, making it difficult to explicitly characterize the consistency between local substructures and fine-grained language fragments on the Eulerian path. To further enhance the model's ability to capture edge neighborhood patterns and relational semantics, a substructure-phrase-level local semantic alignment mechanism is introduced based on Eulerian path serialization, imposing stronger correspondence constraints on local subgraphs and local text descriptions within a unified alignment space.

[0037] Specifically, for the substructure-phrase level local semantic alignment task, firstly, local substructures and text fragments are constructed. For each target node, its local neighborhood subgraph is extracted and divided into several substructures of controllable size using a substructure sampling operator. For each substructure, a reversible Eulerian path serialization method is used to obtain a substructure token sequence. A graph-structured encoder is then used to encode the substructure token sequence to obtain substructure feature representations. For text fragments, local description templates are designed for the substructures, integrating the central node attributes, neighborhood attributes, and local relational patterns into a natural language fragment. This fragment is then mapped to the semantic space by a text feature encoder to obtain local text semantic representations.

[0038] Similarly, after mapping substructure feature representations and local text semantic representations to the same alignment space using a mapping network, the InfoNCE loss is used to construct the substructure alignment loss. It includes the local InfoNCE loss defined from the graph to the text direction. Local InfoNCE loss defined from text to graph direction ,Right now: in, Indicates the first i The number of substructures in a sample and Representing substructures k The corresponding normalized substructure representation and normalized local text semantic representation, and Representing samples respectively j Neutron structure l The corresponding normalized substructure representation and normalized local text semantic representation, Indicates the similarity between the two. This represents the temperature parameter.

[0039] In addition to substructure-level alignment, token-level matching loss is further introduced. This approach aims to characterize the fine-grained correspondence between structural tokens and text tokens within a subgraph. Specifically, it first calculates the similarity between each structural token and its most similar text token as a matching score for each structural token. During training, for positive sample pairs ( i, k Assigning labels = 1, assigning labels to negative sample pairs composed of different substructures or different nodes. =0, construct the binary cross-entropy loss, and obtain the token-level matching loss. : in, For the Sigmoid function, This represents the number of samples in a batch.

[0040] The local semantic alignment loss of the substructure-phrase-level local semantic alignment task for: in, and This represents the weighting parameter.

[0041] After completing the comparison and alignment of graph structure and semantic text, a graph-to-text reconstruction task based on cross-entropy is further introduced to enhance the ability of graph structure representation to preserve and restore the semantic information of nodes. Specifically, a text decoder is introduced, which takes the graph structure representation mapped by the graph feature alignment network as input, and simultaneously takes the local real text attribute sequence as input. The text decoder decodes based on the input to sequentially generate the autoregressive conditional distribution of the real text attribute token sequence. The reconstruction loss corresponding to the graph-to-text reconstruction task is... for: in, Represents a node. The length of the actual text attribute sequence. Indicates the first t All word fragments preceding that position, Nodes that represent text attributes v The corresponding mapped representation, Represents a sequence of actual text attributes. The parameter is The text decoder models the autoregressive conditional distribution of node text.

[0042] In summary, the multi-level cross-modal loss used during the training of the mapping network... Represented as: in, , , These are weight parameters used to balance the relative contributions of the global alignment task, the local alignment task, and the reconstruction task during the training process.

[0043] The intuitive benefits of joint optimization can be summarized as follows: (i) (ii) By using InfoNCE to constrain the alignment of node-level graph-text representations in the global semantic space, the consistency between graph structure vectors and corresponding node text in the embedding space is ensured; The constraints are further refined by substructure-phrase level alignment, avoiding excessive semantic smoothing caused by relying solely on global pooling, and establishing a more refined correspondence between key subgraph patterns and text phrases; (iii) This approach reverse-engineers the semantic integrity of the graph representation from a generative modeling perspective, requiring the graph encoding to recover high-quality node description text when fed into the decoder. The synergistic effect of these three aspects ensures that the final graph structure representation is suitable for downstream node classification tasks and possesses excellent cross-modal semantic alignment and generation capabilities.

[0044] After joint optimization of all the above losses, a general pre-trained large graph structure model is obtained, consisting of a graph structure encoder, a text feature encoder, a mapping network, and a language model introduced for decoding. This general pre-trained large graph structure model can be used for node classification tasks and graph reasoning tasks.

[0045] When used for node classification tasks: the backbone of the graph structure encoder, mapping network, and language model is kept frozen, and only the classification head in the language model is fine-tuned. The specific input is the subgraph of the node to be classified and the text description. The structural graph representation is obtained through Euler serialization and graph structure encoder, and the text semantic representation is obtained through text feature encoder. These two modal representations are concatenated after passing through the mapping network and input into the language model LLM used for decoding. At the same time, the language model is input with language prompts for the node classification task (such as please classify the node). Then the language model outputs the node category label.

[0046] When used for graph reasoning tasks: Based on a general pre-trained graph structure model, cold-start fine-tuning is performed using chain-of-thought data. Subsequently, reinforcement learning training is performed on the model based on a verifiable reward function and grouped policy optimization (GRPO) algorithm to optimize the logic and correctness of the reasoning path.

[0047] The implementation also provides an overall performance evaluation of the above-mentioned pre-trained graph structure large model on the node classification task, specifically constructing the following three schemes: Option 1: Replace only the graph structure input representation and the structure pre-training objective. Specifically, while keeping the overall framework of LLaGA unchanged, replace the neighborhood template with Eulerian path structure serialization as the graph structure input, and introduce the SMTP self-supervised objective on mixed graph data for structure pre-training, optimizing only the graph structure encoding-related modules. This configuration is used to verify the effects of high-fidelity structure serialization and large-scale mixed graph pre-training on downstream tasks, examine the impact of different combinations of cross-modal training objectives on model performance, and characterize the specific contribution of multi-level cross-modal alignment strategies in node classification tasks.

[0048] Option 2: Introduce only a multi-level graph-text alignment mechanism without changing the original LLaGA graph structure linearization strategy. While keeping the ND / HO template input and basic training process unchanged, introduce multi-level graph-text alignment losses such as global contrast, local matching, and generative reconstruction. This configuration is used to verify the independent contribution of the alignment loss design.

[0049] Option 3: Integrate Option 1 and Option 2 to form a complete large-scale graph structure model. This involves using the Eulerian path structure sequence as the graph structure input, first pre-training the SMTP structure, then performing multi-level graph-text alignment, and finally supervised fine-tuning on downstream tasks. This configuration is used to evaluate the synergistic effect of the two core designs and serves as the final method presented in this paper.

[0050] The performance evaluation results of the three schemes are shown in Table 1: Table 1 Analysis of Table 1 shows that Scheme 3 outperforms any single improvement on all three datasets: on the PubMed dataset, it improves accuracy by 2.21 and 2.74 percentage points compared to Scheme 1 and Scheme 2, respectively; on the Cora dataset, it improves accuracy by 1.30 and 2.93 percentage points, respectively; and on the OGB-Arxiv dataset, it improves accuracy by 1.24 and 1.78 percentage points, respectively. Furthermore, Scheme 3 achieves 77.36% accuracy on the OGB-Arxiv dataset. These results indicate that high-fidelity serialization representation on the structural side and multi-level alignment constraints on the semantic side are complementary under node classification settings, and their combination can stably improve test set accuracy under a unified evaluation configuration.

[0051] Based on the same inventive concept, the embodiment also provides a system for constructing a general pre-trained graph structure large model, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned method for constructing a general pre-trained graph structure large model, specifically including the following steps: S1, after obtaining multi-source text attribute graph data and preprocessing it, construct a sample set containing text attribute subgraphs and node text attributes; S2 uses a reversible serialization method based on Eulerian paths to convert each text attribute subgraph into a continuous one-dimensional graph token sequence; using the one-dimensional graph token sequence as input and scheduling mask token prediction as a self-supervised training objective, the graph structure encoder is pre-trained to obtain a graph structure representation with general topological understanding capabilities. S3 employs a text feature encoder to encode the text attributes of nodes corresponding to text attribute subgraphs into text semantic representations. After mapping the graph structure representation and text semantic representation to the same alignment space through a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and graph structure representation in the same alignment space. The mapping network is trained by combining the global representation alignment task, the substructure-phrase level local semantic alignment task, and the graph-to-text reconstruction task, resulting in a general pre-trained graph structure large model composed of a graph structure encoder, a text feature encoder, a mapping network, and a language model introduced for decoding.

[0052] At the hardware level, in addition to the processor and memory, the system also includes internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the method for constructing the general pre-trained graph structure large model described in S1-S3 above. Of course, in addition to the software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0053] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described method for constructing a general pre-trained graph structure large model, specifically including the following steps: S1, after obtaining multi-source text attribute graph data and preprocessing it, construct a sample set containing text attribute subgraphs and node text attributes; S2 uses a reversible serialization method based on Eulerian paths to convert each text attribute subgraph into a continuous one-dimensional graph token sequence; using the one-dimensional graph token sequence as input and scheduling mask token prediction as a self-supervised training objective, the graph structure encoder is pre-trained to obtain a graph structure representation with general topological understanding capabilities. S3 employs a text feature encoder to encode the text attributes of nodes corresponding to text attribute subgraphs into text semantic representations. After mapping the graph structure representation and text semantic representation to the same alignment space through a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and graph structure representation in the same alignment space. The mapping network is trained by combining the global representation alignment task, the substructure-phrase level local semantic alignment task, and the graph-to-text reconstruction task, resulting in a general pre-trained graph structure large model composed of a graph structure encoder, a text feature encoder, a mapping network, and a language model introduced for decoding.

[0054] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0055] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing a general pre-trained large graph structure model, characterized in that, Includes the following steps: After preprocessing the multi-source text attribute graph data, a sample set containing text attribute subgraphs and node text attributes is constructed. Each text attribute subgraph is converted into a continuous one-dimensional graph token sequence using a reversible serialization method based on Eulerian paths. The graph structure encoder is pre-trained using the one-dimensional graph token sequence as input and scheduling mask token prediction as a self-supervised training objective to obtain a graph structure representation with general topological understanding capabilities. A text feature encoder is used to encode the text attributes of nodes corresponding to text attribute subgraphs into text semantic representations. After mapping the graph structure representation and text semantic representation to the same alignment space through a mapping network, a multi-level cross-modal loss is constructed based on the text semantic representation and graph structure representation in the same alignment space. The mapping network is trained by combining the global representation alignment task, the substructure-phrase level local semantic alignment task, and the graph-to-text reconstruction task, resulting in a general pre-trained graph structure large model composed of a graph structure encoder, a text feature encoder, a mapping network, and a language model introduced for decoding.

2. The method for constructing a general pre-trained graph structure large model according to claim 1, characterized in that, A reversible serialization method based on Eulerian paths is used to convert each text attribute subgraph into a continuous one-dimensional graph token sequence, including: First, connectivity enhancement, Euler property preprocessing, and path generation are performed; Then, node re-indexing and cyclic re-indexing are performed to construct the node sequence; Finally, attribute attachment is performed based on the node sequence, and the node index, edge type, edge attribute features and node attribute features are transformed into discrete tokens, which are then combined to form the final one-dimensional graph token sequence.

3. The method for constructing a general pre-trained graph structure large model according to claim 1, characterized in that, The graph structure encoder adopts a bidirectional Transformer structure and uses scheduling mask token prediction as the self-supervised training objective to pre-train the graph structure encoder, including: For the input token sequence, first sample a set of mask positions according to the scheduling policy. ， The tokens at these mask positions are replaced with a special marker [MASK] to obtain the masked input sequence. The graph encoder is then used to encode the masked input sequence, and a prediction head is introduced to predict the mask tokens to obtain the prediction probability distribution. The loss function is defined based on the average cross-entropy of the prediction probability distributions at all mask positions to optimize the graph encoder parameters.

4. The method for constructing a general pre-trained graph structure large model according to claim 1, characterized in that, The mapping network includes a graph feature alignment network and a text feature alignment network. The two networks have the same structure and both use a mapping function to map the input features to a fixed scale, so as to map the graph structure representation and the text semantic representation to the same alignment space.

5. The method for constructing a general pre-trained graph structure large model according to claim 1, characterized in that, The multi-level cross-modal loss includes the global representation alignment loss corresponding to the joint global representation alignment task, the local semantic alignment loss corresponding to the substructure-phrase level local semantic alignment task, and the reconstruction loss corresponding to the graph-to-text reconstruction task.

6. The method for constructing a general pre-trained graph structure large model according to claim 1 or 5, characterized in that, For the global representation alignment task, reference contrastive learning uses InfoNCE loss to construct the global representation alignment loss. It includes InfoNCE loss defined from the graph to text direction, based on normalized graph structure representation and normalized text semantic representation. InfoNCE loss defined from text to graph direction .

7. The method for constructing a general pre-trained graph structure large model according to claim 1 or 5, characterized in that, For the substructure-phrase level local semantic alignment task, a local semantic alignment loss is constructed. This includes a token-level matching loss constructed based on the degree of matching between structured tokens and text tokens. It also includes InfoNCE loss defined from the graph to text direction based on normalized substructure representation and normalized local text semantic representation. InfoNCE loss defined from text to graph direction The substructure representation is obtained by dividing the text attribute subgraph into substructures and passing them through a graph structure encoder and a mapping network. The local text semantic representation is obtained by extracting text fragments from node text attributes and passing them through a text feature encoder and a mapping network.

8. The method for constructing a general pre-trained graph structure large model according to claim 1 or 5, characterized in that, For graph-to-text reconstruction tasks, the constructed reconstruction loss is... The cross-entropy of the autoregressive conditional distribution of the real text attribute token sequence output by the text decoder; The text decoder takes the graph structure representation mapped by the graph feature alignment network as input, and the local real text attribute sequence as input. Based on the input, it decodes and generates the autoregressive conditional distribution of the real text attribute token sequence in sequence.

9. A system for constructing a general pre-trained graph structure large model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the method for constructing a general pre-trained graph structure large model as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the method for constructing a general pre-trained graph structure large model as described in any one of claims 1-8.

Images