A method and system for constructing a heterogeneous graph representation learning model based on mixed experts

Abstract

The application discloses a heterogeneous graph representation learning model construction method based on mixed experts, and belongs to the technical field of artificial intelligence. The method extracts node features and node type identifiers from a heterogeneous graph, precomputes structure position encoding and neighborhood relationship context vectors; then, the features are sent into a deep architecture alternately stacked by a global Transformer and a local MoE-GNN, the local module is driven by a multidimensional heterogeneous perception gating network, the routing score is calculated by combining node features, types and neighborhood context, and the node is dynamically dispatched to a GNN expert pool with non-shared parameters to execute local topology feature aggregation; finally, the model construction is completed through end-to-end training. While retaining the long-range dependence capturing capability, the application realizes adaptive modeling of local heterogeneous specificity, significantly improves the accuracy of heterogeneous graph representation learning through the multidimensional gating mechanism of node types and topological environment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and graph neural network technology, and in particular to a method and system for constructing a heterogeneous graph representation learning model based on hybrid experts. Background Technology

[0002] With the widespread application of artificial intelligence technology in complex decision-making systems (such as financial risk control, social network analysis, and industrial code analysis), real-world systems often need to be constructed as heterogeneous graph data containing multiple node types and relationship types. In these complex heterogeneous graph systems, different types of entities often contain drastically different logical semantics and have varying requirements for the scope of information capture. For example, in bank transaction records, large corporate transfer patterns and personal consumption patterns differ fundamentally in behavioral logic.

[0003] Traditional graph neural networks (such as GCN) [1] GIN [2] GAT [3] GatedGCN [4] When processing graph-structured data, methods such as [list of methods] employ a message-passing paradigm, updating the central node representation by iteratively aggregating features of neighboring nodes. While these methods are highly effective for homogeneous graphs, they suffer from inherent limitations in heterogeneous graphs: firstly, all nodes share the same set of parameters and aggregation logic, making it impossible to dynamically adjust the modeling strategy based on node type; secondly, message passing is limited by the local receptive field, and stacking multiple layers can lead to oversmoothing and information decay, making it difficult to capture long-range dependencies.

[0004] To overcome the locality limitations of traditional GNNs, many studies in recent years have attempted to introduce Transformer architectures (such as SAN). [5] Graphormer [6] Transformers, through their self-attention mechanism, allow direct interaction between any two nodes and possess the ability to capture global dependencies. However, pure Transformer architectures often neglect the fine-grained local topological structure of graphs, performing poorly in tasks requiring characterization of node neighborhood patterns. Therefore, alternating hybrid architectures (such as GraphGPS) are often used. [7] TransGNN [8] A method was proposed that uses alternating stacking of Transformer and GNN to achieve collaborative modeling of global dependencies and local topology.

[0005] To overcome the capacity limitations and homogeneous representations of single graph models on complex data, researchers have begun to introduce the Mixture of Experts (MoE) mechanism into the field of graph learning. The core idea of ​​MoE is to dynamically select several expert modules for each input through a gating network, achieving a divide-and-conquer modeling strategy. A representative work is GMoE. [9] This paper introduces MoE (Modal-based Evolution) into graph learning for the first time. It dynamically selects the most suitable expert for each node's neighborhood pattern using multiple non-shared parameter GNN experts and a gating mechanism. The gating network takes the features of the central node as input, calculates the routing score using MLP (Mean-Low Path Programming), and selects the top-k experts for a weighted sum as the local aggregation output for that node. Based on this, MoE-HCT...

[10] By introducing node type labels into a gating network and explicitly injecting node type embeddings to activate specific experts, node type specialization is initially achieved. The input to the gating network is a fusion vector of node features and type embeddings, and the underlying layer uses a heterogeneous graph attention network for first-order neighborhood aggregation.

[0006] While existing graph representation learning techniques have made some progress in processing complex data, they still have the following shortcomings when dealing with large-scale heterogeneous graphs with complex semantics:

[0007] First, existing global-local alternating architectures lack heterogeneity-specific modeling capabilities, leading to homogenization of local features. In recent years, hybrid architectures that alternately stack Transformer and GNN layers have proven effective in combining the advantages of both. However, the GNN module responsible for extracting local topological features in these architectures still has a single, fixed structure; that is, all nodes use the same GNN aggregator for local feature extraction, which cannot adapt to the diverse node groups in heterogeneous graphs. This globally-focused, locally coarse-grained local modeling approach results in insufficient accuracy in local feature extraction, limiting the overall expressive power of the model.

[0008] Secondly, the gating mechanisms of existing graph hybrid expert models lack the ability to perceive the micro-topological environment, resulting in low expert assignment accuracy. Existing graph models incorporating hybrid expert mechanisms typically rely solely on the original features of nodes or simply introduce static node type labels when calculating routing scores, completely ignoring the real micro-topological environment in which the nodes reside. This single-dimensional gating network, lacking relational context, prevents the model from accurately discerning the logical subgraph environment in which a node is situated. Consequently, it cannot provide highly specific dynamic assignment of local experts to similar nodes in different complex connectivity environments, limiting the accuracy of heterogeneous graph representation learning.

[0009] Finally, existing technologies lack an organic integration of global vision capture and local heterogeneous perception. Existing heterogeneous graph hybrid expert methods introduce type-aware expert selection mechanisms at the feature mapping layer, but the underlying architecture remains based on local heterogeneous graph attention networks, limited to attention aggregation within first-order neighborhoods, and unable to capture long-range dependencies across multi-hop topologies. While existing alternating hybrid architectures achieve synergy between global vision and local modeling, their local modules lack heterogeneous perception capabilities. Both technical approaches address only one aspect of the problem, failing to achieve an organic fusion of global vision and local heterogeneous perception. Summary of the Invention

[0010] Terminology Explanation:

[0011] 1. Heterogeneous graph: refers to a graph data structure that contains two or more node types and / or two or more edge (relationship) types. For example, an academic network may contain multiple node types such as author, paper, conference, etc., and multiple edge types such as write, cite, publish. In real-world scenarios, different types of nodes often represent entities of different natures and have different feature spaces and behavioral logics.

[0012] 2. Graph Neural Networks (GNNs): A deep learning architecture specifically designed for processing graph-structured data. Its core mechanism is message passing, which involves iteratively aggregating the features of a node itself and its neighbors to update the feature representation of the target node, thereby capturing local topological information of the graph. Common GNN models include GCNs (Graph Convolutional Networks), GINs (Graph Isomorphic Networks), and GATs (Graph Attention Networks).

[0013] 3. Transformer: A neural network architecture based on a self-attention mechanism that captures long-range dependencies by allowing direct interaction between any two positions in a sequence. In this invention, the Transformer is used to capture global dependencies between graph nodes, overcoming the limitations of the local receptive field of GNNs.

[0014] 4. Hybrid Expert Model (MoE): A sparsely activated neural network architecture consisting of a gating network and multiple independent subnetworks (called experts). For each input sample, the gating network dynamically calculates the routing score based on the input features, while only a few of the most relevant experts are sparsely activated for computation. Their outputs are then fused according to weights to obtain the final output, thereby improving the overall expressive power of the model without significantly increasing the computational cost of a single inference.

[0015] 5. Gated Network: In the MoE architecture, this is the sub-network responsible for calculating the activation weights of each expert. For the current node, the gated network outputs the node's activation scores for each expert based on its input information (such as node features, type, neighborhood context, etc.), and selects the experts with the highest scores for weighted aggregation.

[0016] 6. Position Encoding: A vector generated from the global topological properties of the pre-computed graph. In this invention, by adding it to the node features in the Transformer module and inputting it, it can serve as a structural prior to help the Transformer, which is not originally aware of the graph structure, to perceive the relative position of nodes in the global graph.

[0017] 7. Neighborhood Relationship Context Vector: A concept proposed in this invention, referring to a low-dimensional fixed-length vector obtained by statistically encoding the distribution of various types of nodes within the first-order neighborhood of a target node. This vector explicitly quantifies the local micro-topological environment in which the node resides.

[0018] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for constructing a heterogeneous graph representation learning model based on hybrid experts. This addresses the problems of homogenization of local GNN feature extraction in existing alternating hybrid architectures, lack of micro-topological environment perception capability in the gating mechanism of existing graph hybrid expert models, and the inability of existing technologies to organically combine global vision capture and local heterogeneous perception.

[0019] When dealing with complex, large-scale heterogeneous graph data, existing graph representation learning methods suffer from two major technical drawbacks. First, the local feature extraction in alternating architectures is homogenized. Existing alternating architectures, which integrate global attention and local graph neural networks, often employ a fixed, parameter-shared single expert network for their local GNN modules. Since nodes in heterogeneous graphs may belong to different categories and reside in different local subgraph structures, their aggregation requirements are inherently different. This homogenized design of a single-structure GNN cannot dynamically adjust the modeling strategy according to the diverse node types in the heterogeneous graph, resulting in coarse local feature extraction. Second, the gating mechanism of graph hybrid experts lacks awareness of the micro-environment. Existing graph models that introduce hybrid expert mechanisms often rely solely on the static features or identity labels of the central node when calculating expert routing scores, neglecting the real topological micro-environment in which the node is located—that is, lacking awareness of the context of neighborhood relationships. Due to this lack of information, when two nodes have the same type label but are in completely different local subgraph environments, the gating network cannot distinguish their differences and will activate the same expert for them. This results in the expert selection not matching the actual environment of the node, further limiting the model's overall expressive power on heterogeneous graphs.

[0020] To address the shortcomings of existing technologies, this invention aims to provide a method for constructing heterogeneous graph representation learning models based on hybrid experts. This invention seeks to overcome the representational bottleneck of traditional single-expert graph networks by constructing a model architecture that alternately stacks a global Transformer and a heterogeneous-aware MoE-GNN. This achieves high-precision collaborative modeling of local heterogeneous specificities while capturing global macro-topological dependencies. Furthermore, it reconstructs existing static isolated gating mechanisms and proposes a multi-dimensional heterogeneous-aware gating network. By jointly using the original features, type embeddings, and neighborhood relation context vectors of input nodes, the model is endowed with the ability to dynamically perceive local micro-topological environments. This allows for accurate matching of the most suitable feature aggregation experts for different nodes, solving the problem of expert mismatch caused by missing environmental information and significantly improving the overall accuracy and expressive power of the model in complex heterogeneous graph scenarios.

[0021] This invention proposes a method and system for constructing a heterogeneous graph representation learning model based on hybrid experts. The method first extracts basic node features and node type identifiers from the input heterogeneous graph data, and pre-computes structural position codes and neighborhood relation context vectors. Then, these features are fed into a deep architecture that decouples and alternately stacks a global Transformer and local MoE-GNN. The global module is used to capture macroscopic long-range trends across topological limitations; the local module is driven by a multi-dimensional heterogeneous perception gating network, jointly calculating routing scores based on node features, types, and neighborhood context, and dynamically dispatching nodes to a GNN expert pool with non-shared parameters to perform local topological feature aggregation. Finally, through multi-layer alternating collaboration and end-to-end backpropagation fine-tuning, the heterogeneous graph model is constructed.

[0022] The proposed method for constructing a heterogeneous graph representation learning model based on hybrid experts includes the following steps:

[0023] S1. Graph Data Acquisition and Preprocessing: Acquire raw heterogeneous graph data, construct node attribute feature tensors and adjacency matrices; calculate structural position encoding for all nodes in the graph, and simultaneously traverse the first-order neighbor receptive field of each target node, count the connectivity distribution frequency of different types of neighbor nodes, and generate the neighborhood relation context vector of the node.

[0024] S2, Three-element driven local gate scoring: Enter the l-th alternating processing layer, construct a learnable single-layer linear heterogeneous perception gate network containing an adaptive noise branch, embed the node type obtained by learning the mapping of the node's current input features and corresponding category labels, and concatenate the neighborhood relation context vector constructed in S1 in the feature dimension, and calculate the activation distribution score of the current node in the local expert pool corresponding to each expert through the Softmax function.

[0025] S3. Local MoE-GNN Expert Pool Micro-Specific Aggregation: A local module is constructed with k GNN experts whose parameters are not shared. These experts are isomorphic in terms of macro-network type, all using the same GCN network structure. However, the weight matrix parameters of each expert are independently initialized and not shared. Only through parameter independence and the diversity of random initialization is the convergence of each expert to different local feature subspaces during gradient descent, i.e., adaptive differentiation into different aggregation modes. Experts are selected using a TopK strategy based on the activation distribution scores in S2. The sparsely activated local GNN experts only perform neighbor information aggregation within a finite number of hops along the underlying real physical topology edges. The outputs of each activated expert are combined with the activation distribution scores for weighted summation, capturing highly specific micro-level fine node features. This invention uses an alternating architecture; a single-layer MoE-GNN only performs first-order neighbor aggregation, while long-range dependencies are delegated to the subsequent Transformer. Therefore, the neighbor information aggregation within a finite number of hops is a single-hop (1-hop) first-order neighbor information aggregation.

[0026] S4. Global attention long-range modeling with injected structural priors: Receive the local micro-level fine node features output from S3, add and fuse the pre-computed global graph structure position encoding in S1 with these local features, and then feed them into the global Transformer layer; capture the global macro trend by calculating the long-range attention interaction score between any pair of nodes in the graph, and generate the final global enhanced output representation of this layer.

[0027] S5, Hierarchical Stacking Loop and Task Joint Fine-tuning: The local processing of S2 to S3 and the global processing of S4 are treated as a concatenated sub-unit, stacked cyclically through L layers; the two modules are executed sequentially and alternately, with the output of the previous module serving as the input of the next module, and the output of the next module serving as the input of the next layer of MoE-GNN; residual connections and hybrid normalization are embedded within and between modules; in terms of the normalization strategy, each normalization operation occurs independently within each module, i.e., the local GNN expert contains batch normalization, the global Transformer contains layer normalization, and the module... No additional layer normalization is set between blocks; residual connections occur after module processing and include two layers of connections: the first layer is module-level residuals, which add and fuse the output features of MoE-GNN with the output features of Transformer; the second layer is layer-skip residuals, which directly add the initial input features of the current alternating layer with the module-level residual output after activation function processing to form the final cross-layer fusion representation; finally, the representation of the global Transformer output of the Lth layer is fed into the predictor, and the loss function is calculated in combination with the graph task's true label, and end-to-end model optimization is completed through backpropagation.

[0028] The structural position encoding in S1 of this invention includes random walk position encoding and Laplace position encoding.

[0029] The Laplace position code is calculated as follows:

[0030] First, calculate the degree matrix D and adjacency matrix A of the graph; then calculate the Laplacian matrix of the graph. We perform eigenvalue decomposition on L and take the eigenvectors corresponding to the m smallest non-zero eigenvalues ​​as the Laplacian position codes of each node.

[0031] The random walk position code is calculated as follows:

[0032] First, calculate the transition matrix. For node i, the characteristic value of the random walk at step t is This refers to the probability of returning to node i after t random walks starting from node i. After obtaining the feature sequence of length L_walk, the step index with the highest probability is taken as the category, and mapped to the random walk position encoding through a learnable embedding layer. ;

[0033] The neighborhood relation context vector in S1 of this invention is a low-dimensional fixed-length vector obtained by statistically encoding the distribution of each type of node in the first-order neighborhood of the target node.

[0034] The heterogeneous gating network in S2 of this invention takes as input a concatenated vector of node features, node type embedding, and neighborhood relation context vector, and outputs as the activation probability distribution of each expert.

[0035] In the S3 of this invention, the GNN experts are graph convolutional network layers with independently initialized parameters, and the parameters are not shared among the experts.

[0036] In S3 of this invention, the TopK strategy is used to select the K experts with the highest activation scores, satisfying the following conditions: , where E is the total number of experts in the expert pool.

[0037] The loss function in S5 of this invention is the total loss. ;in, For mission losses, Let λ be the load balancing loss, and let λ be the load balancing loss weighting coefficient. The load balancing loss is the sum of the gating probabilities of each expert and the sum of the squared coefficients of variation of the noisy expected selection rate.

[0038] The residual connection in S5 of this invention is a skip connection that spans two levels: local processing and global processing. It directly adds and fuses the module input features with the Transformer output features. Specifically, the residual connection includes two fusion paths: the first is module-level residual, which directly adds the output features of the local MoE-GNN module with the output features of the global Transformer module; the second is layer-level skip residual, which adds the initial input node features of the current alternating layer with the module-level residual output after processing by the nonlinear activation function, forming the final cross-layer fused representation.

[0039] Based on the above, this invention also proposes a heterogeneous graph representation learning system based on hybrid experts, comprising:

[0040] The multidimensional prior and feature construction module is used to extract the node semantic matrix and the edge adjacency matrix from the original heterogeneous graph, and to calculate the node position encoding and the neighborhood relationship context features that represent the micro-connectivity frequency.

[0041] The microscopic local heterogeneous feature extraction module includes a ternary fusion gate composed of a noisy single-layer linear mapping network and a hybrid expert pool composed of k parallel graph neural networks with non-shared parameters. The ternary fusion gate takes node features, node type embeddings, and neighborhood relation context vectors as input to calculate expert assignment probability weights. The hybrid expert pool dynamically routes nodes to the corresponding GNN experts according to the assignment probability weights and performs topological feature message passing within the graph neighborhood.

[0042] The long-range macro-attention modeling module encapsulates a multi-head self-attention computation network, which receives refined node features from local modules, fuses structural position encoding, and then executes global long-range attention interactions.

[0043] The model alternation coordination and gradient training module is used to manage the depth alternation stacking order from local network layers to global network layers, embed layer normalization and jump residual paths in the network, combine the target loss calculated by the task prediction layer, and control the backpropagation gradient update of the entire neural network architecture.

[0044] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the heterogeneous graph representation learning model construction method based on hybrid experts described in the present invention.

[0045] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the heterogeneous graph representation learning model construction method based on hybrid experts described in the present invention.

[0046] Compared with existing technologies, the heterogeneous graph representation learning model construction method and system based on hybrid experts proposed in this invention have the following advantages:

[0047] To address the shortcomings of existing graph alternation processing models, which often employ monotonous or fixed non-heterogeneous perception mechanisms in the local feature aggregation stage, leading to homogenization of complex graph topological feature representations, this invention innovatively proposes a dimensionality-upgraded collaborative architecture consisting of a local heterogeneous multi-expert pool followed by a global self-attention network. In the local modules of the alternation architecture, a multi-expert MoE-GNN replaces the original single-expert fixed GNN. Each expert's parameters are independent and not shared. Through different random initializations, the experts naturally form differentiated aggregation patterns during training. A gating network dynamically selects the most suitable expert combination for different nodes.

[0048] Since the node groups in heterogeneous graphs have different local structural characteristics, a single fixed GNN cannot simultaneously meet the needs of all groups. By adopting a multi-expert MoE, different experts can focus on different types of local patterns, and differentiated aggregation can be achieved through the routing of the gating network. This enables the local modules of the alternating architecture to gain adaptive modeling capabilities for diverse node groups.

[0049] This invention upgrades heterogeneous perception with an alternating architecture, proposing a collaborative modeling architecture that alternately stacks a global Transformer and local MoE-GNN. It breaks through the limitations of existing frontier alternating models where local GNNs share parameters and are homogeneous under a single expert model. By sparsely activating specific GNN experts, the local modules of the alternating architecture possess adaptive modeling capabilities for diverse node groups, achieving complementary modeling of global long-range dependencies and microscopic heterogeneous specificity.

[0050] This invention proposes a multi-dimensional micro-environment perception gating mechanism. It reconstructs the routing decision space of a hybrid expert model and constructs a ternary joint gating network that integrates original node features, node type embeddings, and neighborhood relation context vectors. This overcomes the shortcomings of existing gating systems that isolate the central node, enabling the model to dynamically match local feature extraction strategies based on the real micro-logical subgraph environment.

[0051] This invention addresses the shortcomings of existing graph network routing or gating mechanisms, which rely solely on node attribute scoring and lack dynamic perception of the micro-connected subgraph environment (neighborhood distribution). This invention designs a gating network with micro-environment perception. By introducing neighborhood context vectors into the network input, the gating network can simultaneously perceive node type information (type embedding) and environmental information (neighborhood type distribution), achieving an upgrade from single-dimensional identity perception to multi-dimensional environment perception.

[0052] Because gating networks are aware of the local topological environment (the distribution of various types of nodes in the first-order neighborhood) of a node during decision-making, when two nodes of the same type are in different environments, the gating network can assign different experts to them, rather than indiscriminately activating the same group of experts. This allows each node to obtain an aggregation result that better matches its actual local pattern, thereby improving the accuracy of local feature extraction.

[0053] This invention addresses the limitations of existing graph models, which often fall into a single paradigm: simple graph neural networks are limited by the number of message-passing layers and cannot capture global long-range dependencies; simple graph Transformers lack graph topological inductive bias and neglect local heterogeneous feature extraction; and some existing hybrid models simply mechanically combine the two, resulting in insufficient refinement of both global and local feature extraction. This invention innovatively proposes a collaborative modeling architecture that alternately stacks heterogeneous perception MoE-GNN and global Transformer. Within the same model framework, a complementary mechanism is constructed that first refines microscopic local features and then macroscopically captures long-range interactions, achieving complementary modeling of the two core capabilities of local heterogeneous perception and global vision capture.

[0054] Because large-scale heterogeneous graphs contain both complex underlying local topological logic and macroscopic long-range dependencies that transcend hop count limitations, a single computational paradigm cannot adequately address both. By adopting an alternating stacking architecture, the global Transformer layer no longer receives coarse, homogeneous node representations, but rather high-quality local features that have undergone deep refinement and denoising by heterogeneous experts. Simultaneously, global self-attention provides a direct information path for cross-regional nodes, overcoming the excessive smoothing bottleneck caused by deep stacking in traditional graph networks. The orderly collaboration and complementary optimization of these two types of modules within the framework enable the model to achieve a comprehensive representational capability that can both identify microscopic heterogeneous specificities and control global long-range dependencies.

[0055] To verify the effectiveness of this invention, the heterogeneous graph representation learning model (TransGMoE) based on hybrid experts constructed in this invention underwent rigorous empirical evaluation on multiple datasets of the Open Graph Framework (OGB) benchmark and was compared with the closest existing technology, GMoE. Under the objective standard of training for 80 epochs on the training set and averaging the results of more than five independent experiments, this invention achieved significant technical progress. The experimental results are shown in the table below:

[0056] Dataset Task type Evaluation indicators GMoE TransGMoE (This invention) ogbg-molhiv Two categories ROC-AUC 0.7354 0.7786 ogbg-molesol return RMSE (the lower the better) 1.2816 1.0113 ogbg-moltoxcast Multi-tag classification ROC-AUC 0.6046 0.6471

[0057] On the OGBG-MOLOV dataset (where positive samples account for only about 3%), the ROC-AUC of this invention is improved by about 5.9% compared to GMoE, indicating that this invention has a significant performance advantage on extremely imbalanced data. On the OGBG-MOLESOL graph regression task dataset, the RMSE of this invention reaches 1.0113, which is about 21.1% lower than GMoE, indicating that this invention also has a significant accuracy improvement in regression tasks. On the OGBG-MOLESOL-CAST dataset (multi-label classification task, 617 sub-tasks), the ROC-AUC of this invention reaches 0.6471, which is about 7.0% higher than GMoE, indicating that this invention is also effective in multi-label classification prediction scenarios. The heterogeneous graph representation learning method proposed in this invention is also applicable to graph data containing a single node type but with significant heterogeneity in node attributes (i.e., there are essential differences between nodes). In the field of graph neural networks, molecular graphs are often treated as isomorphic graphs. However, in reality, molecular graphs contain different types of atoms (such as carbon C, oxygen O, and nitrogen N), and different atoms themselves have different chemical properties and physical meanings. These differences are explicitly expressed through the atomic type encoding (such as atomic number, chirality, etc.) in the node features. This invention treats atomic type as a node type identifier, and regards molecular graphs containing different atomic properties and chemical bond types as a typical heteromorphic graph. Therefore, in this scenario, "heterogeneous perception" perceives the differences in the chemical identity of nodes, rather than the differences in the node types of the graph structure.

[0058] The experimental data above demonstrate that this invention consistently outperforms the closest existing technology, GMoE, in classification, regression, and multi-task scenarios, verifying the effectiveness and generalization ability of the proposed solution. Furthermore, the average convergence performance over a stable 80 epochs on the test set shows that this scheme significantly expands the overall parameter expression space and performance ceiling of the deep graph model while effectively avoiding gradient explosion and feature redundancy caused by dense fully connected computation. This ensures high convergence rate and robust stability during deep learning training, demonstrating significant engineering application value. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1The flowchart shows the method for constructing a heterogeneous graph representation learning model based on hybrid experts according to the present invention. It clearly shows the data flow from multi-dimensional prior preprocessing, micro-level local heterogeneous multi-expert aggregation, to macro-level long-range self-attention capture, which is a deep alternating learning process of local and then global depth.

[0061] Figure 2 This is a block diagram illustrating the architectural principle of the heterogeneous graph representation learning model based on hybrid experts in this invention. It details the tensor flow and module coupling relationships between the bottom-level input feature layer, the first-level local heterogeneous multi-expert extraction module (including a heterogeneous perceptual routing gate and a MoE-GNN expert pool), the second-level long-range macroscopic capture module (global Transformer), and the cross-level residual connections. Detailed Implementation

[0062] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.

[0063] This invention discloses a method for constructing a heterogeneous graph representation learning model based on hybrid experts, belonging to the field of artificial intelligence technology. This method extracts node features and node type identifiers from heterogeneous graphs, pre-computes structural position encodings and neighborhood relation context vectors; then, the features are fed into a deep architecture that alternately stacks global Transformers and local MoE-GNNs. The local modules are driven by multi-dimensional heterogeneous perception gating networks, jointly calculating routing scores based on node features, types, and neighborhood context, and dynamically dispatching nodes to a GNN expert pool with non-shared parameters to perform local topological feature aggregation; finally, the model is constructed through end-to-end training. This invention, while retaining the ability to capture long-range dependencies, achieves adaptive modeling specific to local heterogeneity, and significantly improves the accuracy of heterogeneous graph representation learning through a multi-dimensional gating mechanism based on node type and topological environment.

[0064] The proposed method for constructing a heterogeneous graph representation learning model based on hybrid experts includes the following steps:

[0065] Step 1: Graph Data Acquisition and Preprocessing. Acquire the raw heterogeneous graph data and construct node attribute feature tensors and adjacency matrices. Calculate structural positional encoding for all nodes in the graph, including random walk positional encoding and Laplace positional encoding. Simultaneously, traverse the first-order neighbor receptive field of each target node, count the connectivity distribution frequency of different categories of neighboring nodes, and generate the neighborhood relation context vector for that node.

[0066] Step 2: Three-element driven local gated scoring. Entering the l-th alternating processing layer, local heterogeneous extraction is initiated first. A learnable single-layer linear heterogeneous perception gated network with an adaptive noise-adding branch is constructed. The node's current input features, the node type obtained by the learnable mapping of the corresponding category label, and the neighborhood relation context vector constructed in Step 1 are concatenated along the feature dimension. The activation distribution score of the current node in the local expert pool is calculated using the Softmax function.

[0067] Step 3: Local MoE-GNN Expert Pool Micro-Specific Aggregation. A local module is constructed containing k GNN experts with mutually exclusive parameters. Experts are selected using a TopK strategy based on the activation distribution scores from Step 2. The sparsely activated local GNN experts perform neighbor information aggregation only along the underlying real physical topology edges within a finite number of hops. The outputs of each activated expert are combined with their activation distribution scores and weighted summed. The output captures highly specific micro-level, fine-grained node features.

[0068] Step 4: Injecting global attention long-range modeling with structural priors. Receive the local micro-level fine-grained node features output from Step 3, add and fuse the pre-computed global graph structure location encoding from Step 1 with these local features, and then feed them into the global Transformer layer. By calculating the long-range attention interaction scores between any pair of nodes in the graph, the local receptive field limitation of graph structure message passing is broken, global macro-trends are captured, and the final globally enhanced output representation of this layer is generated.

[0069] Step 5: Layered Stacking and Joint Task Fine-tuning. Steps 2 to 3 (local processing) and Step 4 (global processing) are treated as a concatenated sub-unit and stacked cyclically through L layers. The two modules execute sequentially and alternately; the output of the previous module is the input of the next module, and the output of the next module is the input of the next layer of the MoE-GNN. Residual connections and hybrid normalization are embedded within and between modules. In terms of the normalization strategy, each normalization operation occurs independently. Within each module—that is, within the local GNN expert module, batch normalization is included; within the global Transformer module, layer normalization is included; and so on. No additional layer normalization is applied between blocks; residual connections occur after module processing and consist of two layers: the first layer is a module connection. Block-level residuals are used to fuse the output features of MoE-GNN with those of Transformer; the second layer involves hierarchical jumps. The residual is obtained by directly adding the initial input features of the current alternating layer to the module-level residual output after processing by the activation function. This forms the final cross-layer fusion representation; Finally, the representation output by the Lth layer global Transformer is fed into the predictor, and the loss function is calculated by combining it with the true labels of the graph task. End-to-end model optimization is then completed through backpropagation.

[0070] Based on the above methods, this invention also proposes a heterogeneous graph representation learning system based on hybrid experts, comprising:

[0071] Multidimensional Prior and Feature Construction Module: As an input tensor processing unit, it is used to extract the node semantic matrix and the edge adjacency matrix from the original graph, and to calculate the node position encoding and the neighborhood relation context features that represent the micro-connectivity frequency.

[0072] The microscopic local heterogeneous feature extraction module, located at the pre-order level of the forward propagation architecture, comprises a ternary fusion gate consisting of a noisy single-layer linear mapping network and a hybrid expert pool consisting of parallel graph neural networks with k non-shared parameters. This module focuses on capturing the microscopic specificities of heterogeneous graphs through feature distribution and neighborhood aggregation. The ternary fusion gate takes the concatenated tensor h of node features, node type embeddings, and neighborhood relation context as direct input, utilizing the main gating weight matrix W. gate With noise weight matrix W noise The parallel calculation of the noisy routing score is given by the following formula:

[0073] ,

[0074] in, For a random Gaussian noise matrix that follows a standard normal distribution, A very small constant to prevent numerical overflow. This indicates element-wise multiplication. Ultimately, the network calculates a score for this noisy addition. Top-K truncation is used and normalized by the Softmax function to output expert assignment probability weights; the hybrid expert pool dynamically routes nodes to the corresponding GNN experts according to these probability weights, and performs topological feature message passing in the graph neighborhood.

[0075] The long-range macro-attention modeling module is located at the end of the forward propagation architecture and internally encapsulates a multi-head self-attention computation network. Its feature input channels connect to the aggregated feature output channels of the local modules, focusing on upscaling refined local features to the global long-range dependency space.

[0076] Model Alternation Coordination and Gradient Training Module: Responsible for managing the depth alternation stacking order of the aforementioned local network layers to global network layers, while embedding layer normalization and jump residual paths in the network, combining the target loss calculated by the task prediction layer, and controlling the backpropagation gradient update of the entire neural network architecture.

[0077] Figure 1This demonstrates the core execution flow of model construction. At the top is the input original heterogeneous graph, with a downward arrow pointing to step 1: graph data acquisition and preprocessing, completing basic feature extraction, structural position encoding calculation, and first-order neighborhood relation context vector statistics. The data then flows into the alternating module loop (a total of L layers). Inside the alternating layer, the data first directly enters step 2: the ternary joint gating scoring unit, where probabilities are calculated jointly by feature, type embedding, and relational context. The gating signal and the current feature are then sent to step 3: the local MoE-GNN expert aggregation pool, where micro-neighbor aggregation is performed along graph edges. After the local refined output features are merged, they flow directly to step 4: the global Transformer unit, where pre-calculated position encoding priors are fused and global long-range interactions are performed. If L layers are not reached, the data returns to step 2 via the residual bus for a new loop. After reaching the preset number of layers, the data finally flows to the bottom step 5: the task fine-tuning and classification prediction module.

[0078] Figure 2 This paper details the data flow and coupling relationship between the l-th alternating layer and the neural network module in this invention. Input Feature Layer: The bottom layer of the architecture receives four sets of tensor data from the previous processing module, including node output features, node type embeddings, neighborhood relation context vectors, and a graph topological adjacency matrix. First Layer (Local Heterogeneous Multi-Expert Extraction Module (Local Level)): This is the bottom-level feature extraction stage of the model architecture. Node features, node type embeddings, and neighborhood relation context vectors are concatenated along the feature dimension and used as input to the heterogeneous perceptual gating network to calculate the assigned probability weights. These weights, along with the current node features, are passed to the local MoE-GNN expert aggregation pool. This pool consists of k parallel local GNN expert networks with mutually exclusive parameters. Combined with the graph topological adjacency matrix, it performs micro-topological feature message passing within the graph neighborhood. The feature mapping results of the activated experts are sent to the weighted summation fusion unit, where they are fused according to the assigned weights to generate locally refined node features. Second Layer (Global Long-Range Attention Capture Module (Global Level)): This is the upper-level feature aggregation stage of the model architecture. The locally refined node features output from the first layer are directly injected into the feature fusion layer of the previous layer through forward propagation. They are then added to and fused with the pre-trained structural position encoding of the graph. Subsequently, they are fed into a self-attention network (global Transformer), which calculates the global association matrix between graph nodes through the self-attention mechanism, updating and generating a representation with a long-range macroscopic perspective. Residual and Output Layers: In the deep stacked structure of the model, the node output features of the bottom layer are directly added to and fused with the output tensor of the top-level self-attention network through a residual jump connection spanning two processing layers. Finally, this fused feature is output as the fusion feature of the l-th alternating learning layer to the next network layer.

[0079] Working principle of the architecture of this invention:

[0080] In the process of model inference and forward computation, the deep learning model of this invention follows the feature representation law of low-level micro-specific deconstruction and high-level global long-range fusion:

[0081] Micro-specific dispatching and aggregation (local prior): When tensor data flows through the deep model, heterogeneous perception gating jointly measures the feature space, identity attributes, and immediate neighbor environment (relational context) of nodes, breaking the limitations of traditional graph models in handling homogeneous nodes. The model uses calculated allocation weights to dynamically route nodes to corresponding local GNN experts, performing fine-grained local graph topology message aggregation and outputting high-quality local representations that eliminate homogeneous noise.

[0082] Long-range macroscopic semantic dimensionality enhancement (global fusion): The feature tensor, which has been refined to a fine local level, is then computed along the forward graph and enters the self-attention layer. The model injects graph structure position encoding and uses the attention mechanism to break the constraint of neighbor hop count pair feature propagation, establishing global interaction between local nodes that already possess heterogeneous specificity.

[0083] Closed-loop training principle: The model architecture uses multiple layers of local and then global modules to alternate, so that the graph representation has both the high discrimination of the underlying micro-topology and the macro-view of the top layer across network regions, effectively solving the problem of not being able to take into account both local and global aspects in large-scale complex heterogeneous graph learning.

[0084] Example 1: TransGMoE model in the task of predicting chemical molecular properties

[0085] This embodiment uses the OGBG-molhiv dataset from the Open Graph Benchmark (OGB) as an example to detail the specific implementation process of this invention in the task of predicting the properties of chemical molecules. OGBG-molhiv is a graph-level binary classification dataset containing 41,127 molecular graphs. The task is to predict whether a molecule can inhibit HIV virus replication. This dataset exhibits extreme long-tail imbalance, with positive samples accounting for only about 3%.

[0086] 1. Experimental Environment Configuration

[0087] In this embodiment, the model is implemented based on the PyTorch deep learning framework and the PyTorch Geometric (PyG) graph neural network library, and training is accelerated on an NVIDIA GPU. The main software environment configuration is as follows:

[0088] Operating System: Ubuntu 20.04 LTS

[0089] Deep learning framework: PyTorch 1.12+

[0090] Graph Neural Network Library: PyTorch Geometric 2.2+

[0091] Other dependencies: OGB library (for data loading and evaluation metrics), NumPy, Scikit-learn

[0092] 2. Data preprocessing (corresponding to step 1)

[0093] First, the OGBG-MOLMHIV dataset is read using the standard data loading interface provided by OGB. Each molecular map contains the following raw information:

[0094] Node feature matrix Each node represents an atom, N is the number of atoms in the molecular diagram, and 9 is the dimension of the atomic features (including chemical features such as atom type, atomic number, and chirality).

[0095] An edge index matrix records the chemical bond connections between atoms;

[0096] The edge feature matrix records the equilateral attributes of chemical bond types;

[0097] Image Tag The value indicates whether the molecule has HIV inhibitory activity (1 for positive, 0 for negative).

[0098] For each molecular graph in the dataset, the present invention performs the following preprocessing operations:

[0099] Read node type information: In OGBG-MOLMIVAL, the node type is the atom type, which contains about 20 different atoms (such as carbon C, hydrogen H, oxygen O, nitrogen N, etc.). The atom type is mapped to a category index to form a node type label sequence.

[0100] Calculate the neighborhood context vector: for each atomic node in the graph The frequency distribution of each type of atom in its first-order neighborhood is statistically analyzed to generate a neighborhood context vector. Let the total number of atom types be C. The j-th component is a node. Atoms of type j in first-order neighbors The proportion of all first-order neighbors. If an atom is an isolated atom (without any chemical bonds), its neighborhood context vector is set to zero. In this embodiment, first-order neighbors are defined as the set of atoms directly connected to the atom by chemical bonds.

[0101] (1) Pre-compute Laplacian positional encoding: First, calculate the degree matrix and adjacency matrix of the graph; then calculate the Laplacian matrix of the graph. (Where D is the degree matrix and A is the adjacency matrix), perform eigenvalue decomposition on L, and take the eigenvectors corresponding to the m smallest non-zero eigenvalues ​​as the position codes of each atom to obtain the Laplace position codes. In this embodiment, the Laplace position code is calculated and cached once during the data preprocessing stage, and can be directly read during subsequent training without repeated calculation.

[0102] 3. Model parameter configuration

[0103] In this embodiment, the TransGMoE model uses the following hyperparameter configuration:

[0104] Parameter name Parameter value illustrate Alternating layers L 3 Alternating layers of MoE-GNN and Transformer Number of experts E 4 The MoE-GNN module contains four independent GNN experts. Top-K value K 2 Activate the two experts with the highest scores at each node. Hidden layer dimensions 150 Unified dimension of hidden layers in all modules Random walk step size L_walk 5 RWPE uses walking steps Laplace position encoding dimension m 3 The number of feature vectors used by LapPE Location encoding fusion dimension d 150 Consistent with the dimension of the hidden layer Load balancing loss weight λ 0.5 Auxiliary loss weighting coefficient Learning rate 0.001 Adam optimizer initial learning rate Number of training rounds 80 Total number of training epochs Dropout ratio 0.2 Preventing overfitting

[0105] Among the above parameters, the number of experts E=4, each expert is a GCN (Graph Convolutional Network) layer, and the parameters of each expert are initialized independently to ensure the diversity of the initial state.

[0106] 4. Model training (corresponding to steps 2-5)

[0107] In this embodiment, the specific process of model training is as follows:

[0108] (1) For each training batch, sample a batch of molecular maps from the dataset (batch size set to 32). For each molecular map in the batch, obtain the preprocessed data as described in step 1.

[0109] (2) Perform forward propagation in an alternating order of local to global:

[0110] In the first alternating layer (l=1), the initial node is represented. Input heterogeneous perceptual MoE-GNN module. Gated network uses node features Type embedding and neighborhood context vector The three inputs are concatenated and used as input. The activation scores of each expert are calculated through a noisy single-layer linear mapping network. The top k experts with the highest scores are selected for activation. The aggregated outputs of the activated experts are then summed in weighted order of their scores to obtain a locally refined node representation. .

[0111] Before entering the Transformer module, the merged positional encoding is first constructed and injected into the current node representation. The merged positional encoding is obtained by adding the following two parts:

[0112] ① Laplace position coding : Read directly from the graph data attributes pre-computed and stored in step 1. Transform it to the same dimension as the node representation through a linear projection layer. .

[0113] ② Random walk position coding Calculated dynamically during model training. First, the transition matrix is ​​calculated. For node i, the characteristic value of the random walk at step t (t=1,...,L_walk) is This represents the probability of returning to node i after t random walks starting from node i. After obtaining the feature sequence of length L_walk, the step index with the highest probability is taken as the category, and mapped to the random walk position encoding through a learnable embedding layer. .

[0114] The two position codes are added together to obtain the fused position code, which is then compared with the current node representation. The nodes are added together. Then, the position-encoded enhanced node representations are input into the global Transformer module. Through self-attention and residual connections, the globally enhanced node representations are obtained. , which serves as the input for the next alternating layer.

[0115] This process continues, with a total of L layers of alternating stacking. The node representation output by the last layer is... .

[0116] (3) The input readout layer uses global average pooling to obtain the graph-level representation vector. .

[0117] (4) Input z into the linear prediction layer and output the binary classification prediction probability. The Softmax activation function is used.

[0118] (5) Calculate the total loss ,in For cross-entropy loss, The load balancing loss is a penalty term calculated by combining the sum of the gating probabilities (importance) of each expert with the sum of the squared coefficients of variation of the noisy expected selection rate (load). The Adam optimizer is used for backpropagation and parameter updates.

[0119] (6) After each epoch, evaluate the model performance on the current validation set and record the ROC-AUC metric. A total of 80 epochs are trained, and the average ROC-AUC of 5 runs on the test set is taken as the final performance metric.

[0120] 5. Experimental Results and Effect Verification

[0121] Experimental results on the OGBG-MOTHIV dataset show that, compared to the closest existing technology GMoE, this invention improves the ROC-AUC from 0.7354 to 0.7786, representing a relative improvement of approximately 5.9%. These results validate the effectiveness of this invention on chemical molecular graph classification tasks with extreme long-tail imbalances.

[0122] Example 2: TransGMoE model in regression task

[0123] This embodiment uses the ogbg-molesol dataset in OGB as an example to illustrate the specific implementation of the present invention on a regression task. ogbg-molesol is a graph regression dataset whose task is to predict the logarithmic solubility of compounds in water, using root mean square error (RMSE) as the evaluation metric.

[0124] In this embodiment, except for the following configurations, the remaining model configurations are consistent with those in Embodiment 1:

[0125] Change the output dimension of the prediction layer to 1 (regression task);

[0126] The loss function is changed to mean squared error (MSE) loss;

[0127] The evaluation metric has been changed from ROC-AUC to RMSE, where a lower value indicates better model performance.

[0128] On the OGBG-Molesol dataset, the RMSE achieved by this invention is 1.0113, compared to 1.2816 for the closest prior art, GMoE. This represents an absolute reduction of approximately 0.2703 and a relative reduction of approximately 21.1% in RMSE. This demonstrates that the method of this invention also outperforms existing techniques in regression tasks.

[0129] Replacement example

[0130] Regarding the core technical solution of this invention, without departing from the overall inventive concept of dynamic routing based on micro-environment perception and global-local complementary modeling upgrade, the following alternative solutions do not change the basic implementation of this invention. They can replace the corresponding configurations in the foregoing embodiments under different application scenarios or resource constraints, and can also achieve the inventive purpose of this invention.

[0131] I. Replacement of Expert Types and Quantity Configuration

[0132] In some application scenarios, the type and number of experts in the aforementioned embodiments can be adjusted. Experts can be replaced with other types of GNN layers, such as GIN (Graph Isomorphic Network), GraphSAGE (Graph Sampling and Aggregation Network), GAT (Graph Attention Network), etc. Regardless of the GNN type used, each expert maintains an independent and non-shared parameter configuration, and the routing mechanism of the gating network remains unchanged. Those skilled in the art can select the most suitable GNN type as the expert based on the specific characteristics of the dataset.

[0133] For datasets with few node types and relatively simple structural patterns, the number of experts can be set to 2-4 to reduce the number of model parameters and computational cost, preventing overfitting. For datasets with rich node types and complex and diverse local structural patterns, the number of experts can be extended to 8-16 to enhance model capacity and expressive power, thereby improving local modeling accuracy. The selection of the number of experts E and the Top-K value K must satisfy the following conditions: To maintain sparse activation and computational efficiency.

[0134] II. Replacement of the Graph Structure Position Encoding Scheme

[0135] The preferred approach in this embodiment is to inject the fused Laplacian positional encoding and random walk positional encoding into the Transformer module. As an alternative, depending on the graph size and computational cost, the positional encoding can be replaced with either one, such as DeepWalk-based node embedding or a relative spatial shortest path distance matrix. Regardless of the encoding structure used, the goal is to supplement the graph Transformer with the structural inductive bias of the graph topology, preventing the loss of the original relative positional relationships of nodes during fully connected computation of global self-attention, thus achieving the effect of assisting in global long-range modeling.

[0136] III. Replacement of the Global Long-Term Dependency Capture Module

[0137] This scheme preferentially adopts the standard multi-head self-attention mechanism as the global Transformer module. When dealing with ultra-large-scale graphs with hundreds of thousands or millions of nodes, this global module can be replaced with a linear complexity Transformer (such as Performer, Linformer, etc.), or a graph-specific Transformer with a spatial shortest path bias. These linear attention mechanisms or graph-specific attention variants can still break the neighbor hop count limitation to achieve cross-domain interaction of global information, complete macroscopic complementarity with local modules, and further reduce the quadratic time complexity.

[0138] IV. Replacement of Neighborhood Context Vector Construction Method

[0139] The aforementioned embodiments use the frequency ratio of each type of node within the first-order neighborhood as the neighborhood context vector. As an alternative, the absolute frequency (rather than the ratio) of each type of node within the first-order neighborhood can be used, which is suitable for scenarios where the degree differences between different nodes are not significant and normalization is not required; it can also be extended to the second-order neighborhood, that is, statistically analyzing the distribution of each type of node within two hops of the target node, which is suitable for scenarios where the local structure is relatively sparse and the information from the first-order neighborhood is insufficient to characterize the topological environment; the distribution of each type of edge within the neighborhood can also be used instead of the distribution of node types, which is suitable for scenarios where edge type information is more discriminative than node type information.

[0140] The above alternative solutions do not change the basic implementation of the present invention: the core architecture of alternating stacking of global Transformer and heterogeneous perceptual MoE-GNN, and gating network with node feature / type embedding / neighborhood context joint input. Therefore, they can also achieve the purpose of the present invention and are all within the protection scope of the present invention.

[0141] References

[0142] [1] Kipf TN, Welling M. Semi-supervised classification with graphconvolutional networks[J]. arXiv preprint arXiv:1609.02907, 2016.

[0143] [2] Xu K, Hu W, Leskovec J, et al. How powerful are graph neuralnetworks?[J]. arXiv preprint arXiv:1810.00826, 2018.

[0144] [3] Veličković P, Cucurull C, Casanova A, et al. Graph attentionnetworks[J]. arXiv preprint arXiv:1710.05893, 2017.

[0145] [4] Bresson X, Laurent T. Residual gated graph convnets[J]. arXivpreprint arXiv:1711.07553, 2017.

[0146] [5] Kreuzer D, Beaini D, Hamilton W, et al. Rethinking graphtransformers with spectral attention[J]. Advances in Neural InformationProcessing Systems, 2021, 34: 21618-21629.

[0147] [6] Ying C, Cai T, Luo S, et al. Do transformers really perform badlyfor graph representation?[J]. Advances in Neural Information ProcessingSystems, 2021, 34: 28877-28887.

[0148] [7] Rampášek L, Galkin M, Dwivedi V P, et al. Recipe for a general,powerful, scalable graph transformer[J]. Advances in Neural InformationProcessing Systems, 2022, 35: 14501-14515.

[0149] [8] Zhang P, Yan Y, Zhang X, et al. TransGNN: Harnessing thecollaborative power of transformers and graph neural networks for recommendersystems[C] / / Proceedings of the 47th International ACM SIGIR conference onresearch and development in information retrieval. 2024: 1285-1295.

[0150] [9] Wang H, Jiang Z, You Y, et al. Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling[J]. Advances in NeuralInformation Processing Systems, 2023, 36: 50825-50837.

[0151]

[10] Mao Q, Sun J. Mixture of Experts Enhanced Heterogeneous GraphTransformer[C] / / International Conference on Knowledge Science, Engineering and Management. Singapore: Springer Nature Singapore, 2025: 92-103.

[0152] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0153] As used in this invention, the term "includes" is an open expression, meaning it includes the content specified in this invention but does not exclude other aspects.

[0154] As used in this invention, terms and / or include any one or more of the related listed items and all combinations thereof.

[0155] The scope of protection of this invention is not limited to the above embodiments. All variations and advantages that can be conceived by those skilled in the art within the technical scope disclosed in this invention are included in this invention and are protected by the appended claims.

Claims

1. A method for constructing a heterogeneous graph representation learning model based on hybrid experts, characterized in that, Includes the following steps: S1: Obtain the original heterogeneous graph data and construct the node attribute feature tensor and adjacency matrix; Calculate the structural location encoding for all nodes in the graph, and simultaneously traverse the first-order neighbor receptive field of each target node, count the connectivity distribution frequency of different types of neighbor nodes, and generate the neighborhood relation context vector of the node. S2: Enter the l-th alternating processing layer, construct a learnable single-layer linear heterogeneous perception gated network containing an adaptive noise branch, embed the node type obtained by learning mapping of the node's current input features and corresponding category labels, and concatenate the neighborhood relation context vector obtained in step S1 in the feature dimension, and calculate the activation distribution score of each expert in the local expert pool corresponding to the current node through the Softmax function. S3: Construct k GNN experts with mutually exclusive parameters in the local module, and select experts using the TopK strategy based on the activation distribution scores obtained in step S2; The sparsely activated GNN experts perform neighbor information aggregation within a finite number of hops along the underlying real physical topology edges. The outputs of each activated expert are combined with the activation distribution scores and weighted summation to output micro-level fine node features. S4: Receive the micro-level fine node features output from step S3, add and fuse the structure position encoding pre-calculated in step S1 with the local features, and feed it into the global Transformer layer. Capture the global macro trend by calculating the long-range attention interaction score of any node pair in the graph, and generate the global enhanced output representation of this layer. S5: The local processing of steps S2 to S3 and the global processing of step S4 are treated as a series sub-unit, and L layers are stacked in a loop. The two modules are executed sequentially and alternately. Residual connections and hybrid normalization are embedded within and between modules. The representation of the global Transformer output of the Lth layer is fed into the predictor, and the loss function is calculated in combination with the true label of the graph task. End-to-end model optimization is completed through backpropagation.

2. The method for constructing a heterogeneous graph representation learning model based on hybrid experts according to claim 1, characterized in that, The structural position encoding in step S1 includes Laplace position encoding and / or random walk position encoding; The Laplacian position encoding is calculated as follows: the degree matrix D and adjacency matrix A of the graph are calculated to obtain the Laplacian matrix. Perform eigenvalue decomposition on L, and take the eigenvectors corresponding to the m smallest non-zero eigenvalues ​​as the Laplacian position codes of each node. The random walk position encoding is calculated as follows: calculate the transition matrix. For node i, obtain the random walk return probability sequence from step 1 to step L_walk, take the step index with the highest probability as the category, and map it to the random walk position encoding through a learnable Embedding layer; And / or, The neighborhood relation context vector mentioned in step S1 is a low-dimensional fixed-length vector obtained by statistically encoding the distribution of each type of node in the first-order neighborhood of the target node.

3. The method for constructing a heterogeneous graph representation learning model based on hybrid experts according to claim 1, characterized in that, The GNN experts mentioned in step S3 are graph convolutional network layers with independently initialized parameters. Each expert adopts the same GNN network structure and is differentiated into different local feature aggregation modes through mutually independent weight matrix parameters. And / or, In step S3, the TopK strategy is used to select the K experts with the highest activation scores, satisfying the following conditions: , where E is the total number of experts in the expert pool.

4. The method for constructing a heterogeneous graph representation learning model based on hybrid experts according to claim 1, characterized in that, The loss function mentioned in step S5 is the total loss. ;in, For mission losses, Let λ be the load balancing loss and λ be the load balancing loss weighting coefficient. The load balancing loss is the sum of the gating probabilities of each expert and the sum of the squared coefficients of variation of the noisy expected selection rate. And / or, The residual connection described in step S5 consists of two layers: the first layer is module-level residual, which adds and fuses the output features of MoE-GNN with the output features of Transformer; the second layer is layer-skipping residual, which directly adds the initial input features of the current alternating layer to the module-level residual output after activation function processing to form a cross-layer fusion representation.

5. The method for constructing a heterogeneous graph representation learning model based on hybrid experts according to claim 1, characterized in that, The network structure of the GNN experts is replaced with any one of GIN, GraphSAGE, or GAT, while each expert still maintains an independent configuration with parameters that are not shared between them.

6. The method for constructing a heterogeneous graph representation learning model based on hybrid experts according to claim 1, characterized in that, The global Transformer layer is replaced with a linear complexity Transformer or a graph-specific Transformer with a spatial shortest path bias.

7. The method for constructing a heterogeneous graph representation learning model based on hybrid experts according to claim 1, characterized in that, The neighborhood relation context vector is replaced by any one of the following: the absolute frequency vector of each type of node in the first-order neighborhood, the distribution vector of each type of node in the second-order neighborhood, or the distribution vector of each type of edge in the neighborhood.

8. A heterogeneous graph representation learning system based on hybrid experts, characterized in that, The system is implemented using the method described in any one of claims 1-7, and the system comprises: The multidimensional prior and feature construction module is used to extract the node semantic matrix and the edge adjacency matrix from the original heterogeneous graph, and to calculate the node position encoding and the neighborhood relationship context features that represent the micro-connectivity frequency. The microscopic local heterogeneous feature extraction module includes a ternary fusion gate and a hybrid expert pool composed of k parallel graph neural networks with non-shared parameters. The ternary fusion gate takes the concatenated vector of node features, node type embedding, and neighborhood relation context vector as input to calculate the expert assignment probability weights. The hybrid expert pool dynamically routes nodes to the corresponding GNN experts according to the assignment probability weights and performs topological feature message passing within the graph neighborhood. The long-range macro-attention modeling module encapsulates a multi-head self-attention computation network, which receives refined node features from local modules, fuses structural position encoding, and then executes global long-range attention interactions. The model alternation coordination and gradient training module is used to manage the depth alternation stacking order from local network layers to global network layers, embed layer normalization and jump residual paths in the network, combine the target loss calculated by the task prediction layer, and control the backpropagation gradient update of the entire neural network architecture.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the heterogeneous graph representation learning model construction method based on any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the heterogeneous graph representation learning model construction method based on hybrid experts as described in any one of claims 1-7.

Images