Text attribute graph node classification method based on hybrid variational expectation maximization

Abstract

The invention provides a text attribute graph node classification method based on hybrid variational expectation maximization, and belongs to the technical field of data mining. The method comprises the following steps: constructing a basic dual-module framework based on a pre-training language model and a graph neural network and initializing the basic dual-module framework; executing M-step forward propagation once to generate a structure pseudo tag; performing E-step training on supervision updating of the pre-training language model in the graph neural network; performing M-step training on the graph neural network in supervised updating of the pre-training language model; the E-step training and the M-step training are alternately iterated to obtain a trained language model and a graph neural network; and the test nodes perform prediction by using the trained language model and the graph neural network to obtain a classification result. According to the method, through joint optimization of embedding and labels, the problem that embedding alignment and label supervision are mutually separated is effectively solved, learned node representation has structure-semantic consistency and task discrimination ability at the same time, and the method is remarkably superior to an existing method on multiple text attribute graph reference data sets.

Description

Technical Field

[0001] This invention belongs to the field of data mining technology, and in particular relates to a text attribute graph node classification method based on hybrid variational expectation maximization. Background Technology

[0002] Text attribute graphs (TAPs) are a multi-dimensional representation method that integrates linguistic features and topological relationships. Their core objective is to reveal the implicit semantic connections and attribute distributions within text data through structured modeling. In TAPs, nodes are represented by discrete text units (such as words, sentences, and paragraphs) from traditional natural language processing, while edges are constructed based on syntactic dependencies, co-occurrence frequencies, or semantic similarities between these text units. TAPs are widely used in many real-world scenarios, such as citation networks and social networks. In these scenarios, it is often necessary to predict the labels of unlabeled nodes based on the labels of labeled nodes, i.e., a supervised node classification problem.

[0003] Achieving accurate label prediction on text attribute graphs relies on constructing embeddings that possess both structural-semantic consistency and task-specific discriminative power. Conversely, effective embedding alignment depends on task-specific signals to emphasize relevant features. Methods focused on embedding alignment often neglect downstream task objectives, resulting in embeddings lacking the discriminative power required for accurate label prediction. In contrast, task-driven methods that focus solely on label supervision ignore the importance of integrating structural and semantic information, leading to a high susceptibility to overfitting. This separation ultimately hinders the robustness and uniformity of node representations, limiting their performance on downstream tasks.

[0004] A natural solution to the circular dependency between embedding alignment and label supervision is to optimize these objectives simultaneously. However, joint optimization complicates the loss landscape, making stable and efficient convergence more difficult. Specifically, embedding alignment aims to maximize the mutual information between textual and structural embeddings, ensuring robustness to changes in node features and graph topology. In contrast, label supervision focuses on minimizing prediction errors in downstream tasks, requiring embeddings to be highly separable along task-specific decision boundaries. While these objectives are not inherently contradictory, their joint optimization typically leads to higher variance and slower convergence. Summary of the Invention

[0005] The purpose of this invention is to provide a text attribute graph node classification method based on hybrid variational expectation maximization, so as to solve the technical problem of embedding-label circular dependency in the prior art and improve the node classification ability of the model.

[0006] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0007] A text attribute graph node classification method based on mixture variational expectation maximization, the method comprising the following steps:

[0008] Step S1: Construct and initialize a basic dual-module framework based on a pre-trained language model and a graph neural network;

[0009] Step S2: Perform an M-step forward propagation on the adjacency matrix between nodes and the initial semantic embedding to generate structural pseudo-labels;

[0010] Step S3: The pre-trained language model performs E-step training under the supervision of the graph neural network;

[0011] Step S4: The graph neural network performs M-step training under the supervised update of the pre-trained language model;

[0012] Step S5: E-step training and M-step training alternate and iterate. The pre-trained language model and graph neural network are continuously optimized under mutual supervision, and finally the trained language model and graph neural network are obtained.

[0013] Step S6: The test node uses the trained language model and graph neural network to make predictions and obtain classification results.

[0014] Further, step S1 includes the following steps:

[0015] Step S11: Select a pre-trained language model and a graph neural network as the basic dual-module framework, and initialize the pre-trained language model and the graph neural network;

[0016] Step S12: Obtain input data from the dataset and set control parameters;

[0017] Step S13: Generate the initial semantic embedding.

[0018] Further, step S2 includes the following steps:

[0019] Step S21: Structural Embedding Extraction: The graph neural network takes the adjacency matrix between nodes and the initial semantic embedding as input, and aggregates the neighborhood structural information through graph convolution to obtain the structural embedding of the nodes;

[0020] Step S22: Distribution parameter mapping: The structural embedding of each node is mapped to the mean and standard deviation of the structural embedding through the second MLP linear layer, so that the structural embedding follows a Gaussian distribution;

[0021] Step S23: Gaussian sampling: Use the reparameterization technique to sample the Gaussian distribution that the structure embedding follows, and obtain an M-step sampling distribution instance;

[0022] Step S24: Structural pseudo-label generation: Input all sampled distribution instances into another graph neural network layer, and generate M-step structural pseudo-labels through graph convolution and classification mapping.

[0023] Further, step S3 includes the following steps:

[0024] Step S31: Input the node text sequence into the pre-trained language model to obtain the node text semantic embedding;

[0025] Step S32: Map the semantic embedding of each node text to the mean and standard deviation of the semantic embedding through the first MLP linear layer, so that the semantic embedding follows a Gaussian distribution;

[0026] Step S33: Construct the MSE loss for the E-step and backpropagate to update the pre-trained language model;

[0027] Step S34: Based on the Gaussian distribution described by the semantic embedding mean and standard deviation, obtain the E-step sampling distribution instance through Gaussian sampling;

[0028] Step S35: All sampled distribution instances generate semantic pseudo-labels for step E through the second linear layer;

[0029] Step S36: Construct the cross-entropy loss function of step E to update the parameters of the second linear layer for the objective;

[0030] Step S37: The semantic embedding mean, standard deviation, and semantic pseudo-labels generated in step E serve as supervision in the next step M of training.

[0031] Further, step S4 includes the following steps:

[0032] Step S41: Structure Embedding Extraction: The graph neural network takes the adjacency matrix between nodes and the semantic embedding of nodes as input, and aggregates the neighborhood structure information through graph convolution to obtain the structure embedding of nodes;

[0033] Step S42: Distribution parameter mapping: The structural embedding of each node is mapped to the mean and standard deviation of the structural embedding through the second MLP linear layer, so that the structural embedding follows a Gaussian distribution;

[0034] Step S43: Construct the M-step MSE loss update;

[0035] Step S44: Gaussian sampling: Using the reparameterization technique, sample the Gaussian distribution that the above-mentioned structure embedding follows to obtain an M-step sampling distribution instance;

[0036] Step S45: Structural pseudo-label generation: Input all sampled distribution instances into another graph neural network layer, and generate M-step structural pseudo-labels through graph convolution and classification mapping;

[0037] Step S46: Construct the M-step cross-entropy loss to update the parameters of the graph neural network for the target;

[0038] Step S47: Mean of structural embedding generated in step M Standard deviation and structural pseudo-tags It will play a supervisory role in the next E-step training.

[0039] Furthermore, the MSE loss for the E-step is expressed as follows:

[0040]

[0041] in, This represents the MSE loss at step E; Indicates the embedding dimension; Indicates the number of nodes; and They represent the first Mean and standard deviation of semantic embeddings of each node; , They represent the first The structural embedding mean and standard deviation of each node.

[0042] Furthermore, the MSE loss for the M steps is expressed as follows:

[0043]

[0044] in, This represents the MSE loss over M steps; Indicates the embedding dimension; Indicates the number of nodes; and They represent the first Mean and standard deviation of semantic embeddings of each node; , They represent the first The structural embedding mean and standard deviation of each node.

[0045] Furthermore, the cross-entropy loss function for the E-step is expressed as follows:

[0046]

[0047] in, This represents the cross-entropy loss at the E-step. Represents a set of nodes with unknown labels; Represents a set of known labeled nodes; This represents the distribution of structural pseudo-labels output during the M-step initialization or iteration process, which remains unchanged during the E-step optimization process, thus playing a supervisory role. express An example of the E-step sampling distribution of a node; Indicates the first Category labels for each node; Represents a known set of labels. , The set of indices of the labeled nodes. For a set of nodes; This represents the set of labels for unlabeled nodes. This is the set of indices for unlabeled nodes; Indicates except the first The set of labels for all unlabeled nodes other than the first node; Represents a text attribute graph; This represents the distribution of semantic labels learned by the second linear layer in the E-step; This represents the weighting coefficient.

[0048] Furthermore, the M-step cross-entropy loss is expressed as follows:

[0049]

[0050] in, This represents the cross-entropy loss over M steps; Indicates the weighting coefficient; Indicates the first An example of an M-step sampling distribution for a node; This indicates the distribution of semantic pseudo-labels output in the E-step, which remains unchanged during the M-step optimization process and serves as a supervisory function. This represents the structural labels learned by the M-step graphical neural network; Indicates the first Category labels for each node; Represents a known set of labels. , The set of indices of the labeled nodes. For a set of nodes; This represents the set of labels for unlabeled nodes. This is the set of indices for unlabeled nodes; Indicates except the first The set of labels for all unlabeled nodes other than the first node; Represents a text attribute graph.

[0051] Compared with the prior art, the present invention has the following beneficial technical effects:

[0052] 1) Hybrid latent variable joint modeling: The node structure embedding (continuous latent variable) and unobserved node labels (discrete latent variable) are modeled in a unified manner. The joint optimization of embedding alignment and label supervision is achieved through the EM framework, which overcomes the performance bottleneck caused by the separate optimization of the two.

[0053] 2) E-step-M-step dual-branch alternating optimization: In the E-step, the language model updates the semantic embedding and generates semantic pseudo-labels under the supervision of the structural module; in the M-step, the graph neural network recalibrates the structural embedding and generates structural pseudo-labels under the supervision of the semantic module; the two steps alternate and iterate to achieve mutual reinforcement of structural and semantic information.

[0054] 3) Joint training of MSE loss and cross-entropy loss: Simultaneously optimize embedding distribution alignment (MSE loss) and label classification (cross-entropy loss) to ensure that the learned representation has both structural-semantic consistency and discriminative ability for downstream tasks.

[0055] 4) This invention effectively solves the problem of the separation between embedding alignment and label supervision by jointly optimizing embedding and label, so that the learned node representation has both structural-semantic consistency and task discrimination ability, and significantly outperforms existing methods on multiple text attribute graph benchmark datasets. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of the text attribute graph node classification method based on hybrid variational expectation maximization of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] This invention proposes a text attribute graph node classification method based on hybrid variational expectation maximization (EM), aiming to improve classification performance through joint optimization of embeddings and labels. The method treats node embeddings as continuous latent variables and unobserved labels as discrete latent variables, constructing a joint optimization objective based on this hybrid latent variable. Based on the expectation maximization (EM) algorithm, this objective is decomposed into alternating E-steps and M-steps, with mutual refinement of embeddings and labels achieved through the integration of dedicated modules: in the E-step, the language module (based on LM) refines semantic embeddings and generates semantic pseudo-labels under the supervision of the structure module; in the M-step, the structure module (based on GNN) recalibrates structural embeddings and optimizes structural pseudo-labels under the supervision of the semantic module. Through iterative optimization, better node classification results are ultimately achieved.

[0060] This invention proposes a text attribute graph node classification method based on hybrid variational expectation maximization, such as... Figure 1 As shown, the method includes the following steps:

[0061] Step S1: Construct and initialize a basic dual-module framework based on a pre-trained language model and a graph neural network.

[0062] Step S11: Select a pre-trained language model (LM) and a graph neural network (GNN) as the basic dual-module framework, and initialize the pre-trained language model and the graph neural network.

[0063] Specifically, pre-trained language models can be BERT, RoBERTa, DeBERTa, etc.; graph neural networks can be GCN, GAT, SAGE, etc.

[0064] Step S12: Obtain input data from the dataset and set control parameters.

[0065] Text attribute graph It can be defined as a triple. In the form of, Represents a set of nodes. Indicates the first 1 node Indicates the number of nodes; Represents the adjacency matrix between nodes; This represents a sequence of text elements for each node, corresponding to the text attributes associated with each node. The node classification task aims to learn a mapping function. Given the labels of a subset of observable nodes Predicting the label of unobserved nodes in the following case ,in It is the set of indices of the labeled nodes. It is the set of indices of unlabeled nodes. Indicates the first The category label of each node, This represents the learnable parameters of the model.

[0066] Specifically, input data is obtained from the dataset, which includes: (1) node text sequences. (2) Adjacency matrix between nodes .

[0067] The control parameters include: (1) the total number of iterations I; and (2) the balance coefficient between distribution alignment and classification supervision in the loss function. (3) Other parameters required for model training, such as the rate of decline and batch size; (4) Known label set ,in The set of indices of the labeled nodes. This is the set of indices for unlabeled nodes; Indicates the first Category labels for each node; This represents the set of labels for unlabeled nodes. , ; Indicates except the first The set of labels for all unlabeled nodes other than the first node; Represents a text attribute graph.

[0068] Step S13: Initial semantic embedding generation.

[0069] Node text sequence Input the pre-trained language model, perform one forward inference (without updating parameters), and obtain the initial semantic embeddings of the nodes. This serves as the initial input to the graph neural network. Indicates the first The initial semantic embedding of each node; This represents the total number of nodes.

[0070] Step S2: Perform an M-step forward propagation on the adjacency matrix between nodes and the initial semantic embedding to generate structural pseudo-labels.

[0071] Before formally entering the alternating E-step and M-step iterations, a complete M-step forward propagation is first performed using the initialized graph neural network to generate structural pseudo-distribution embeddings and structural pseudo-labels, which serve as supervision signals for the first round of E-step optimization. The specific process is as follows:

[0072] Step S21: Structure Embedding Extraction: Graph Neural Network uses the adjacency matrix between nodes Taking the initial semantic embedding as input, the neighborhood structure information is aggregated through graph convolution to obtain the node structure embedding, as shown below:

[0073]

[0074] in, Indicates the first The structural embedding of each node; This represents a graph neural network.

[0075] Step S22: Distribution parameter mapping: Structural embedding of each node The second MLP linear layer maps the structural embeddings to their mean and standard deviation, ensuring that the structural embeddings follow a Gaussian distribution, as shown below:

[0076]

[0077] in, , They represent the first The structural embedding mean and standard deviation of each node; Indicates the first The structure of each node is embedded.

[0078] Step S23: Gaussian Sampling: Using the reparameterization trick, sample the Gaussian distribution followed by the structural embedding to obtain an M-step sampling distribution instance. , means as follows:

[0079]

[0080] in, Indicates the distribution from the standard normal distribution Random noise in the mid-sample; Indicates the first An example of an M-step sampling distribution for a given number of nodes.

[0081] Step S24: Structural pseudo-label generation: Input all sampled distribution instances into another graph neural network layer, and generate M-step structural pseudo-labels through graph convolution and classification mapping. .

[0082] Furthermore, the structural embedding mean of the nodes generated in the M-step process. and standard deviation and structural pseudo-labels It will be used as a fixed monitoring signal in subsequent E-steps.

[0083] Step S3: The pre-trained language model performs E-step training under the supervision of the graph neural network.

[0084] Step S31: Embedding Distribution Alignment Optimization

[0085] Step S311: Input the node text sequence into the pre-trained language model to obtain the node text semantic embedding, as shown below:

[0086]

[0087] in, Indicates the first Individual node text semantic embedding , Indicates the embedding dimension; This represents a pre-trained language model; Indicates the first Each node text.

[0088] Step S312: Map the semantic embedding of each node text to the mean and standard deviation of the semantic embedding through the first MLP linear layer, so that the semantic embedding follows a Gaussian distribution.

[0089] Specifically, the text semantic embedding of each node The semantic embedding mean is mapped through the first linear layer of the MLP. and standard deviation To explicitly introduce uncertainty, the semantic embedding of each node follows a Gaussian distribution:

[0090]

[0091] in, and They represent the first Mean and standard deviation of semantic embeddings of each node; Indicates the first Semantic embedding of each node; This represents the set of all nodes whose semantic embeddings follow a Gaussian distribution.

[0092] Step S313: Construct the E-step MSE loss, and backpropagate to update the pre-trained language model. The E-step MSE loss is represented as follows:

[0093]

[0094] in, This represents the MSE loss at step E; Indicates the embedding dimension.

[0095] Structural embedding mean of nodes generated in M ​​steps and standard deviation It remains unchanged during the E-step optimization process, serving a supervisory role.

[0096] Step S32: Tag Optimization

[0097] Step S321: Based on semantic embedding mean and standard deviation The Gaussian distribution described is obtained by Gaussian sampling to obtain an E-step sampling distribution instance.

[0098] Specifically, the reparameterization technique is used to refine the mean. and standard deviation Sampling is performed on the described Gaussian distribution to obtain an E-step sampling distribution instance. , means as follows:

[0099]

[0100] in, Indicates the distribution from the standard normal distribution Random noise in the sampled data.

[0101] Step S322: Generate E-step semantic pseudo-labels for all sampled distribution instances through the second linear layer. .

[0102] Step S323: Construct the E-step cross-entropy loss function to update the parameters of the second linear layer. The E-step cross-entropy loss function is expressed as follows:

[0103]

[0104] in, This represents the cross-entropy loss at the E-step. Represents a set of nodes with unknown labels; Represents a set of known labeled nodes; The distribution of the structural pseudo-labels, which is initialized in the M-step (or output during the iteration), remains unchanged during the E-step optimization and serves as a supervisory function. express An example of the E-step sampling distribution of a node; Indicates the first Category labels for each node; Represents a known set of labels. , The set of indices of the labeled nodes. For a set of nodes; This represents the set of labels for unlabeled nodes. This is the set of indices for unlabeled nodes; Indicates except the first The set of labels for all unlabeled nodes other than the first node; Represents a text attribute graph; This represents the distribution of semantic labels learned by the second linear layer in the E-step; This represents the weighting coefficient.

[0105] Part One Loss Encourage E-step label predictions to approximate the structural pseudo-label distribution of the M-step, the second part of the loss. The aim is to use known, real labels for supervision.

[0106] Step S33: Pseudo-tag transmission

[0107] E-step generated semantic embedding mean Standard deviation and semantic pseudo-tags It plays a supervisory role in the next M-step training.

[0108] Step S4: The graph neural network performs M-step training under the supervision of the pre-trained language model.

[0109] The M-step keeps the parameters of the first and second linear layers of the MLP unchanged, and the semantic embedding mean of the nodes generated in the E-step is... Standard deviation and semantic pseudo-tags It remains unchanged during the M-step optimization process, serving as a fixed supervisory signal to guide the parameter updates of the graph neural network.

[0110] Step S41: Embedded distribution alignment optimization.

[0111] Step S411: Structure Embedding Extraction: Graph Neural Network uses the adjacency matrix between nodes Taking node semantic embeddings as input, we aggregate neighborhood structure information through graph convolution to obtain the node structure embeddings, as shown below:

[0112]

[0113] in, Indicates the first The structural embedding of each node; This represents a graph neural network.

[0114] Step S412: Distribution parameter mapping: Structural embedding of each node The second MLP linear layer maps the structural embeddings to their mean and standard deviation, ensuring that the structural embeddings follow a Gaussian distribution, as shown below:

[0115]

[0116] in, , They represent the first The structural embedding mean and standard deviation of each node; Indicates the first The structure of each node is embedded.

[0117] Step S413: Construct the M-step MSE loss update, where the M-step MSE loss is represented as follows:

[0118]

[0119] in, This represents the MSE loss over M steps.

[0120] Step S42: Tag Optimization

[0121] Step S421: Gaussian Sampling: Using the reparameterization trick, sample the Gaussian distribution followed by the structural embedding to obtain an M-step sampling distribution instance. , means as follows:

[0122]

[0123] in, Indicates the distribution from the standard normal distribution Random noise in the mid-sample; Indicates the first An example of an M-step sampling distribution for a given number of nodes.

[0124] Step S422: Structural pseudo-label generation: Input all sampled distribution instances into another graph neural network layer, and generate M-step structural pseudo-labels through graph convolution and classification mapping. .

[0125] Step S423: Construct the M-step cross-entropy loss to update the parameters of the graph neural network for the target. The M-step cross-entropy loss is represented as follows:

[0126]

[0127] in, This represents the cross-entropy loss over M steps; Indicates the weighting coefficient; Indicates the first An example of an M-step sampling distribution for a node; This indicates the distribution of semantic pseudo-labels output in the E-step, which remains unchanged during the M-step optimization process and serves as a supervisory function. This represents the structural labels learned by the M-step graph neural network.

[0128] Step S5: E-step training and M-step training alternate and iterate. The pre-trained language model and graph neural network are continuously optimized under mutual supervision, and finally the trained language model and graph neural network are obtained.

[0129] Specifically, the E-step training in step S3 and the M-step training in step S4 are repeated, and the pre-trained language model and graph neural network are continuously optimized under mutual supervision until the number of iterations reaches the preset value I. After the iterations are completed, a fully trained language model and graph neural network are obtained.

[0130] Step S6: The test node uses the trained language model and graph neural network to make predictions and obtain classification results.

[0131] Specifically, for each test node, a pre-trained language model and a graph neural network are used for prediction, resulting in two independent classification results. The final node classification result can be selected from the two classification results of the language model and the graph neural network, whichever has higher confidence or better validation performance, or an ensemble strategy can be used to combine the two classification results for output.

[0132] This invention proposes a Hybrid Variational Expectation-Maximization (HVEM) method for text attribute graph node classification. It introduces a novel approach to jointly optimize labels and embeddings by modeling embeddings and labels as hybrid latent variables and using a variational inference-based approach to alternately perform embedding alignment and label supervision. Iterative E-steps and M-steps achieve dynamic alignment between embeddings and task-specific labels, ensuring that the learned representations maintain structural-semantic consistency and relevance to downstream tasks. This invention evaluates HVEM on seven TAG datasets, and experimental results demonstrate its competitive performance and scalability, achieving significant improvements over existing baselines.

[0133] Dataset:

[0134] The Cora dataset contains 2,708 scientific publications, categorized into seven groups (such as neural networks, genetic algorithms, and reinforcement learning). The citation network contains 5,429 edges, with each paper either citing at least one other paper or being cited by another paper.

[0135] The CiteSeer dataset contains 3,186 scientific papers divided into six domains (including machine learning, information retrieval, and artificial intelligence). The task is to classify papers based on their titles and abstracts.

[0136] The WikiCS dataset is built on Wikipedia and designed specifically for graph neural network benchmarking. It contains article nodes from 10 computer science fields, with features derived from the article text and exhibiting high connectivity.

[0137] The ArXiv-2023 dataset represents a citation network of computer science papers published on arXiv in 2023 and later. Nodes correspond to papers, and directed edges represent citation relationships. The goal is to categorize papers into 40 subject areas.

[0138] The Ele-Photo dataset, derived from the Amazon Electronics dataset, uses nodes to represent electronic products and edges to indicate frequent shared purchases or browsing relationships. Each node is labeled according to a three-level product category, and text attributes are based on user reviews. The task is to classify products into 12 categories.

[0139] OGBN-Products Dataset: A scaled-down version of the original OGBN-Products dataset, created using a node sampling strategy. It contains 54,025 nodes and 74,420 edges, where nodes represent Amazon products and edges represent co-purchase relationships. The classification task involves assigning products to one of 47 top-level categories. We selected its subgraph for evaluation.

[0140] The OGBN-ArXiv dataset models the citation network of computer science papers on arXiv, derived from the MAG database. In this directed graph, nodes represent individual papers, and edges represent citation links between papers. The goal is to classify each paper into one of 40 predefined categories.

[0141] Evaluation indicators:

[0142] Accuracy and Standard Deviation: This invention uses node classification accuracy as an evaluation metric, calculates the percentage of correctly predicted node categories in the test dataset out of the total number of test nodes, and provides the average accuracy and standard deviation of five test results.

[0143] Compare to baseline:

[0144] GNN-based methods and LM-based methods: GNN-based methods include GCN, SAGE, and GAT combined with shallow node embeddings, while LM-based methods include fully fine-tuned RoBERTa-large, BERT-large, and DeBERTa-base.

[0145] Methods that integrate LM-GNN: CoGSL and Nodeformer represent models that use only static bag-of-words embeddings as semantic representations and combine them with graph neural networks for node classification. GIANT, GraphFormers, SimTeG, TAPE, GLEM, ENGINE, and ConGraT represent hybrid LM-GNN paradigms that use LM to enhance semantic information.

[0146] Table 1. HVEM node classification results on different datasets and baselines

[0147]

[0148] Note: The values ​​in the table represent node classification accuracy (%), in the format "mean ± standard deviation", and are the statistical results of 5 runs with different random seeds; "---" indicates that there is no corresponding result on this dataset; bold indicates the best result in this column, underline indicates the second best result, and the row containing HVEM (this invention) is bolded to distinguish it; "†" indicates that the result is directly cited from the comparison literature ENGINE, and "★" indicates the result reproduced on the task of this invention using its official code. Abbreviations: MLP – Multi-Layer Perceptron; GCN – Graph Convolutional Network; GAT – Graph Attention Network; SAGE – Graph SAGE (Graph Sample and Aggregate); BERT – Bidirectional Encoder Representations from Transformers; RoBERTa – Robustly Optimized BERT Pretraining Approach; DeBERTa – Decoding-enhanced BERT with Disentangled Attention; GraphFormers, Nodeformer, CoGSL, GIANT, GLEM, ENGINE, TAPE, ConGraT, and SimTeG are existing baseline methods in this field; HVEM – Hybrid Variational Expectation-Maximization proposed in this invention.

[0149] As shown in Table 1, HVEM demonstrates significant performance improvements over traditional GNN-based and LM-based methods. Specifically, HVEM achieves a 7.20% improvement in average accuracy compared to traditional GNN methods and a 6.19% improvement compared to LM methods, highlighting its ability to effectively coordinate structural and semantic information. Compared to LM-GNN hybrid methods, HVEM consistently achieves best or near-best performance on most datasets, establishing itself as a robust and generalizable method. HVEM ranks first on five out of seven datasets, maintaining competitiveness even on datasets where other methods perform well. This consistency underscores its ability to generate more cohesive embedding spaces, improving task-level discriminative power and making it well-suited for graph-based classification tasks.

[0150] While HVEM performs poorly on the WikiCS and ArXiv-2023 datasets, TAPE's strength lies in its unique use of GPT-based text interpretation, which significantly enhances the context of text-based node attributes. This advantage is particularly evident on datasets where text attributes are dominant and exhibit rich contextual information. In contrast, while HVEM excels in structure-semantic alignment and label refinement, its current lack of advanced text-specific data augmentation capabilities may limit its effectiveness on these datasets.

[0151] Furthermore, HVEM demonstrates exceptional scalability, performing well across diverse data sizes. HVEM achieves optimal results on small datasets by capturing fine-grained patterns with high precision, while simultaneously delivering near-or better performance on large datasets. This consistency across datasets of varying sizes underscores HVEM's scalability and robustness, making it suitable for resource-constrained environments and large-scale real-world applications.

[0152] Table 2. Comparison of HVEM node classification results on different backbone networks

[0153]

[0154] Note: The values ​​in the table represent node classification accuracy (%), formatted as "mean ± standard deviation", and are statistical results from 5 runs with different random seeds. Rows in the table represent different Graph Neural Networks (GNNs) used as the M-step backbone, and columns represent different pre-trained Language Models (LMs) used as the E-step backbone. Abbreviations: GNN – Graph Neural Network; LM – Pre-trained Language Model; GCN – Graph Convolutional Network; GAT – Graph Attention Network; SAGE – Graph SAGE (Graph Sample and Aggregate); BERT – Bidirectional Encoder Representations from Transformers; RoBERTa – Robustly Optimized BERT Pretraining Approach; DeBERTa – Decoding-enhanced BERT with Disentangled Attention.

[0155] This invention also uses ablation experiments to evaluate the impact of different backbone networks on HVEM performance. As shown in Table 2, HVEM exhibits strong generalization ability across various LM and GNN networks. In particular, simpler backbones sometimes outperform more complex ones. For example, GCN performs best on both datasets, likely due to its shallow architecture, which effectively captures graph structure while mitigating overfitting. On the other hand, DeBERTa shows poor performance, possibly due to its limited ability to capture sufficient semantic information when dealing with low-dimensional features. In contrast, BERT's superior performance highlights the impact of LM bias on the dataset and underscores the importance of tailoring backbone selection to the specific features of each dataset.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text attribute graph node classification method based on hybrid variational expectation maximization, characterized in that, The method includes the following steps: Step S1: Construct and initialize a basic dual-module framework based on a pre-trained language model and a graph neural network; Step S2: Perform an M-step forward propagation on the adjacency matrix between nodes and the initial semantic embedding to generate structural pseudo-labels; Step S3: The pre-trained language model performs E-step training under the supervision of the graph neural network; Step S4: The graph neural network performs M-step training under the supervised update of the pre-trained language model; Step S5: E-step training and M-step training alternate and iterate. The pre-trained language model and graph neural network are continuously optimized under mutual supervision, and finally the trained language model and graph neural network are obtained. Step S6: The test node uses the trained language model and graph neural network to make predictions and obtain classification results.

2. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Select a pre-trained language model and a graph neural network as the basic dual-module framework, and initialize the pre-trained language model and the graph neural network; Step S12: Obtain input data from the dataset and set control parameters; Step S13: Generate the initial semantic embedding.

3. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 2, characterized in that, Step S2 includes the following steps: Step S21: Structural Embedding Extraction: The graph neural network takes the adjacency matrix between nodes and the initial semantic embedding as input, and aggregates the neighborhood structural information through graph convolution to obtain the structural embedding of the nodes; Step S22: Distribution parameter mapping: The structural embedding of each node is mapped to the mean and standard deviation of the structural embedding through the second MLP linear layer, so that the structural embedding follows a Gaussian distribution; Step S23: Gaussian sampling: Use the reparameterization technique to sample the Gaussian distribution that the structure embedding follows, and obtain an M-step sampling distribution instance; Step S24: Structural pseudo-label generation: Input all sampled distribution instances into another graph neural network layer, and generate M-step structural pseudo-labels through graph convolution and classification mapping.

4. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 3, characterized in that, Step S3 includes the following steps: Step S31: Input the node text sequence into the pre-trained language model to obtain the node text semantic embedding; Step S32: Map the semantic embedding of each node text to the mean and standard deviation of the semantic embedding through the first MLP linear layer, so that the semantic embedding follows a Gaussian distribution; Step S33: Construct the MSE loss for the E-step and backpropagate to update the pre-trained language model; Step S34: Based on the Gaussian distribution described by the semantic embedding mean and standard deviation, obtain the E-step sampling distribution instance through Gaussian sampling; Step S35: All sampled distribution instances generate E-step semantic pseudo-labels through the second linear layer; Step S36: Construct the cross-entropy loss function of step E to update the parameters of the second linear layer for the objective; Step S37: The semantic embedding mean, standard deviation, and semantic pseudo-labels generated in step E serve as supervision in the next step M of training.

5. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 4, characterized in that, Step S4 includes the following steps: Step S41: Structure Embedding Extraction: The graph neural network takes the adjacency matrix between nodes and the semantic embedding of nodes as input, and aggregates the neighborhood structure information through graph convolution to obtain the structure embedding of nodes; Step S42: Distribution parameter mapping: The structural embedding of each node is mapped to the mean and standard deviation of the structural embedding through the second MLP linear layer, so that the structural embedding follows a Gaussian distribution; Step S43: Construct the M-step MSE loss update; Step S44: Gaussian sampling: Using the reparameterization technique, sample the Gaussian distribution that the above-mentioned structure embedding follows to obtain an M-step sampling distribution instance; Step S45: Structural pseudo-label generation: Input all sampled distribution instances into another graph neural network layer, and generate M-step structural pseudo-labels through graph convolution and classification mapping; Step S46: Construct the M-step cross-entropy loss to update the parameters of the graph neural network for the target; Step S47: Mean of structural embedding generated in step M Standard deviation and structural pseudo-tags It will play a supervisory role in the next E-step training.

6. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 4, characterized in that, The MSE loss for the E-step is expressed as follows: in, This represents the MSE loss at step E; Indicates the embedding dimension; Indicates the number of nodes; and They represent the first Mean and standard deviation of semantic embeddings of each node; , They represent the first The structural embedding mean and standard deviation of each node.

7. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 5, characterized in that, The MSE loss for M steps is expressed as follows: in, This represents the MSE loss over M steps; Indicates the embedding dimension; Indicates the number of nodes; and They represent the first Mean and standard deviation of semantic embeddings of each node; , They represent the first The structural embedding mean and standard deviation of each node.

8. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 4, characterized in that, The cross-entropy loss function for the E-step is expressed as follows: in, This represents the cross-entropy loss at the E-step. Represents a set of nodes with unknown labels; Represents a set of known labeled nodes; This represents the distribution of structural pseudo-labels output during the M-step initialization or iteration process, which remains unchanged during the E-step optimization process, thus playing a supervisory role. express An example of the E-step sampling distribution of a node; Indicates the first Category labels for each node; Represents a known set of labels. , The set of indices of the labeled nodes. For a set of nodes; This represents the set of labels for unlabeled nodes. This is the set of indices for unlabeled nodes; Indicates except the first The set of labels for all unlabeled nodes other than the first node; Represents a text attribute graph; This represents the distribution of semantic labels learned by the second linear layer in the E-step; This represents the weighting coefficient.

9. The text attribute graph node classification method based on hybrid variational expectation maximization according to claim 5, characterized in that, The M-step cross-entropy loss is expressed as follows: in, This represents the cross-entropy loss over M steps; Indicates the weighting coefficient; Indicates the first An example of an M-step sampling distribution for a node; This indicates the distribution of semantic pseudo-labels output in the E-step, which remains unchanged during the M-step optimization process and serves as a supervisory function. This represents the structural labels learned by the M-step graphical neural network; Indicates the first Category labels for each node; Represents a known set of labels. , The set of indices of the labeled nodes. For a set of nodes; This represents the set of labels for unlabeled nodes. This is the set of indices for unlabeled nodes; Indicates except the first The set of labels for all unlabeled nodes other than the first node; Represents a text attribute graph.

Images