Two-way GCN-BERT scientific data classification method based on rotation coding and dynamic gating

Abstract

The application discloses a bidirectional GCN-BERT scientific data classification method based on rotation coding and dynamic gating. Firstly, the original text data is preprocessed, and a classification mark containing global information is obtained by using a pre-training model and added to the beginning of the input text. Secondly, the classification mark is optimized in features by a CorNet neural network, and then position enhancement and global interaction are performed on the mark by a rotation position enhanced multi-layer feature interaction network RP-MLFIN. Then, the mark is transmitted to a bidirectional graph convolutional neural network for feature interaction and graph convolution processing to generate a graph-level representation. Finally, the classification mark after feature optimization and the graph-level representation are transmitted to a classifier respectively to generate BERT prediction and GCN prediction, and the prediction results are fused by an attention mechanism to generate a final classification decision. The application improves the processing capacity of text data with strong context dependence and the stability and robustness of the classification result.

Description

Technical Field

[0001] This patent relates to the field of text classification technology, and in particular to a bidirectional GCN-BERT scientific data text classification method based on rotational encoding and dynamic gating. Background Technology

[0002] In the field of text classification technology, classifying scientific data has always been a challenging task. Traditional text classification methods, such as rule-based methods and shallow learning models, while achieving some success in certain situations, often fall short when dealing with large-scale, complex scientific data rich in semantic information. In recent years, with the rapid development of deep learning technology, text classification methods based on deep neural networks have gradually emerged, providing new solutions to the problem of scientific data classification.

[0003] Graph Convolutional Networks (GCNs), as powerful tools for processing graph-structured data, can effectively capture the topological relationships between nodes, thereby learning node representations with rich structural information. GCNs generate graph-level representations by aggregating the features of neighboring nodes and performing linear transformations on each node. However, when processing text data, GCNs often ignore word order and contextual semantic information, which limits their performance in text classification tasks to some extent.

[0004] The emergence of the BERT pre-trained model has revolutionized the field of natural language processing. By pre-training on large-scale text corpora, BERT learns rich semantic information and language patterns, enabling it to generate high-quality semantic embeddings for each word in the text. In text classification tasks, utilizing the semantic features extracted by BERT can significantly improve classification accuracy. The RoBERTa model is an optimized version of the BERT model, significantly improving its performance through a more refined pre-training strategy and a larger dataset. Its structure is similar to BERT, but improvements such as dynamic masking, larger training batches, and removal of the NSP task allow the model to learn richer semantic information. However, the RoBERTa model primarily focuses on the global semantic information of the text, with insufficient consideration for the local structure and contextual relationships of the text. Summary of the Invention

[0005] To overcome the limitations of existing methods, this invention proposes a bidirectional GCN-BERT scientific data classification method based on rotational encoding and dynamic gating. This invention fully combines the high-quality semantic embedding of BERT with the strong ability of GCN to capture topological information, achieving a complementary advantage. The method constructs a multi-stage feature enhancement pipeline to sequentially achieve deep semantic mining of text and topological position-aware modeling. A multi-layer feature interaction network based on rotational positional encoding aligns the global semantics of the text with the position of the graph structure, and bidirectional gated graph convolutional units construct a spatiotemporal joint feature propagation mechanism. Furthermore, the introduction of the CorNet network and BiGRU module further enhances the model's ability to capture and understand semantic and contextual information. Combined with a dynamic weight allocation strategy using an attention mechanism, this method effectively improves the accuracy, robustness, and generalization performance of scientific data classification while maintaining linear computational complexity. The method includes the following steps:

[0006] Step S1: Perform adaptive preprocessing on the original text data to obtain the input data for training.

[0007] Step S2: Use the RoBERTa pre-trained model to extract deep semantic features from the sample data in the input data, obtain a classification token (CLS token) containing global information, add it as a special symbol at the beginning of the input text, and its corresponding output vector represents the global semantic information of the entire text.

[0008] Step S3: The CLS token is optimized using a CorNet neural network to enhance its feature representation.

[0009] Step S4 further enhances the position and global interaction of the optimized CLS token through a Rotation-Position-Enhanced Multilayer Feature Interaction Network (RP-MLFIN). This network combines rotation-position encoding and linear attention, and enhances the local position information modeling capability through depthwise separable convolutions and convolutional position encoding. Its linear complexity and efficient feature interaction mechanism enable the CLS token to simultaneously contain textual semantics, topological position, and global dependencies.

[0010] Step S5: Embed a Bidirectional Gated Recurrent Unit (BiGRU) into the GCN to construct a BiGRU-based bidirectional graph convolutional neural network. After the output of each GCN layer and before the input of the GRU, a Dynamically Gated Feature Enhancement Network (DGFEN) is added. This network is a lightweight feature enhancer that combines gating mechanisms and additive attention.

[0011] Step S6: The RP-MLFIN-optimized CLS token is passed to a bidirectional graph convolutional neural network for feature interaction and graph convolution processing to generate a graph-level representation.

[0012] Step S7: The CLS token optimized by features and the graph-level representation processed by graph convolution are respectively passed to the classifier for classification prediction, generating BERT prediction and GCN prediction. The weights of the two are dynamically allocated through an attention mechanism, and the prediction results are fused to generate the final classification decision.

[0013] Preferably, the adaptive preprocessing operations for the sample data in step S1 include: data cleaning, removal of stop words from the sample data, generation of word and document embedding vectors, and calculation of the adjacency matrix. The normalized adjacency matrix is ​​then converted into a Deep Graph Library (DGL) graph object. This graph object is specifically designed for efficiently storing graph structure data containing nodes, edges, and their features, and for supporting complex graph operations, thus enabling efficient graph convolution operations. The normalization calculation formula for the adjacency matrix A with added self-loops is as follows:

[0014]

[0015] Where: I is the identity matrix, and D is the degree matrix after adding self-loops. It is the inverse square root of the degree matrix, that is:

[0016] D ii =∑ j A ij +1

[0017] Where: D ii Let A represent the degree of node i, i.e., the number of its neighboring nodes. ij This indicates whether there is an edge between nodes i and j; if so, it is 1, otherwise it is 0.

[0018] Preferably, the generation of word and document embedding vectors includes: loading pre-trained GloVe word vectors as word embedding vectors, and calculating the average word vector of each document as document embedding vectors.

[0019] Preferably, the construction of the DGL graph object includes: constructing a heterogeneous graph containing both word nodes and document nodes, and calculating pointwise mutual information (PMI) and term frequency-inverse document frequency (TF-IDF) respectively. Edges between words and between words and documents are defined based on PMI and TF-IDF respectively. The weight of the edge between two nodes i and i is defined as:

[0020]

[0021] Preferably, obtaining the CLS token containing global information in step S2 includes: using BERT to learn rich language structure and semantic information through pre-training; after encoding the input text, using a multi-layer bidirectional Transformer to capture contextual information; at the beginning of each input sequence, BERT inserts a CLS token, whose corresponding output vector integrates the semantic information of the entire sentence.

[0022] Preferably, the CorNet residual network described in step S3 enhances the expressive power of features through residual connections and nonlinear transformations, including: applying an activation function to the output distribution and introducing nonlinearity by applying an ELU activation function to the context vector. The transformed output distribution is then added to the original output distribution to form residual connections.

[0023] Preferably, the Rotation Position Enhanced Multilayer Feature Interaction Network (RP-MLFIN) described in step S4 is based on an improved Transformer architecture. Through a triple design of convolution enhancement, hierarchical feature modulation, and rotation position encoding, it achieves efficient long sequence modeling. By encoding absolute position information into the Q / K vector of attention computation through complex space rotation operations, it preserves relative positional relationships more naturally compared to traditional position encoding. Low-complexity attention is achieved using ELU-activated QK projection and rotation position encoding, while retaining the dual-residual structure of attention residuals and MLP residuals, thus enhancing gradient flow.

[0024] Step S4-1, Convolution Enhancement, captures local patterns such as phrases and local dependencies in the sequence by applying depthwise separable convolutions to the input features. Specifically, this includes: performing independent depthwise convolution operations on each channel to extract local features; using 1x1 pointwise convolutions to fuse channel information and enhance feature representation; and adding the convolution output to the original input to preserve the original feature information.

[0025] Step S4-2, hierarchical feature modulation, progressively optimizes feature representation through multi-layer feature interaction and dynamic gating mechanisms. Specifically, this includes: extracting higher-level features by stacking multiple Transformer layers; dynamically adjusting the contribution of each layer's features using gating weights to focus on key information; and fusing features from different levels through weighted summation or concatenation to generate the final representation.

[0026] Step S4-3, rotation position encoding, embeds position information into the feature vector through complex rotation operations. Specifically, this includes: treating the feature vector as a complex number and encoding position information through a certain rotation angle; dividing the feature dimension into multiple two-dimensional subspaces, rotating each subspace independently; applying rotation position encoding to the query Q and key K respectively, and calculating attention scores carrying relative position information.

[0027] Preferably, the Dynamic Gated Feature Enhancement Network (DGFEN) described in step S5 is a lightweight feature enhancer that integrates gating mechanisms and additive attention. It achieves efficient feature enhancement through dynamic feature selection and global-local feature interaction, enabling the GRU to receive more expressive inputs and effectively improving model performance without significantly increasing the model's computational burden.

[0028] The gating mechanism dynamically adjusts the importance of features through learnable weights. Specifically, it involves: generating gating weights by linearly transforming the input features; and dynamically weighting the input features using these gating weights. Additive attention models the global dependencies between features through nonlinear transformations. Specifically, it involves: projecting the input features into query Q and key K respectively; calculating attention scores using an additive function; and aggregating global information based on the attention weights. DGFEN combines the gating mechanism and additive attention in a cohesive design, forming a two-stage "selection-enhancement" process.

[0029] Preferably, the bidirectional graph convolutional neural network includes multiple graph convolutional layers and bidirectional gated recurrent units. Node features are extracted through the graph convolutional layers, and then BiGRU is used to capture the positive and negative contextual information of the features, ultimately generating a more comprehensive feature representation. The calculation formula for the graph convolutional operation of each layer is as follows:

[0030]

[0031] Wherein: H (l) W is the input feature matrix of the l-th layer. (l) is the weight matrix of layer l. (A+I) is the adjacency matrix after adding self-loops. D is the degree matrix after adding self-loops. σ is the activation function.

[0032] Bidirectional graph convolutional neural networks consist of multiple graph convolutional layers and bidirectional gated recurrent units. Node features are extracted through graph convolutional layers, enhanced by DGFEN, and then input into BiGRU to capture the positive and negative contextual information of the features, ultimately generating a more comprehensive feature representation.

[0033] Preferably, the attention mechanism weights described in step S7 are dynamically allocated using a fully connected layer and a Softmax function to calculate the attention weights, dynamically adjusting the contribution ratio of BERT and GCN, and fusing the output predictions of BERT and GCN through the attention weights to generate the final classification decision.

[0034] The substantial effects of this invention are as follows:

[0035] This invention optimizes the capture and representation of contextual information. It incorporates a bidirectional gated recurrent unit (BiGRU) into the traditional GCN, adding a dynamically gated feature enhancement network after the output of each GCN layer and before the input to the GRU. Through dynamic feature selection and global-local feature interaction, efficient feature enhancement is achieved, enabling the GRU to receive more expressive input and allowing the model to simultaneously capture both positive and negative contextual information, thus mitigating the feature loss problem that may occur with unidirectional propagation mechanisms. This bidirectional feature capture method enhances the comprehensiveness of feature representation, helping to improve the processing capability of context-dependent text data without significantly increasing the computational burden of the model.

[0036] An innovative design for feature enhancement and fusion is employed. A neural network (CorNet) is introduced to optimize the features of the CLS token, and further position enhancement and global interaction are achieved through a Rotation-Position Enhanced Multilayer Feature Interaction Network (RP-MLFIN). This network combines rotational position encoding and linear attention, and enhances the modeling ability of local positional information through depthwise separable convolution and convolutional position encoding. This allows the CLS token to simultaneously contain textual semantics, topological position, and global dependencies, strengthening the diversity and representational power of BERT output features. Simultaneously, an attention mechanism is used to weight and fuse the prediction results of BERT and GCN, achieving a dynamic balance of prediction information in different feature spaces and improving the stability and robustness of classification results. This feature optimization and fusion design demonstrates the technical advantages of this method in complex feature processing and information integration.

[0037] Robustness to scientific data classification tasks. Scientific data is often characterized by diversity and complexity. This invention utilizes Deep Graph Library (DGL) modeling to effectively characterize the adjacency relationships and feature distributions of scientific data. Simultaneously, it combines a bidirectional graph convolutional neural network to deeply mine the implicit topological structures and semantic relationships between data points. This design significantly enhances the model's adaptability to the characteristics of scientific data and improves its generalization performance and robustness to complex data.

[0038] In summary, this invention, through multi-network module collaborative design and innovative feature optimization strategies, fully realizes the efficient integration of textual semantic information and data structure information, and is an innovative technology with substantial improvement in the field of text classification. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0040] Figure 2 This is a schematic diagram of the CorNet neural network structure;

[0041] Figure 3 A schematic diagram of the process for optimizing and enhancing CLS marking;

[0042] Figure 4 This is a schematic diagram of the structure of a bidirectional gated loop unit;

[0043] Figure 5 This is a schematic diagram illustrating the computation process of graph convolution in a bidirectional graph convolutional neural network. Detailed Implementation

[0044] The technical solution of the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0045] This invention proposes a bidirectional GCN-BERT scientific data classification method based on rotational coding and dynamic gating, such as... Figure 1 As shown, it includes the following steps:

[0046] Step S1: Adaptive preprocessing is performed on the original text data to obtain the input data for training. Data cleaning involves removing stop words from the sample data, generating word and document embedding vectors using word vectors, calculating and normalizing the adjacency matrix, and constructing a heterogeneous graph representing document-word relationships using PMI and TF-IDF weights. This heterogeneous graph is used as input to a text classification model based on a graph convolutional network (GCN). This includes the following sub-steps:

[0047] Step S1-1: Data loading and preprocessing. Among them,

[0048] Word vector loading: Load pre-trained GloVe word vectors from the file glove.6B.300d.txt as word embedding vectors.

[0049] Data splitting involves reading document names and content from the prepared files {dataset}.txt and {dataset}.clean.txt. Then, based on the labels in the document name file (e.g., (index, train / test, label), the data is split into training and testing sets, and the training and testing sets are shuffled. The training set is then divided into true training and validation sets in a 9:1 ratio. Document names and content are shuffled according to the shuffled training and testing indices. These shuffled datasets are saved to new files {dataset}_shuffle.txt and {dataset}_shuffle.txt.

[0050] Vocabulary construction involves building a vocabulary from the shuffled document content. Stop words, such as "the," "a," and "is," are removed using `nltk.corpus.stopwords`. Text cleanup is performed, including lowercase conversion and punctuation removal. Word occurrences are counted, mapping each word to a list of documents in which it appears. The number of documents each word appears is calculated. A word-id map is created, mapping words to their corresponding IDs. The vocabulary is then saved to `{dataset}_vocab.txt`.

[0051] Tag extraction: Extract unique tags from document metadata and save them to {dataset}_labels.txt.

[0052] Step S1-2: Feature vector construction. Wherein,

[0053] The document feature vector (x, tx) is created for each document in both the training and test sets by averaging the word vectors of the words in the document. The resulting vector is stored as a sparse matrix using `scipy.sparse.csr_matrix`, where `x` is used for training and `tx` is used for testing. The formula for the document vector of each dimension `j` is:

[0054]

[0055] Where: world_vector i [j] represents the j-th dimension of the embedding vector of the i-th word in the document. doc_len is the number of words in the document. Checks have been added to handle cases where doc_len is 0, NaN, or Inf to prevent errors.

[0056] The label vector (y, ty) is created for each document using one-hot encoding. y is used for training, and ty is used for testing.

[0057] Combined Feature Vector (allx): Creates a combined feature matrix allx, which contains the document feature vector x and the word embedding vector.

[0058] Steps S1-3: Construct the graph object. Among them,

[0059] Window co-occurrence captures the co-occurrence relationships of words by creating a context window of size window_size from the segmented sentence.

[0060] Word pair counting is used to calculate the number of times word pairs co-occur within a window.

[0061] Adjacency Matrix (adj): An adjacency matrix adj is constructed to represent the graph, with edges weighted using vertex mutual information (PMI). The normalized calculation formula for the adjacency matrix A with added self-loops is as follows:

[0062]

[0063] Where: I is the identity matrix, and D is the degree matrix after adding self-loops, which is a diagonal matrix. It is the inverse square root of the degree matrix, that is:

[0064] D ii =∑ j A ij +1

[0065]

[0066] Where: n is the number of nodes. A ij This indicates whether an edge exists between nodes i and j; a value of 1 indicates an edge exists, and a value of 0 indicates a boundary exists. D ii This represents the degree of node i, which is the number of its neighboring nodes.

[0067] The formula for calculating PMI is as follows:

[0068]

[0069] Where: P(w1,w2) is the frequency of words w1 and w2 occurring together. P(w1) and P(w2) are the frequencies of words w1 and w2 occurring individually, respectively.

[0070] Document-Word Edges: Edges between documents and words are added to the adjacency matrix, with weights equal to the product of term frequency (TF) and inverse document frequency (IDF), calculated as follows:

[0071]

[0072] TF-IDF(t,d) = TF(t,d) × IDF(t)

[0073] Where: t represents a specific term, i.e., the word whose importance is to be evaluated. d represents a document, i.e., a text segment containing term t. D represents the document set, and N represents the total number of documents in the entire document set. |d∈D:t∈d| represents the number of documents containing term t, i.e., the number of documents in the document set that contain that term. TF(t,d) represents the number of times term t appears in document d.

[0074] Edges between words and between words and documents are defined based on PMI and TF-IDF, respectively. The weight of the edge between two nodes i and i is defined as:

[0075]

[0076] The process of constructing a DGL graph object mainly involves defining the graph structure and setting the features of nodes and edges. First, the indices of source and target nodes are extracted from the adjacency matrix, and these indices are used to create a DGL graph instance. Next, if a node feature matrix exists, it is added to the graph as node features; specifically, this involves converting the normalized adjacency matrix into a list of source and target nodes, creating a graph object using DGL's `graph` function, and then adding node and edge features by setting `ndata` and `edata` respectively, thus completing the construction of the graph object.

[0077] Step S2: Extract deep semantic features from the sample data using the RoBERTa pre-trained model to obtain the CLS token containing global information. RoBERTa is a powerful pre-trained language model capable of learning rich contextual information and semantic representations in text. The RoBERTa model architecture is a multi-layer Transformer network that captures relationships between words through a self-attention mechanism. In addition to the text sequence itself, a special CLS token (Classification token) is added to the beginning of the RoBERTa input. Each layer of RoBERTa encodes the entire input sequence, and the final output of the CLS token is designed to aggregate the information of the entire input sequence, thus serving as the representation vector of the entire sentence. This design ensures that the output vector of the CLS token contains global semantic information of the input text.

[0078] Load the pre-trained RoBERTa model and its corresponding tokenizer from the Hugging Face model library. The tokenizer is responsible for converting the input text into a sequence of token IDs that the RoBERTa model can understand. Obtain the output feature dimension of the penultimate layer of the RoBERTa model. This is because RoBERTa models typically include a final linear layer for task-specific fine-tuning, and the output of the penultimate layer is usually more suitable as a general semantic feature representation. Define a linear classifier that maps the output features of the BERT model to the number of classes.

[0079] During forward propagation, the input token ID sequence (input_ids) and attention mask are passed to the RoBERTa model. The hidden state matrix output by the RoBERTa model is obtained, and the output vector of the CLS token is selected as the semantic representation of the text. These features contain global contextual information and can represent the semantics of the text better than traditional bag-of-words models or TF-IDF methods.

[0080] Step S3 involves optimizing the CLS token's features using a CorNet residual network to enhance its feature representation; residual connections and nonlinear transformations are used to further enhance the expressive power of the features, such as... Figure 3 As shown, this includes: applying an activation function to the output distribution and applying the ELU activation function to the context vector to introduce nonlinearity. The transformed output distribution is then added to the original output distribution to form a residual connection. The structure of the CorNet neural network is as follows: Figure 2 As shown.

[0081] The input CLS features are nonlinearly transformed using the Sigmoid activation function:

[0082]

[0083] The input-output distribution is transformed into a context vector through a linear layer:

[0084] c = W1x + b1

[0085] The context vector is nonlinearized by applying the ELU activation function:

[0086] c = ELU(c)

[0087] The processed context vector is then transformed back to the output distribution through another linear layer:

[0088] x′=W2c+b2

[0089] The transformed output distribution is added to the original input output distribution to achieve residual connection:

[0090] x out =x′+x

[0091] Where: x is the output distribution of the input. W1, W2, b1, b2 are the weights and biases of the linear layer.

[0092] Step S4 further enhances the position and global interaction of the optimized CLS token through the Rotation Position Enhancement Multilayer Feature Interaction Network (RP-MLFIN). Based on an improvement on the Transformer, the RP-MLFIN achieves efficient long sequence modeling through a triple design of convolution enhancement, hierarchical feature modulation, and rotation position encoding. Absolute position information is encoded into the Q / K vector of attention computation through complex space rotation operations, preserving relative positional relationships more naturally compared to traditional position encoding. Low-complexity attention is achieved using ELU-activated QK projection and rotation position encoding, while retaining the dual residual structure of attention residuals and MLP residuals, enhancing gradient flow.

[0093] Given position p and frequency θk Its complex rotation encoding is:

[0094]

[0095] Where i is the imaginary unit. θ k It is a frequency-based scaling parameter.

[0096] Complex rotation operations are achieved by multiplying by the original feature:

[0097]

[0098] Q / K projection is linearized using the ELU function:

[0099] Q′=ELU(Q)+1, K′=ELU(K)+1

[0100] Here, Q and K are the query and key vectors, respectively. The ELU function is used to make both Q′ and K′ positive values, thus enabling cumulative calculations.

[0101] Step S5, embed a bidirectional gated recurrent unit (BiGRU) in the GCN, such as Figure 4 As shown, a bidirectional graph convolutional neural network based on BiGRU is constructed. A Dynamically Gated Feature Enhancement Network (DGFEN) is added after the output of each GCN layer and before the input to the GRU. DGFEN is a lightweight feature enhancer that combines gating mechanisms with additive attention. It achieves efficient feature enhancement through dynamic feature selection and global-local feature interaction, enabling the GRU to receive more expressive inputs, effectively improving model performance without significantly increasing the model's computational burden.

[0102] The dynamic gating weight A is obtained through the learnable parameter W. g The generation and calculation formula is as follows:

[0103]

[0104] Among them: Q norm It is the normalized positional feature. W g ∈R d×1 It is a learnable gating parameter vector. It is a scaling factor to prevent gradient explosion.

[0105] The formula for global feature aggregation is as follows:

[0106]

[0107] Among them: A i Q is the attention weight at position i. norm,i This is the normalized i-th positional feature. The calculation formulas for feature modulation and residual fusion are as follows:

[0108] G expand =Expand(G,N)

[0109] Y = W2·GELU(W1·(G expand ⊙K norm ))+Q

[0110] Among them: G expand This indicates that the global feature G is copied N times along the sequence dimension. W1 and W2 are learnable fully connected parameters. K norm These are the normalized local features. Q is the original feature. GELU is the Gaussian error linear unit activation function.

[0111] Step S6 involves feeding the optimized CLS token into a bidirectional graph convolutional neural network for feature interaction and graph convolution processing to generate a graph-level representation. Node features are extracted through the graph convolutional layer, and then BiGRU is used to capture the positive and negative contextual information of the features, ultimately generating a more comprehensive feature representation, such as... Figure 5 As shown.

[0112] The algorithm iterates through all graph convolutional layers, applying dropout regularization before each layer except the first. Finally, it performs the graph convolution operation, storing the output h of each GCN layer into the gru_input list, ready to be passed to the BiGRU. The graph convolution formula is as follows:

[0113]

[0114] Wherein: H (l) W is the input feature matrix of the l-th layer. (l) is the weight matrix of layer l. (A+I) is the adjacency matrix after adding self-loops. D is the degree matrix after adding self-loops. σ is the activation function.

[0115] If `gru_input` is not empty, it needs to be reversed and stacked, inputting the GCN outputs into the GRU in descending order of depth. Then, the outputs of all GCN layers are stacked at `dim=1` to satisfy the GRU input format. The stacked `gru_input` is then passed to the BiGRU, extracting the hidden states of the last time step `h` and the first time step `h_reverse` from the BiGRU output, representing the forward and reverse context information of the sequence respectively, and concatenating them. For the forward GRU, the calculation formula is as follows:

[0116] z_t=sigmoid(W_z·x_t+U_z·h(t_1))

[0117] r_t=sigmoid(W_r·x_t+U_r·h(t_1))

[0118] h′_t=tanh(W·x_t+U(r_t⊙h_(t_1)))

[0119] h_t=(1-z_t)⊙h_(t-1)+z_t⊙h′_t

[0120] Where: x_t is the input of GRU at time t, i.e., the sequence of output features from each layer of GCN concatenated. h_(t) is the hidden state of GRU at time t. z_t, r_t are the activation values ​​of the GRU update gate and reset gate. W_z, U_z, W_r, U_r, W, U are the weight matrices of GRU. ⊙ is the Hadamard product (element-wise multiplication).

[0121] For reverse GRU: the computation is similar to that of forward GRU, except that the time series is computed in reverse.

[0122] Step S7: The optimized CLS token and the features processed by graph convolution are passed to the classifier to generate BERT predictions and GCN predictions. The weights of both are dynamically allocated using an attention mechanism, and the prediction results are fused to generate the final classification decision. A fully connected layer and a Softmax function are used to calculate the attention weights, dynamically adjusting the contribution ratio of BERT and GCN. The output predictions of BERT and GCN are fused using the attention weights to generate the final classification decision. The prediction results of BERT and GCN are concatenated, and the attention weights are calculated through a linear layer, as shown in the following formula:

[0123]

[0124] The output of the linear layer is converted into attention weights using the Softmax function:

[0125]

[0126] w att =Softmax(a)

[0127] The final prediction result is obtained by weighting the prediction results of BERT and GCN using attention weights:

[0128] P combined =w att [0]·P cls +w att [1]·P gcn

[0129] The Log_Softmax function is used to normalize the fused prediction results to obtain the final prediction probability distribution:

[0130]

[0131] P final =log_Softmax(P combined )

[0132] Where: P cls P is the Softmax probability output of the BERT classifier. gcn This is the Softmax probability output of the GCN. 'a' is the output of the linear layer. 'z' is... i It is the i-th element of the input vector. n is the dimension of vector z, i.e., the number of categories. It is the sum of the exponents of all input elements, used for normalization. att It is the attention weight. P combined This is the prediction result after fusion. P final It is the final predicted probability distribution.

[0133] Next, experiments demonstrate the effectiveness of the bidirectional BERT-GCN scientific data classification method based on rotational position encoding and dynamic gating on classification tasks. Training was conducted on four public datasets, with accuracy used as the metric for classification results. Detailed dataset information is shown in Table 1. Experimental results of classification accuracy for different models on different text classification datasets are shown in Table 2.

[0134] Table 1

[0135]

[0136] Table 2

[0137]

[0138] The model of this invention was compared with state-of-the-art pre-trained models and GCN models: TextGCN, TextING, BERT, BERTGCN, RoBERTaGCN, and BERTGAT. Experimental results show that the proposed method outperforms baseline models such as TextGCN, TextING, BERT, BERTGCN, RoBERTaGCN, and BERTGAT on the MR, R8, and R52 datasets. Although slightly inferior to BERTGCN on the ohsumed dataset, it remains highly competitive. These experimental results strongly validate the effectiveness of the proposed method.

[0139] To evaluate model performance and verify the roles of the CorNet and RP-MLFIN networks, as well as the importance of the DGFEN and BiGRU modules, this study designed two ablation experiments: The first experiment evaluated the impact of removing the CorNet and RP-MLFIN networks on model performance. Specifically, the accuracy of the model using the present invention and the model without the CorNet and RP-MLFIN networks were compared. The second experiment aimed to investigate the impact of removing the DGFEN and BiGRU modules on model performance. Specifically, the accuracy of the model using the present invention and the model without the DGFEN and BiGRU modules were compared. In this way, the performance was compared with the original model, thereby accurately evaluating the roles of the CorNet and RP-MLFIN networks and the DGFEN and BiGRU modules in the model. The results of the accuracy comparison experiments are shown in Table 3.

[0140] Table 3

[0141]

[0142] This invention does not directly use the CLS token extracted by the RoBERTa model, but instead inputs it into the CorNet and RP-MLFIN networks for feature enhancement. The CorNet and RP-MLFIN networks utilize multi-layer nonlinear transformations, context modeling, residual connections, convolutional enhancement, hierarchical feature modulation, and rotational position encoding to effectively enhance the deep semantic features contained in the CLS token, making it more expressive. This optimized CLS feature provides high-quality input for subsequent graph convolutional operations in GCN.

[0143] This invention embeds DGFEN and BiGRU modules into a GCN to construct a bidirectional graph convolutional neural network. The output sequence of a multi-layer GCN is passed through DGFEN and then input into BiGRU. BiGRU captures the inter-layer sequence dependencies in the GCN output, thereby capturing the information interaction between different levels during graph convolution. The bidirectional graph convolutional neural network utilizes the graph structure information of the GCN, the dynamic feature filtering and global-local feature interaction of DGFEN, and the contextual modeling capabilities of BiGRU, achieving complementary advantages.

[0144] This invention dynamically fuses BERT predictions from optimized and enhanced CLS tokens and graph convolutional predictions from GCNs using an attention mechanism. The attention mechanism dynamically assigns weights to the BERT and GCN predictions based on the input samples, achieving more efficient prediction fusion and fully leveraging the advantages of both. This dynamic fusion strategy allows the model to adaptively adjust prediction weights according to data characteristics, thereby improving robustness and generalization ability.

Claims

1. A bidirectional GCN-BERT scientific data classification method based on rotational coding and dynamic gating, characterized in that, Includes the following steps: Step S1: Perform adaptive preprocessing on the original text data to obtain the input data for training; Step S2: Use the RoBERTa pre-trained model to extract deep semantic features from the sample data in the input data, and obtain the CLS token containing global information as a symbol to add to the beginning of the input text; Step S3: The CLS token is optimized using a CorNet neural network to enhance its feature representation. Step S4: The CLS token with optimized features is augmented and interacts globally through the Rotation Position Enhancement Multilayer Feature Interaction Network (RP-MLFIN). The rotation position enhanced multilayer feature interaction network RP-MLFIN is an improvement on Transformer. It achieves long sequence modeling through a triple design of sequentially concatenated convolution enhancement, hierarchical feature modulation, and rotation position encoding. The specific implementation process is as follows: Step S4-1: Apply depthwise separable convolution to the input features, perform depthwise convolution operation independently on each channel to extract local features; use 1x1 pointwise convolution to fuse channel information, and add the convolution output to the original input; Step S4-2: The hierarchical features are extracted step by step by stacking multiple Transformer layers; the contribution of each layer of features is dynamically adjusted using gating weights, and the features of different layers are fused by weighted summation or concatenation to generate the final representation. Step S4-3, rotation position encoding embeds position information into the feature vector through complex rotation operations. The feature vector is treated as a complex number, and the position information is encoded by a preset rotation angle. The feature dimension is divided into multiple two-dimensional subspaces, and each subspace is rotated independently; rotation position encoding is applied to query Q and key K respectively, and attention scores carrying relative position information are calculated. Step S5: The RP-MLFIN optimized CLS token is passed to a bidirectional graph convolutional neural network for feature interaction and graph convolution processing to generate a graph-level representation. The bidirectional graph convolutional neural network includes multiple graph convolutional layers and bidirectional gated recurrent units. It extracts node features through graph convolutional layers, performs feature enhancement using DGFEN, and then inputs them into a BiGRU to capture the positive and negative contextual information of the features, ultimately generating a comprehensive feature representation. The specific implementation process is as follows: Bidirectional gated recurrent unit (BiGRU) is embedded in GCN to construct a bidirectional graph convolutional neural network based on BiGRU; a dynamic gated feature enhancement network (DGFEN) is added after the output of each GCN layer and before the input of the GRU. The Dynamically Gated Feature Enhancement Network (DGFEN) is a lightweight feature enhancer that integrates gating mechanisms and additive attention. The gating mechanism dynamically adjusts the importance of features through learnable weights, generating gating weights through a linear transformation of the input features. These gating weights are then used to dynamically weight the input features. Additive attention models the global dependencies between features through nonlinear transformations, projecting the input features as query Q and key K respectively. An attention score is calculated using an additive function. Global information is aggregated based on the attention weights. DGFEN combines the gating mechanism and additive attention in a two-stage design. In step S6, the CLS token with optimized features and the graph-level representation are passed to the classifier to generate BERT predictions and GCN predictions. The weights of the two are dynamically allocated through an attention mechanism, and the prediction results are fused to generate the final classification decision.

2. The bidirectional GCN-BERT scientific data classification method based on rotational encoding and dynamic gating according to claim 1, characterized in that, The adaptive preprocessing in step S1 includes: data cleaning, removing stop words from the sample data, generating word and document embedding vectors, calculating the adjacency matrix A, and converting the normalized adjacency matrix into a graph object DGL.

3. The bidirectional GCN-BERT scientific data classification method based on rotational encoding and dynamic gating according to claim 2, characterized in that, The generation of word and document embedding vectors includes: loading pre-trained GloVe word vectors as word embedding vectors, and calculating the average word vector of each document as document embedding vectors.

4. The bidirectional GCN-BERT scientific data classification method based on rotational encoding and dynamic gating according to claim 3, characterized in that, In the implementation of the graph object DGL: a heterogeneous graph containing both word nodes and document nodes is constructed, and the pointwise mutual information (PMI) and term frequency-inverse document frequency (TF-IDF) are calculated respectively; edges between words and between words and documents are defined based on PMI and TF-IDF respectively; two nodes , The weight of the edge between and is defined as: 。 5. The bidirectional GCN-BERT scientific data classification method based on rotational encoding and dynamic gating according to claim 4, characterized in that, The step S2 of obtaining the CLS token containing global information includes: using the language structure and semantic information learned by BERT through pre-training, the input text is encoded and then the context information is captured using a multi-layer bidirectional Transformer; at the beginning of each input sequence, BERT inserts a CLS token, and its corresponding output vector integrates the semantic information of the entire sentence text.

6. The bidirectional GCN-BERT scientific data classification method based on rotational encoding and dynamic gating according to claim 5, characterized in that, The attention mechanism described in step S6 uses a fully connected layer and a Softmax function to calculate attention weights, dynamically adjusts the contribution ratio of BERT and GCN, and fuses the output predictions of BERT and GCN through attention weights to generate the final classification decision.

Images