Efficient graph comparison learning short text classification method based on heterogeneous multiple views

Abstract

The invention provides an efficient graph contrast learning short text classification method based on heterogeneous multiple views. The method comprises the following steps: S1, constructing a multi-source text graph; s2, performing representation learning on the heterogeneous text graph; s3, performing comparative learning on the heterogeneous multiple views; and S4, outputting a text classification result. According to the method, multi-source information mining of text content can be focused to overcome the defect that semantic information is insufficient, efficient and stable performance improvement is achieved, and then the robustness of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of text classification, and in particular to a short text classification method based on efficient graph contrast learning using heterogeneous multi-views. Background Art

[0002] The short text classification (STC) task has a wide range of applications, but the scarcity of semantic information and insufficient labeled data hinder accurate and efficient text recognition in industrial practice. Recent research has demonstrated the effectiveness of graph contrastive learning for text classification. However, contrastive learning is often accompanied by noise when introducing external corpora to strengthen semantic relationships, and the graph feature compression process further weakens the original semantics of the text. These chain reactions are detrimental to improving the accuracy and robustness of STC tasks. Summary of the Invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes an efficient graph contrast learning short text classification method based on heterogeneous multi-views.

[0004] In order to achieve the above-mentioned object of the present invention, the present invention provides an efficient short text classification method based on heterogeneous multi-view graph contrast learning, comprising the following steps:

[0005] S1, constructing a multi-source text graph;

[0006] S2, learning heterogeneous text graph representations;

[0007] S3, comparative learning of heterogeneous multi-views;

[0008] S4, output text classification results.

[0009] In a preferred embodiment of the present invention, step S1 includes:

[0010]

[0011] Among them, A w It is an adjacency matrix based on word co-occurrence statistics in a corpus generated within a text dataset;

[0012] [A w ] ij Represents the connection weight between different words i and j;

[0013] max(,) means taking the larger value;

[0014] PMI(,) represents the PMI value between word i and word j in point mutual information calculation;

[0015] Represents the real value of word i in the adjacency matrix;

[0016] Represents the real value of word j in the adjacency matrix;

[0017]

[0018] Among them, PMI(,) represents the PMI value between word i and word j in point mutual information calculation;

[0019] Indicates the number of words and their corresponding sliding windows;

[0020] represents the count of word i sliding window;

[0021] represents the count of word j sliding window;

[0022]

[0023] Among them, A e represents the entity adjacency matrix;

[0024] [A e ] ij Represents the adjacency value between entity i and entity j;

[0025] max(,) means taking the larger value;

[0026] cos(,) represents cosine similarity;

[0027] represents entity i;

[0028] Represents entity j.

[0029] In a preferred embodiment of the present invention, step S2 includes:

[0030]

[0031] in, Indicates the Embedded features in layer GCN;

[0032] σ() represents the activation function;

[0033] express The degree matrix

[0034] represents the self-loop adjacency matrix;

[0035] Indicates the Embedded features in layer GCN;

[0036] represents the trainable hyperparameter matrix;

[0037]

[0038] Among them, M e (i,j) represents the relationship between text i and entity j;

[0039] Indicates that if text i is contained in entity j;

[0040] M e (i,j)=1 means that text i is contained in entity j;

[0041] M e (i,j)=0 means that text i is not contained in entity j;

[0042] Z o =M o H o ,o∈{w,e,m} (6)

[0043] Among them, Z o Represents high-dimensional aggregate features for extracting multi-source information from short texts;

[0044] M o Represents the constructed text MD matrix;

[0045] H o Represents the original semantic embedding features;

[0046] Z org =Z w ||Z e ||Z m (7)

[0047] Among them, Z org represents the first contrastive view used for contrastive learning;

[0048] Z w Indicates word contrast attempt;

[0049] Z e Indicates entity comparison attempt;

[0050] Z m Represents the POS tag comparison view;

[0051] || represents feature concatenation operation.

[0052] In a preferred embodiment of the present invention, step S3 includes the following steps:

[0053] S31, graph-enhanced contrastive learning;

[0054] S32, contrastive learning for noise dimensionality reduction;

[0055] S33, Cluster Contrastive Learning.

[0056] In a preferred embodiment of the present invention, step S31 includes:

[0057]

[0058] Among them, H o,λ represents the enhanced feature map;

[0059] λ represents the intensity of the introduced random noise;

[0060] H o Represents the original semantic embedding features;

[0061] |||| means arbitrary modulus;

[0062] Z o,λ =M o H o,λ ,o∈{w,e,m} (9)

[0063] Z o,λ Represents the embedded features after graph augmentation operation;

[0064] M o Represents the constructed text MD matrix;

[0065] H o,λ represents the enhanced feature map;

[0066] Z emg =Z w,λ ||Z e,λ ||Z m,λ (10)

[0067] Among them, Z emg Indicates enhanced contrast view;

[0068] Z w,λ Representation word enhanced contrast view;

[0069] Z e,λ Represents entity enhanced contrast view;

[0070] Z m,λ Indicates POS tag enhancement comparison attempt;

[0071] || represents feature concatenation operation;

[0072]

[0073] represents contrast loss;

[0074] exp() represents the exponential function with the natural base e as the base;

[0075] P i Represents a positive sample pair (P i ,P j ) in the left sample;

[0076] P j Represents a positive sample pair (P i ,P j ) in the right sample;

[0077] P k Represents a positive sample pair (P i ,P j ) in the sample;

[0078] τ represents the temperature parameter of contrastive learning;

[0079] N represents the total number of short texts;

[0080] represents the indicator function;

[0081]

[0082] in, represents the total loss of the GECL module;

[0083] N represents the total number of short texts;

[0084] represents contrast loss.

[0085] In a preferred embodiment of the present invention, step S32 includes:

[0086]

[0087] in, represents the reversible diagonal matrix after full rank decomposition;

[0088] Σ p×q is a singular real-valued matrix;

[0089] M o,r =U o,r Σ o,r V o,r Τ ,o∈{w,e,m} (14)

[0090] Among them, M o,r Represents the characteristic matrix after MD matrix noise decomposition;

[0091] Σo,r Represents the first r feature matrices after singular value decomposition;

[0092] V o,r represents a right singular column vector;

[0093] V o,r Τ Indicates V o,r The transpose of

[0094] Z o,r =M o,r H o ,o∈{w,e,m} (15)

[0095] Among them, Z o,r Represents the text representation vector after noise decomposition;

[0096] M o,r Represents the characteristic matrix after MD matrix noise decomposition;

[0097] H o Representation represents the original semantic embedding features;

[0098] Z aug =Z w,r ||Z e,r ||Z m,r (16)

[0099] Among them, Z aug represents the noise decomposition contrast view;

[0100] Z w,r represents the noise decomposed word contrast view;

[0101] Z e,r Represents a noise decomposition entity comparison view;

[0102] Z m,r Represents the noise decomposition POS tag comparison view;

[0103] || represents feature concatenation operation;

[0104]

[0105] in, represents the contrast loss function of NDCL;

[0106] N represents the total number of short texts;

[0107] represents contrast loss;

[0108] γ represents the regularization coefficient;

[0109] W ndRepresents the trainable hyperparameter matrix of the NDCL module;

[0110] represents the square of L2 regularization.

[0111] In a preferred embodiment of the present invention, it also includes:

[0112]

[0113] in, represents the optimization objective function of CCL;

[0114] N represents the total number of short texts;

[0115] S i represents the set with the same predicted label as text i and does not contain text i; |S i | represents the set S i size;

[0116] Q i Represents a positive sample pair (Q i ,Q j ) in the left sample;

[0117] Q j Represents a positive sample pair (Q i ,Q j ) in the left sample;

[0118] Q k Represents a positive sample pair (Q i ,Q j ) in the sample;

[0119] τ represents the temperature parameter of contrastive learning;

[0120] χ represents the adaptive sliding loss weight;

[0121] Represents the indicator function.

[0122] In a preferred embodiment of the present invention, it also includes:

[0123]

[0124] in, represents the cross entropy loss;

[0125] T lab Represents a collection of label texts;

[0126] c weak represents a weak label set;

[0127] y ij represents the pseudo label of text j;

[0128] σ() represents the activation function;

[0129] W cls represents the trainable hyperparameter matrix for the cross entropy loss function;

[0130] Υ() represents the feature mapping function;

[0131] Z org represents the original text embedding vector;

[0132]

[0133] in, Represents the total loss function of the HEGCL model;

[0134] represents the cross entropy loss;

[0135] represents the optimization objective function of CCL;

[0136] represents the contrast loss function of NDCL;

[0137] represents the total loss of the GECL module;

[0138] η and Indicates control parameters.

[0139] The present invention also discloses a computer system, comprising:

[0140] processor;

[0141] a memory for storing processor-executable instructions;

[0142] Wherein, the processor is configured to implement the efficient graph contrast learning short text classification method based on heterogeneous multi-views when executing the executable instructions.

[0143] The present invention also discloses a computer-readable storage medium, comprising:

[0144] a memory having a computer program stored thereon;

[0145] A processor is used to execute the program in the memory to implement the short text classification method based on efficient graph contrast learning of heterogeneous multiple views.

[0146] In summary, due to the adoption of the above-mentioned technical solutions, the present invention can focus on multi-source information mining of text content to overcome the defect of insufficient semantic information, achieve efficient and stable performance improvement, and thus enhance the robustness of the model.

[0147] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0148] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0149] Figure 1 It is a schematic diagram of the HEGCL framework of the present invention.

[0150] Figure 2 is the present invention τ, η and Schematic diagram of classification results on all datasets.

[0151] Figure 3 Schematic diagram of the classification results of r and λ on all data sets of the present invention.

[0152] Figure 4 Schematic diagram of performance evaluation of training data with different proportions in the present invention.

[0153] Figure 5 This is a schematic diagram of the visualization comparison of GIFT and HEGCL clustering in the present invention. DETAILED DESCRIPTION

[0154] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0155] This patent proposes a high-efficiency graph contrastive learning model (HEGCL) based on heterogeneous multi-views for STC tasks. First, a multi-source heterogeneous graph at the word level, entity level, and tag level is constructed for the original text, which effectively avoids the problem of external noise confusing the semantics of the original text. Secondly, a two-layer graph convolutional network and the original text main-term (MD) matrix are used to extract enhanced views to form multi-source feature maps. The enhanced view can supplement the lost semantic information when performing feature compression operations in text representation learning to overcome the semantic weakening defects in the text semantic information mining process. Finally, multi-view contrastive learning of the three modules GECL, NDCL, and CCL is performed for STC. We conducted extensive experiments on six real datasets and the results showed that HEGCL has higher classification performance and model robustness than other state-of-the-art methods on STC tasks.

[0156] 1 Introduction

[0157] Text classification (TC) is widely recognized as the most fundamental and typical task in natural language processing (NLP) and has long been a research hotspot in the field. Currently, deep learning (DL)-based TC methods have achieved great success in processing text rich in semantic information and are widely used in downstream tasks such as machine translation, sentiment analysis, news recommendation, and false information detection. However, many related research methods require large amounts of labeled data and computing power, which increases the cost of data annotation and computing resources and is inconsistent with reality. Furthermore, the large amount of short text such as news, tweets, and comments generated by social media is large in scale and semantically sparse. This makes traditional DL methods face even greater challenges in tackling short text classification. Recently, the combination of graph neural networks (GNNs), which effectively capture the semantic relationships between short texts, and self-supervised contrastive learning (CL) algorithms, trained with small amounts of labeled data, has achieved encouraging results on STC tasks.

[0158] There are two typical approaches to using graph contrastive learning (GCL) to generate augmented views for classification tasks. First, these approaches use random noise perturbation and node removal to generate enhanced contrastive views that bring positive CL pairs closer together and push negative pairs apart, resulting in a more accurate semantic representation of short texts, which can be used to improve classification performance. However, these approaches can cause irreversible information loss for short texts where semantic information is extremely scarce. For example, in a user review like "I don't like these shoes," randomly removing the word "no" can completely reverse the sentiment of the text, ultimately leading to incorrect label predictions and poor recommendation. Second, in scenarios with abundant semantic information, the CL augmentation strategy, widely used in computer vision and text classification tasks, employs fixed heterogeneous graph node / edge perturbations to mine long-range dependencies between different entities. However, for short texts with less semantic information, the noise generated by augmenting the relationships between entities and category-related words can affect the learning process of the contrastive views, thus limiting model training and optimization. Therefore, how to improve graph contrastive learning while reducing noise interference and study a graph contrastive learning strategy suitable for STC tasks that improves classification performance and enhances model robustness is crucial to the development and application of short text classification.

[0159] The GCL method learns representations by comparing local graph structural information, making it an advanced self-supervised representation learning method. It inherits most of the advantages of supervised learning methods and can learn and predict with minimal labeled data. It outperforms supervised learning models and pre-trained language models in many tasks. Recently, attempts have been made to improve graph contrast learning by employing strategies such as improved graph contrast, the construction of specialized convolutional kernels, regularization, and the integration of large language models. These methods have been widely applied in tasks such as short text classification, recommender systems, and image segmentation, achieving promising results. However, any single optimization scheme that improves the GCL paradigm only marginally improves the overall model performance. For example, constructing specialized convolutional kernels for feature extraction in graph convolutional networks (GCNs) can produce better high-dimensional information graphs, improving model performance to a certain extent. However, this approach does not address the semantic sparsity inherent in short texts, which lack semantic information, and its contribution to enhancing model robustness is minimal.

[0160] To solve the above problems, we propose a model HEGCL for STC. In short texts, HEGCL focuses on multi-source information mining of text content to overcome the lack of semantic information, achieve efficient and stable performance improvement, and thus enhance the robustness of the model. First, for the original text, we construct graph structures at the word level, entity level, and Part-Of-Speech (POS) tag level (including verbs, nouns, and their variants in the text). Secondly, we use a standard two-layer GCNs to extract the high-dimensional feature map H of the multi-source information graph. o , and construct the original text MD matrix M o , polymer H o and M o Generate text embedding Z org We are working on H o Perform graph enhancement to generate enhanced view H o,λ , for M o Perform singular value decomposition to obtain the enhanced view M with top r ranking o,r Finally, text embeddings for contrastive learning are generated after text representation learning, and contrastive learning is performed on them under three different constraints. Specifically: 1) Graph Enhanced Contrastive Learning (GECL): o and H o,λ Perform feature map aggregation to generate text embedding Z emg , and Z org The sample pairs that form contrasting views are used for GECL; 2) Noise Decomposition-Oriented Contrastive Learning (NDCL): After performing singular value decomposition and approximate maximum r selection on the MD matrix, H o and M o,r Perform aggregation operation to generate text embedding Z aug , and Z orgThe sample pairs that form contrasting views are used for NDCL; 3) Clustering-oriented contrastive learning (CCL): Using the label data and its corresponding feature map as the information source, the Kmeans model is introduced to learn the labels and features, and then weak labels are assigned to the unlabeled text, and the original text is embedded in Z org Perform CCL process.

[0161] The main contributions of this invention are summarized as follows:

[0162] This paper proposes a new efficient short text classification model HEGCL, which is committed to deeply mining the multi-perspective semantic information of the original short text and effectively reducing the impact of noise on the STC task.

[0163] By representing the multi-source information and generating contrastive views of the original semantics of short texts, we integrate GECL, NDCL, and CCL to guide the learning process of heterogeneous feature multi-views.

[0164] Extensive comparative experiments and analyses were conducted on six short text datasets, and the experimental results verified the advanced nature of the HEGCL model.

[0165] 2 Related Work

[0166] In this section, we review the related research on short text classification, graph neural networks for short text classification, and contrastive learning for short text classification.

[0167] 2.1 Short Text Classification

[0168] Limited by insufficient data annotation and uneven category distribution for short texts, the STC task places more stringent and comprehensive semantic mining requirements on its processing models. Some methods have introduced active sampling strategies and class distribution feature selection strategies to effectively address the insufficient data annotation issues for short texts. However, while these methods focus on the distribution and annotation of the overall data, they often overlook the difficulty of semantic mining within the short text sentences themselves. Therefore, Ma et al. employed a local self-attention mechanism to enhance the original semantic information and designed a deformable context extraction module to replace the traditional fixed sliding window, improving the model's effectiveness for STC. Xu et al. integrated an adaptive text classification framework (AdaDEM) based on DensNet, significantly improving semantic mining performance across multiple basic backbone networks for STC tasks. These methods improve classification performance to a certain extent by strengthening and integrating classifiers. However, they struggle to balance performance advantages with computational resources. Consequently, methods based on pretrained language models (PLMs) have emerged. The representative BERT and RoBERTa learn semantic relationships between words and sentences through mask and dynamic mask prediction respectively, and have advanced performance advantages in multiple NLP tasks including STC. To address the flexibility and interpretability issues of PLMs, Wang et al. proposed an inductive text learning framework based on perceptual text semantic interaction (PaSIG), which achieved significant performance improvements and fast text reasoning in STC tasks. In response to the interpretability issue of PLMs models, Wen et al. constructed a model based on a multi-head graph attention network for text classification tasks to make interpretable predictions for the feature extraction process. It aims to make decisions through the weights obtained from the linear representation of each graph attention to enhance the transparency of the model, and has verified its advanced interpretability of black box models through a large number of case experiments.

[0169] 2.2GNNs for STC

[0170] Graph-based processing schemes, with their flexibility and advanced deep relationship mining capabilities, have demonstrated excellent performance in images, text, and other information. By capturing heterogeneous topological relationships and mining graph structures, they have been widely applied in research such as image segmentation, text classification, and recommender systems. TextGCN, the first GCN to be used for STC tasks, provided a new research path. Its variants, HeteGCN, STGCN, and Induct-GCN, have been improved for STC. However, research on GCN improvements has shown that further information extraction from sparse graph features in STC tasks results in a significant loss of semantic information, presenting certain limitations. To address this, Velickovic et al. proposed a novel graph attention network (GAT), which effectively improves the overall performance of STC tasks by capturing local and global semantic relationships in text and adaptively learning the importance weights of graph nodes. In text classification work based on GAT, Linmei et al. proposed a heterogeneous graph attention network (HGAT) to address the lack of labeling and sparsity of short text data. Its heterogeneous information network (HIN) structure effectively captures additional semantic information from short texts. Subsequently, Sun et al. combined HGAT's heterogeneous information network with contrastive learning to propose the NC-HGAT model, significantly improving the accuracy of short text classification based on HGAT. Yang et al. extended HGAT to multi-label classification, achieving very good results and arguing that HGAT possesses advanced information mining advantages in both conductive and inductive learning. Graph-based deep learning methods, particularly GCN and GAT and their variants, have significantly improved the performance of short text classification tasks by efficiently mining heterogeneous topological relationships and adaptively learning node weights, demonstrating excellent semantic information capture.

[0171] 2.3CL for STC

[0172] For traditional machine learning and deep learning methods, CL's processing model undoubtedly provides a new research path for the training process, more aligned with the development and application of model optimization methods under real-world conditions. The core idea of ​​CL is a self-supervised learning paradigm that learns pairs of positive and negative examples in an embedding space, aligning the representations of similar texts and different texts. Sun et al. first applied graph contrastive learning to short text classification, finding its excellent performance in training with a small number of examples and helping the model capture textual features. Subsequently, Su et al. combined k-nearest neighbor (kNN) with CL to form a new training framework to obtain rich graph embeddings from known instances of similar categories. They validated this framework's advanced multi-label classification performance and portability across multiple models, including pre-trained and non-pre-trained models. Zhang et al. constructed a new relational learning network (R2-Net) from the perspective of mining the value of label information. They incorporated external knowledge from a corpus to enhance label embeddings for contrastive learning to guide the training process of short text classification. Furthermore, numerous CL variants have emerged, including those for graph-structured segmentation, robust causal contrastive analysis, and contrastive adversarial training. While these methods are effective for STC, their high computational overhead and complex model architectures prevent them from balancing efficiency and robustness to maximize performance gains. Liu et al. proposed an improved graph contrastive learning model called GIFT, which achieves significant gains for STC tasks. Therefore, exploring efficient short text classification models that balance effort and benefit is crucial.

[0173] 3 Proposed method

[0174] 3.1 Main Framework

[0175] In this section, we will give a detailed introduction to the main framework and model principles of the HEGCL model for STC tasks. Figure 1 The overall framework of HEGCL is shown. Its main components include multi-source text graph representation learning and heterogeneous multi-view comparative learning. HEGCL constructs a graph structure of source text from multiple information sources, uses heterogeneous graphs for feature enhancement to explore the high-dimensional semantic structure of multi-source graphs, and then generates heterogeneous text feature embeddings for comparative learning through text representation learning.

[0176] Specifically: 1) Multi-source text graph representation learning: Considering that semantic information from multiple sources plays a significant role in semantic mining of short texts, the HEGCL model introduces semantic information at the word level, entity level, and POS tag level for graph construction. A two-layer graph convolutional network (GCN) and MD matrix construction are used to learn graph representations of multi-source text graph embeddings to generate high-dimensional feature graphs Ho and Mo. 2) Heterogeneous multi-view contrastive learning: To improve the effectiveness and robustness of contrastive learning, the graph enhancement module and maximum rank decomposition are used to enhance the feature graphs Ho and Mo, respectively, to form heterogeneous feature multi-views. Based on the heterogeneous enhanced multi-views, sample pairs can be closer to the same category text in the sample space, while promoting further separation of negative sample pairs. Then, when performing GECL, NDCL, and CCL, more category distinctions are obtained, which reasonably expands the semantic information of short texts and effectively improves the classification performance of STC.

[0177] 3.2 Learning Multi-source Text Graph Representations

[0178] 3.2.1 Multi-source text graph construction

[0179] In order to effectively avoid the semantic interference of external knowledge on the text dataset itself, we use the corpus formed between sentences in the text as auxiliary knowledge for the construction of the graph structure. First, the construction of the heterogeneous graph uses a part-of-speech tagger to annotate the information of the preprocessed short text sentences. The word-level, entity-level and POS-level corpora of the dataset can be obtained. Similar auxiliary information is supplemented for semantically sparse text to alleviate the occurrence of semantic sparsity and ambiguous misjudgments. The multi-source information graph includes word graphs Entity Graph and POS graph The specific construction process is as follows:

[0180] The word graph is represented as It is composed of short text words that have been preprocessed. is a collection of words, using the pre-trained GloVe_300d word vector for word embedding, expressed as is a collection of words The size of f w It is word frequency. It is an adjacency matrix based on word co-occurrence statistics in a corpus generated within a text dataset. In order to establish strong and weak relationships between nodes between important words in a short text, A w Each value in is calculated using point-wise mutual information (PMI), and the calculation principle is shown in formula (1).

[0181]

[0182] Among them, A wIt is an adjacency matrix based on word co-occurrence statistics in a corpus generated within a text dataset;

[0183] [A w ] ij Represents the connection weight between different words i and j;

[0184] max(,) means taking the larger value;

[0185] PMI(,) represents the PMI value between word i and word j in point mutual information calculation;

[0186] Represents the real value of word i in the adjacency matrix;

[0187] Represents the real value of word j in the adjacency matrix;

[0188] Among them A w Each value of represents the PMI value corresponding to node i and node j and the maximum value of 0, generating the connection strength relationship between nodes and edges in the information graph. The PMI value of the edge weight between node i and node j is calculated using formula (2):

[0189]

[0190] Among them, PMI(,) represents the PMI value between word i and word j in point mutual information calculation;

[0191] Indicates the number of words and their corresponding sliding windows;

[0192] represents the count of word i sliding window;

[0193] represents the count of word j sliding window;

[0194] in represents the number of words and their corresponding sliding windows, and Then Λ represents the sliding window count of word i, and Λ is the total number of sliding windows in the short text dataset that constitutes the corpus. Λ(i,j) represents the correlation between node i and node j, and Λ(i) represents the sliding window at corpus i.

[0195] The entity graph is represented as It consists of entities residing in the short text knowledge graph. It is a collection of entities that can provide auxiliary semantic information for short texts. is the entity embedding after TransE operation, is a collection of entities The size of f eIndicates the frequency of entities. It can supplement the missing information of the same semantics of different short sentences, thereby enhancing the multi-semantic information of heterogeneous graphs. Entity adjacency matrix The cosine similarity between entity pairs is used to obtain the similarity, and its calculation principle is shown in formula (3).

[0196]

[0197] Among them, A e represents the entity adjacency matrix;

[0198] [A e ] ij Represents the adjacency value between entity i and entity j;

[0199] max(,) means taking the larger value;

[0200] cos(,) represents cosine similarity;

[0201] represents entity i;

[0202] represents entity j;

[0203] When the cosine similarity between entities is closer to 1, it can be considered that the correlation between the entity pairs is stronger. When , it is considered that there is no semantic information correlation between the entity pairs and the value is assigned to 0.

[0204] For short texts such as a sentence, a short comment, or a title, accurate expression of nouns and accurate identification of verbs can better determine the overall direction of the content.

[0205] POS tag graph The tags are composed of verbs, nouns and their variants in short texts. They can perform more fine-grained semantic mining of short texts through accurate division of grammatical structures and eliminate ambiguous relationships between entities and modifiers. is the POS tag set, Generate token embeddings by initializing the one-hot vectors composed of token sets, POS tag set The size of f m Indicates the frequency of POS tags in the corpus, marking the adjacency matrix Use A w The same generation method.

[0206] We use word-level, entity-level, and POS tag sets as graph nodes, and the corresponding adjacency matrix as weighted edge relationships. We construct a heterogeneous view of multi-source short texts, extracting richer semantic features from the original text. This approach has the advantage of not introducing external noise and effectively compensating for the lack of semantic information in short texts.

[0207] 3.2.2 Heterogeneous Text Graph Representation Learning

[0208] After constructing the text graph structure, we will use two layers of GCNs to encode it to effectively combine the structural information and node features of the heterogeneous graph. The feature graph extraction principle of GCNs is shown in Equation (4).

[0209]

[0210] in, Indicates the Embedded features in layer GCN;

[0211] σ() represents the activation function;

[0212] express The degree matrix

[0213] represents the self-loop adjacency matrix;

[0214] Indicates the Embedded features in layer GCN;

[0215] represents the trainable hyperparameter matrix;

[0216] in, Indicates the Embedded features in layer GCN. When H (0) is the initial feature X. Self-loop adjacency matrix The calculation formula is Where I represents the identity matrix and A represents the real-valued adjacency matrix. yes The degree matrix of Represents the value of the element in the i-th row and i-th column of the degree matrix, Represents the self-loop adjacency matrix The value of the element in row i and column j in . σ represents the activation function, specifically ReLU; is a trainable hyperparameter matrix. The symmetric normalization operation and the first-order neighbor aggregation of heterogeneous feature maps are used for training optimization, which can effectively ensure the stability of features and prevent gradient explosion and gradient disappearance problems.

[0217] Therefore, we construct a multi-source information graph Formula (4) can be used to extract and update node embeddings of multi-source information Where o∈{w,e,m}. Interaction between heterogeneous graphs generated by different information sources can more accurately express the semantic information of short texts.

[0218] However, there is still a problem of semantic loss in the process of feature extraction of multi-source information graphs by GCNs. In order to effectively retain the feature information of short texts, we adopt A specific matrix (MD) for different short texts is constructed to retain the important semantic information of the short text. Specifically: for the MD matrix construction of word-level and POS tag-level feature maps, the TF-IDF values ​​of words and POS tags in the corpus are used to create them, and we can get and Where N is the total number of short texts, M w Represents the word MD matrix, M m represents the POS tag MD matrix. And the entity level The MD matrix is ​​constructed through inclusion relationship, and the calculation principle is shown in formula (5).

[0219]

[0220] Among them, M e (i,j) represents the relationship between text i and entity j;

[0221] Indicates that if text i is contained in entity j;

[0222] M e (i,j)=1 means that text i is contained in entity j;

[0223] M e (i,j)=0 means that text i is not contained in entity j;

[0224] From this, we construct the text MD matrix M o The semantic view of multi-source information of short text can be represented as text embedding Z through the following aggregation operation org , as shown in Equation (6) and Equation (7).

[0225] Z o =M o H o ,o∈{w,e,m} (6)

[0226] Among them, Z o Represents high-dimensional aggregate features for extracting multi-source information from short texts;

[0227] Mo Represents the constructed text MD matrix;

[0228] H o Represents the original semantic embedding features;

[0229] Z org =Z w ||Z e ||Z m (7)

[0230] Among them, Z org represents the first contrastive view used for contrastive learning;

[0231] Z w Indicates word contrast attempt;

[0232] Z e Indicates entity comparison attempt;

[0233] Z m Represents the POS tag comparison view;

[0234] || represents feature concatenation operation;

[0235] in Represents the high-dimensional aggregate features extracted from short text multi-source information. At the same time, the first contrast view Z for contrastive learning is obtained org . We will then o and M o Perform view enhancement and singular value decomposition selection to obtain H o,λ and M o,r , and then get two more comparison views.

[0236] 3.3 Heterogeneous Multi-view Contrastive Learning

[0237] 3.3.1 Image-enhanced contrastive learning

[0238] Graph enhancement contrastive learning effectively represents graph structure data, so that the original graph can be transformed to generate a new enhanced view. In the contrastive learning module for graph enhancement, we will introduce a random noise perturbation strategy to the original feature graph H o Perform enhancement operations. The construction principle of the enhanced view is as follows:

[0239]

[0240] Among them, H o,λ represents the enhanced feature map;

[0241] λ represents the intensity of the introduced random noise;

[0242] H o Represents the original semantic embedding features;

[0243] |||| means arbitrary modulus;

[0244] Where λ is the strength of the introduced random noise. In order to enhance the feature map H o,λ The enhanced perturbation of high-dimensional feature maps when aggregated into contrasting views, λ∈{0.1, 0.3, 0.5, 0.7, 0.9}. Then, according to Equations (6) and (7), we transform M o With H o,λ Perform information aggregation to generate enhanced contrast view Z emg .

[0245] Z o,λ =M o H o,λ ,o∈{w,e,m} (9)

[0246] Z o,λ Represents the embedded features after graph augmentation operation;

[0247] M o Represents the constructed text MD matrix;

[0248] H o,λ represents the enhanced feature map;

[0249] Z emg =Z w,λ ||Z e,λ ||Z m,λ (10)

[0250] Among them, Z emg Indicates enhanced contrast view;

[0251] Z w,λ Representation word enhanced contrast view;

[0252] Z e,λ Represents entity enhanced contrast view;

[0253] Z m,λ Indicates POS tag enhancement comparison attempt;

[0254] || represents feature concatenation operation;

[0255] Then, Z emg With Z org Aggregate and generate text representation Z, and map Z to the hidden space using contrast loss through the projection head Φ(·). Then normalize the hidden representation of the contrast view and pass it into the GECL contrast loss. That is, P = norm(Φ(Z)), where norm() represents the normalization operation. At this time, the contrast loss It can be expressed as formula (12).

[0256]

[0257] represents contrast loss;

[0258] exp() represents the exponential function with the natural base e as the base;

[0259] P i Represents a positive sample pair (P i ,P j ) in the left sample;

[0260] P j Represents a positive sample pair (P i ,P j ) in the right sample;

[0261] P k Represents a positive sample pair (P i ,P j ) in the sample;

[0262] τ represents the temperature parameter of contrastive learning;

[0263] N represents the total number of short texts;

[0264] represents the indicator function;

[0265]

[0266] in, represents the total loss of the GECL module;

[0267] N represents the total number of short texts;

[0268] represents contrast loss;

[0269] Among them (P i ,P j ) is a positive sample pair, i and j represent the index of the same sentence. is the indicator function, set to 1 when k≠i, otherwise 0. τ is the temperature parameter of contrastive learning, which controls the weight distribution of positive and negative samples in contrastive loss, and τ∈{0.1, 0.3, 0.5, 0.7, 0.9} is any value in the set. In summary, the loss value corresponding to each index of the short text can be calculated Finally, the total loss of the GECL module is calculated by formula (12).

[0270] GECL learns the essential characteristics of graph data by enhancing the feature maps extracted by GCNs, which is used for comparative learning to effectively train the semantic mining capabilities of the model and improve the generalization performance of the model.

[0271] 3.3.2 Contrastive Learning with Noise Dimensionality Reduction

[0272] GECL generates enhanced text views to guide contrastive learning. The random noise strategy can effectively distinguish the positive and negative sample pairs of CL. On the other hand, it inevitably falls into the defects of traditional contrastive learning optimization schemes such as feature loss and ambiguity. Therefore, in order to balance the generalization performance and noise resistance of the contrastive learning paradigm, we introduce the approximate maximum singular value selection strategy of singular value decomposition (SVD) for different types of MD matrices M. o Perform feature map enhancement.

[0273] Specifically, we first define a full rank r max The matrix M p×q , if there exists a unitary matrix U p×p and V q×q So that M=UΣV Τ .in and denote the orthogonal matrices to be left-multiplied and right-multiplied, respectively. It's M p×q The number of column vectors, A real-valued matrix that stores the singular values ​​in descending order. Σ can be expressed as Equation (13).

[0274]

[0275] in, represents the reversible diagonal matrix after full rank decomposition;

[0276] Σ p×q is a singular real-valued matrix;

[0277] Where Δ is a reversible diagonal matrix that implements M p×q Singular value decomposition of the matrix. The first 10% or even the first 2% of the singular values ​​of Δ account for more than 90% of the total singular values. Therefore, we have the r on the main diagonal of Δ max The values ​​are sorted and the top-r ranked values ​​are selected as the maximum information retention matrix for the heterogeneous feature map M o The feature enhancement is achieved by reducing the data dimension and effectively denoising the features generated by the text. The calculation process of SVD decomposition and top-r selection is as follows:

[0278] M o,r =U o,r Σ o,r V o,r Τ ,o∈{w,e,m} (14)

[0279] Among them, M o,r Represents the characteristic matrix after MD matrix noise decomposition;

[0280] Σ o,r Represents the first r feature matrices after singular value decomposition;

[0281] V o,r represents a right singular column vector;

[0282] V o,r Τ Indicates V o,r The transpose of

[0283] After Σ o,r The most important semantic information of the MD matrix is ​​obtained by approximating the top-r of r∈{5,10,15,20,25}. Then, U o,r and V o,r The column vectors are M o,r The left singular vector and right singular vector of the singular value are then used as the denoising view of the MD matrix for the heterogeneous noise decomposition view of the STC task to guide contrastive learning. At this time, according to the heterogeneous information graph aggregation operation of Equations (6) and (7), the noise decomposition contrast view Z can be obtained aug .

[0284] Z o,r =M o,r H o ,o∈{w,e,m} (15)

[0285] Among them, Z o,r Represents the text representation vector after noise decomposition;

[0286] M o,r Represents the characteristic matrix after MD matrix noise decomposition;

[0287] H o Representation represents the original semantic embedding features;

[0288] Z aug =Z w,r ||Z e,r ||Z m,r (16)

[0289] Among them, Z aug represents the noise decomposition contrast view;

[0290] Z w,r represents the noise decomposed word contrast view;

[0291] Z e,r Represents a noise decomposition entity comparison view;

[0292] Z m,r Represents the noise decomposition POS tag comparison view;

[0293] || represents feature concatenation operation;

[0294] Similar to GECL, we also set Z org and Z aug Aggregation generates Z, and through projection and normalization operations P = norm(Φ(Z)), it is passed into Equations (11) and (12) for contrastive learning optimization. However, using high-quality training data to train the model will cause overfitting. To solve this problem, we added an L2 regularization term to the traditional contrastive learning loss to limit the complexity of the model and prevent overfitting through effective parameter regulation. Specifically, the contrastive loss function of NDCL obtained by calculation using Equation (11) is as follows:

[0295]

[0296] in, represents the contrast loss function of NDCL;

[0297] N represents the total number of short texts;

[0298] represents contrast loss;

[0299] γ represents the regularization coefficient;

[0300] W nd Represents the trainable hyperparameter matrix of the NDCL module;

[0301] represents the square of L2 regularization;

[0302] in represents the total loss of the noise decomposition contrastive learning module, W nd is the trainable hyperparameter matrix of the NDCL module. γ is the regularization coefficient, which is set to 5e-3.

[0303] By integrating GECL and NDCL, we implement an efficient contrastive learning model for STC tasks that effectively balances model robustness and enhances noise resistance.

[0304] 3.3.3 Cluster Contrastive Learning

[0305] As mentioned above, relying on information enhancement or denoising of text graphs can achieve very good STC task performance improvement for CL. However, in the contrastive learning paradigm of self-supervised learning, we have very little labeled data information. Therefore, how to effectively use labeled data to guide the training process of contrastive learning to enhance the robustness of the contrastive learning ensemble model is an important factor taken into consideration in the improvement work of the patent of this invention. By using the prior knowledge brought by the very few labeled texts, a constrained Kmeans clustering algorithm is introduced to assign weak labels to unknown texts. The clustering-oriented method guides the contrastive learning of the assigned weak labels to bring together the text features of similar semantic distributions and push apart the text features of different semantic distributions, thereby realizing a clustering-guided contrastive learning process.

[0306] Specifically, the self-supervised process of contrastive learning assigns a small amount of labeled data to each category in the dataset. This gives Kmeans a clear spatial definition condition for the original high-dimensional feature embedding vector Z generated after feature extraction and text representation learning stages. org Provide contrastive learning for clustering optimization process. Let Z org For a given text T = {t1, t2, ..., t n}The corresponding feature embedding vector represents the labeled text set as Where s is the total number of label texts, y i Represents text Z org,i The corresponding label, L l To determine whether a feature is labeled, when l = 1, there is corresponding category information c. By processing the text embeddings of labels of the same category, we can obtain the cluster center range of that category. During the training process, we assign weak labels to the embedding vectors of unknown labels based on their distribution. This allows us to adjust and bring together text features corresponding to weak labels with the same distribution to guide the CCL process.

[0307]

[0308] At this point, we can use the projection head Ψ(·) with category information judgment to embed the original text into the vector Z org Mapping to the CL latent space generates Z, and through the normalization operation Q = norm(Ψ(Z)), norm() represents the normalization operation. The optimization objective function of CCL is shown in Equation (18).

[0309]

[0310] in, represents the optimization objective function of CCL;

[0311] N represents the total number of short texts;

[0312] Si represents the set with the same predicted label as text i and does not contain text i;

[0313] |S i | represents the set S i size;

[0314] Q i Represents a positive sample pair (Q i ,Q j ) in the left sample;

[0315] Q j Represents a positive sample pair (Q i ,Q j ) in the left sample;

[0316] Q k Represents a positive sample pair (Q i ,Q j ) in the sample;

[0317] τ represents the temperature parameter of contrastive learning;

[0318] χ represents the adaptive sliding loss weight;

[0319] represents the indicator function;

[0320] Where Q has the same meaning as P above, S i represents the set that has the same predicted label as text i but does not contain text i. χ is the adaptive sliding loss weight, which decreases linearly with the increase of epochs in the range of 0.9 to 0.4.

[0321] 3.4 Model Optimization

[0322] Embed vector Z with original text org and its assigned weak label set c weak As the known information, a new projection head Υ(·) is introduced to convert Z org Mapping to the cross-entropy loss function latent space is used for conventional classification. Projection head Υ(·), projection head Φ(·), and projection head Ψ(·) are different feature map mapping methods, and are built-in functions of third-party libraries. The projection heads involved represent differences between different high-dimensional feature map mapping methods, and different feature maps generate different probability distributions. Specifically, it can be expressed as:

[0323]

[0324] in, represents the cross entropy loss;

[0325] T lab Represents a collection of label texts;

[0326] c weak represents a weak label set;

[0327] y ij represents the pseudo label of text j;

[0328] σ() represents the activation function;

[0329] W cls represents the trainable hyperparameter matrix for the cross entropy loss function;

[0330] Υ() represents the feature mapping function;

[0331] Z org represents the original text embedding vector;

[0332] where y ij is the pseudo label of text j, W cls Represents the trainable hyperparameter matrix for the cross entropy loss function. i ∈T lab And the predicted label c i ∈c weak When , the cross entropy loss function can be optimized and used for STC.

[0333] In summary, the total loss function of the HEGCL model can be expressed as:

[0334]

[0335] in, Represents the total loss function of the HEGCL model;

[0336] represents the cross entropy loss;

[0337] represents the optimization objective function of CCL;

[0338] represents the contrast loss function of NDCL;

[0339] represents the total loss of the GECL module;

[0340] η and Indicates control parameters;

[0341] where η and To control the parameters, we set them to the same value to stabilize the training optimization process of NDCL and CCL modules. And set the weight of GECL module to The contributions of NDCL and CCL to different contrastive learning are balanced. The training execution pseudo code of the HEGCL model is shown in Algorithm 1.

[0342] 4 Experiments

[0343] In this section, we conduct extensive experiments on six real-world datasets and 18 baselines to verify the rationality and effectiveness of the HEGCL method proposed in this patent on STC tasks. We first provide a detailed introduction to the datasets used, baselines used for comparison, evaluation metrics, and implementation details. We then verify the advanced performance advantages of the HEGCL model on STC tasks through extensive experiments and analysis.

[0344] 4.1 Dataset

[0345] We conduct experiments on six datasets: MR, Twitter, Snippets, Ohsumed, TagMyNews, and StackOverflow. Table 1 summarizes the basic information of these datasets, which are then introduced in detail below.

[0346] Table 1 Statistics of evaluation datasets

[0347]

[0348] MR (https: / / www.cs.cornell.edu / people / pabo / movie-review-data / ), proposed by Pang and Lee, is a binary classification dataset consisting of 10,662 movie reviews. The dataset has 5,331 review texts in each of the positive and negative categories, with an average text length of 7.6.

[0349] Twitter (http: / / cs.stanford.edu / people / alecmgo / trainingandtestdata.zip) is a binary sentiment classification dataset collected by NLTK. It contains 5,000 comments with positive and negative emotions respectively, with an average text length of 3.5.

[0350] Snippets (https: / / github.com / jacoxu / STC2 / tree / master) is the structured text processed by Google search results. It contains 12,340 text data in 8 categories, with an average text length of 14.5.

[0351] The Ohsumed (https: / / github.com / yao8839836 / text_gcn) dataset is derived from the MEDLINE database of the U.S. National Library of Medicine. 7,400 articles in 23 categories were extracted as the evaluation dataset, with an average text length of 6.8.

[0352] TagMyNews (https: / / github.com / cskarthik93 / TagMyNews-Classification) aggregates news headline classification tasks from multiple news websites (BBC, Reuters). It contains 32,549 data in seven categories, including economy, technology, and sports, with an average text length of 8.3.

[0353] StackOverflow (https: / / github.com / jacoxu / StackOverflow / tree / master) is a structured statistical data set based on questions and answers in the IT community. It contains 20,000 texts with 20 different types of titles, with an average text length of 5.1.

[0354] The parameter settings are controlled according to the SOTA method of self-supervised learning for text classification. 40 labeled data of each category of text data are used for training and verification of the model (20 labeled data for each category), and all the remaining texts are used for testing. The performance of the model is evaluated by simulating the lack of real labels through training and testing with fewer text labels.

[0355] 4.2 Baseline Model

[0356] We compare the performance of the proposed HEGCL model with the following models to verify the advancedness and effectiveness of our work. The baseline models are introduced as follows:

[0357] Traditional machine learning models: TF-IDF+SVM, an STC scheme that uses TF-IDF for text feature representation and SVM as a classifier; LDA+SVM, a scheme that uses LDA for text feature representation and SVM as a classifier; PTE, which uses known labeled data and unlabeled data to learn and predict text embedding to improve classification performance.

[0358] Deep learning-based models: CNN, a CNN model first applied to text classification using one-dimensional convolutions; LSTM, an LSTM with long-range dependencies, for modeling text classification tasks. Both models use GloVe_300d for word embedding. BERT, a fine-tuned pre-trained language model using a bidirectional Transformer ensemble; and RoBERTa, a robustly optimized BERT variant based on BERT. This paper uses the basic architecture of both models for pre-trained embeddings and fine-tunes the classifier for the STC task.

[0359] Graph-based classification models: TLGNN, a text-level GNN text classification model with global parameter sharing; HyperGAT, which introduces a hypergraph attention network structure on the GNN feature processing paradigm to capture high-order interactions between words; TextING, which captures key contextual features that cannot be obtained in text graphs through inductive analysis of text; DADGNN, which uses a deep attention expansion scheme to overcome the over-smoothing problem that occurs in GNNs during multi-layer graph processing; TextGCN, which is the first to use word-level co-occurrence and document relations to obtain text graph embeddings, forming a two-layer GCN text classification method.

[0360] Deep network models for STC tasks: STCKA, which introduces external knowledge sources to enhance the semantic information of short texts, thereby constructing a deep model of knowledge-driven enhanced attention; STGCN, which is the first to use GCN to classify short texts on special text graphs; HGAT, which uses a heterogeneous GNN semi-supervised learning method to overcome the problems of sparsity and insufficient labeled data in short text classification; SHINE, which enhances the interaction between nodes of the same type by embedding word-level hierarchical heterogeneous graphs of the corpus, and achieves a significant performance improvement in STC tasks with less labeled training data; NC-HGAT, which combines heterogeneous graph attention networks with graph contrastive learning for the first time, providing a new research path for STC tasks; GIFT, which achieves a significant improvement in STC task performance by constructing an improved graph contrastive learning that integrates multiple components of Kmeans clustering, classification, and contrastive learning.

[0361] 4.3 Evaluation indicators

[0362] Previous research has typically used two typical evaluation metrics: accuracy (ACC) and macro F1-scale (F1). ACC provides the most intuitive reflection of the predictive effect of classification performance and provides a direct quantitative evaluation standard for our method. However, there is an imbalance in the number of category samples in different datasets, and the effective evaluation of the F1 metric can overcome this problem and fully demonstrate the robust performance of the model. To comprehensively evaluate the text classification performance of the HEGCL model, this patent will use the above two metrics for experiments and comparisons.

[0363] 4.4 Training Execution Details

[0364] In the STC task, the HEGCL model uses GloVe to embed word vectors and represent text heterogeneous graphs for the text of the dataset. Then, two layers of GCN and the MD matrix of the original semantic graph are used to encode different heterogeneous graphs to generate H and M feature graphs. The sample pair H is generated by controlling the feature graph enhancement to compare the results. o,λ, the feature enhancement intensity control parameter is λ. SVD maximum rank decomposition is used to generate the enhanced TD matrix M of the heterogeneous graph o,r , the maximum rank decomposition number is r. The temperature parameter in contrastive learning is set to τ, and the control parameters of the loss function are set to η and The search range settings for the above five control hyperparameters are shown in Table 2.

[0365] Table 2 Hyperparameter settings

[0366]

[0367] Based on the parameter selection range in Table 2, we will use a grid search method for HEGCL to search for the optimal hyperparameter settings to obtain the best model performance. HEGCL is trained using the Adam optimizer, and training is terminated when the total loss does not decrease after 10 epochs. Other parameter settings are: 100 epochs, 128 intermediate hidden layer dimensions, 1.0e-3 learning rate, and 0.9 dropout. Other baseline models use default settings or grid search methods to obtain the best performance for the STC task. We run each method independently 10 times and take the average as the experimental result. The experimental server platform is: Intel (R) Xeon (R) Gold 6226R CPU @ 3.90GHz (32 cores and 64 threads), 128-GB memory, and two NVIDIA RTX A6000 GPUs.

[0368] In the performance experiment and result analysis of the model, we will conduct the following experiments: 1) Experiment with HEGCL through grid search for the best parameter setting scheme, and compare the results with other classic models and SOTA methods to verify the advanced text mining advantages of HEGCL; 2) Explore the sensitivity of different hyperparameters to model performance and search for the best combined hyperparameter setting scheme; 3) Use different training scales to compare STC performance to verify the rationality and efficiency of our scheme; 4) Perform t-SNE dimensionality reduction clustering visualization comparison on the features of the training output to show the advancedness and robustness of HEGCL; 5) Use ablation experiments to verify the contribution of each improved scheme of the introduced model to the model.

[0369] 4.5 Experimental Results and Analysis

[0370] 4.5.1 Comparative test

[0371] To verify the efficiency and advancement of the HEGCL model proposed in this patent for STC tasks, we compared three machine learning methods, four deep learning methods, five graph-based methods, and six mainstream deep network state-of-the-art methods. See Section 4.2 for details. Table 3 shows the ACC and F1 of various STC techniques on six real-world datasets.

[0372] Table 3 ACC and F1 score results of 6 datasets

[0373]

[0374] As shown in Table 3, HEGCL surpasses the search baseline methods on five datasets and achieves the highest F1 score on the Ohsumed dataset. This demonstrates that our proposed solution has great superiority in the STC task where text information is sparse. Specifically:

[0375] On the MR dataset, four different types of baseline models mostly achieved scores below 60% for both metrics. In recent years, mainstream deep networks like HGAT, SHINE, NC-HGAT, and GIFT have improved these to 60%-65%. GIFT currently holds the highest score for self-supervised learning on the STC task, achieving 65.12% and 65.16%. HEGCL achieved improvements of 1.73% and 1.77% in ACC and F1, respectively. Furthermore, on the Snippets and StackOverflow datasets, HEGCL also achieved a consistent performance improvement of over 1.5%.

[0376] On the Twitter dataset, most machine learning models, deep learning networks, and graph neural networks scored below 60%, but LSTM and TextGCN achieved 60%. Capturing the semantic order of short texts through LSTM and mining semantic information through TextGCN are quite effective for the STC task. Mainstream STGCN, HGAT, and NC-HGAT all maintained scores between 60% and 65%. SHINE and GIFT, the previous top-performing methods for STC, achieved scores exceeding 70%. Our solution achieved improvements of 2.8% and 3.1% in ACC and F1 over SHINE, and 2.18% and 2.13% over GIFT.

[0377] On the TagMyNews dataset, the larger scale of words and entities in the constructed graph compared to other datasets allows for a stronger ability to capture semantic information in short texts. This results in improvements of 3.4% and 8.79% in ACC and F1, respectively, compared to the state-of-the-art method, GIFT. On the Ohsumed dataset, all other baselines differed by 10%-20% in ACC and F1, while Ohsumed exhibited no class imbalance. This suggests that other baseline methods were insufficiently mining information for certain categories on this dataset. In contrast, the HEGCL model achieved comparable ACC and F1 performance, demonstrating that our solution maintains consistent mining capabilities across different text categories.

[0378] In summary, through performance comparisons across six datasets, we found that traditional methods perform poorly on STC tasks. While the current state-of-the-art STC approach, based on self-supervised deep networks, has achieved significant performance improvements on STC tasks, it still suffers from performance imbalances and other drawbacks that lead to poor classification robustness. HEGCL, proposed in this patent, achieves steady performance improvements while delivering optimal performance and robustness for STC tasks, regardless of the short text distribution and data size.

[0379] 4.5.2τ, η, and Hyperparameter selection and analysis

[0380] In this section, we train all datasets using the small amount of ratio labeled data in Table 1 and the default settings of other parameters in Table 2. We analyze and compare the learning hyperparameters τ and loss functions η and The experimental results are as follows: Figure 2 As shown. For the value of τ, [0.1, 0.3, 0.5, 0.7, 0.9] are used to search and set the optimal contrast learning enhancement rate. For η and The values ​​set are the same and searched in [0.05, 0.15, 0.25, 0.35, 0.45].

[0381] like Figure 2 As shown in (a), the parameter τ of HEGCL is analyzed as follows: when τ = 0.7, the best ACC and F1 indicators of the model on the MR, Twitter and TagMyNews datasets are 67.37%, 75.98%, 66.85% and 67.37%, 75.92%, 65.89% respectively; when τ = 0.3, the best performance achieved by the model on the Ohsumed and StackOverflow datasets are 35.57%, 84.24% and 34.63% and 83.92% respectively; and the best performance on the Snippets dataset is 84.69% and 83.56% when τ = 0.1.

[0382] As shown in the distribution of text datasets in Table 1, when the average length of short texts is long (Snippets), contrastive learning achieves the best classification performance with a temperature parameter of 0.1, the smallest possible value. For datasets with a large number of categories, such as Ohsumed and StackOverflow, a temperature parameter of 0.3 offers a slightly better classification advantage. On the other hand, datasets with a larger training scale and fewer categories require a temperature parameter of 0.7 for optimal classification results.

[0383] For the parameter setting of the loss function, the loss weights of NDCL and CCL are η and The loss weight of GECL is like Figure 2 As shown in (b), HEGCL achieves the best classification performance on the Snippets, Ohsumed, and TagMyNews datasets with a loss weight of 0.05. On the MR, Twitter, and StackOverflow datasets, the highest scores are achieved with loss weights of 0.25, 0.35, and 0.45, respectively. In the comparison and analysis of other experimental results in this paper, we use the best performance obtained through grid search as the parameter setting.

[0384] 4.5.3 r and λ hyperparameter selection and analysis

[0385] When conducting experiments and analysis on r and λ, other settings are consistent with Section 4.5.2. The best approximate selection of rank r and graph feature enhancement control parameter λ for SVD decomposition of MD matrix are respectively selected in the range of [5, 10, 15, 20, 25] and [0.1, 0.3, 0.5, 0.7, 0.9] for network search. The experimental results of each data set are compared as follows: Figure 3 shown.

[0386] like Figure 3 As shown in the figure, different curves represent the experimental results of different data sets under corresponding hyperparameter settings, and the red mark shows the best classification effect found. Figure 3 In (a), due to differences in data distribution and features, the highest classification performance is achieved when r is 10 for MR, 15 for Twitter and TagMyNews, 20 for StackOverflow, and 25 for Snippets and Ohsumed. This indicates that the model needs to be adaptive to the maximum rank decomposition of the MD matrix when processing different datasets to achieve better classification results.

[0387] exist Figure 3In (b), it is clear that when the graph feature enhancement control parameter is 0.1 or 0.3, the scores for all datasets are low, which is significantly different from when λ is larger. Therefore, when performing GECL, a parameter between [0.5 and 0.9] is more effective. For the text datasets used in the experiments, setting the parameter to 0.5 for Twitter and TagMyNews, 0.7 for Snippets and Ohsumed, and 0.9 for MR and StackOverflow, respectively, provides the strongest semantic mining capabilities for the STC task. This demonstrates that the GECL module we introduced has a positive impact on STC.

[0388] 4.5.4 Training Scale Analysis

[0389] In this section, we will analyze the impact of different training ratios on model performance. The first data is the training ratio shown in Table 1 (40 labeled data are selected for each category as training and validation). After that, 10%, 20%, 30%, and 40% of the training ratio are randomly selected (i.e., half as training set and half as validation set). Five models, including TextING, TextGCN, HGAT, SHINE, and GIFT, are used to compare with HEGCL. Since F1 is more comprehensive in model performance evaluation than ACC, we will statistically analyze the experimental results of F1 score. Figure 4 As shown in the figure, (a) is the MR comparison analysis chart, (b) is the Twitter comparison analysis chart, (c) is the Snippets comparison analysis chart, (d) is the Ohsumed comparison analysis chart, (e) is the TagMyNews comparison analysis chart, and (f) is the StackOverflow comparison analysis chart.

[0390] like Figure 4 The F1 score of the model corresponding to the first coordinate of each subgraph shows that when very little labeled data is used for training, the HEGCL model is significantly better than other comparison models. When the training data ratio increases from the default value to 10%, the F1 index of all models will be greatly improved. As the ratio continues to increase, the performance continues to improve slowly. This is consistent with the experimental results of supervised learning. However, Figure 4 In (f), the performance improvement of each model by increasing the scale is not obvious. This is due to the large gap in F1 scores between the compared methods, making it difficult to clearly observe the performance increase when integrated into a single figure for comparison. Overall, the comparison of model training results at different scales further validates the advanced performance of the HEGCL method, which outperforms state-of-the-art methods such as SHINE and GIFT, which use mainstream deep network architectures, on the STC task.

[0391] 4.5.5 Comparative Analysis of HEGCL's Advancedness

[0392] We use t-SNE to visually compare the best performing GIFT and HEGCL. The clustering diagrams after t-SNE dimensionality reduction of the features output by the two models for each dataset are as follows: Figure 5 As shown in the figure; (a) is the clustering graph of GIFT on MR, (b) is the clustering graph of GIFT on Twitter, (c) is the clustering graph of GIFT on Snippets, (d) is the clustering graph of HEGCL on MR, (e) is the clustering graph of HEGCL on Twitter, (f) is the clustering graph of HEGCL on Snippets, (g) is the clustering graph of GIFT onOhsumed, (h) is the clustering graph of GIFT on TagMyNews, (i) is the clustering graph of GIFT on StackOverflow, (j) is the clustering graph of HEGCL on Ohsumed, (k) is the clustering graph of HEGCL on TagMyNews, and (l) is the clustering graph of HEGCL onStackOverflow.

[0393] like Figure 5 As shown in , each color in the subgraph represents the label predicted by the corresponding model. The better the aggregation effect between the same categories, the better the classification performance. Figure 5 Comparison between (a), (b) and (d), (e) shows that on the MR and Twitter datasets, GIFT and HEGCL have basically the same aggregation effect on categories. Figure 5 In the comparison between (c), (i) and (f), (l), it can be clearly seen that our method clusters more concentratedly, and clusters of different categories are concentrated in the same area. On the TagMyNews dataset with high semantic information similarity, Figure 5 The category prediction effect ratio of (k) Figure 5 Middle (h) is better.

[0394] However, when the data size is small and the number of categories is large, the lack of words and entities in graph contrastive learning results in poor classification prediction performance for both methods. Compared with the state-of-the-art methods, our method performs better in complex contexts and is at the same level in other cases, which verifies the advanced nature and stability of HEGCL.

[0395] 4.5.6 Ablation Experiment

[0396] In order to analyze the contribution of GECL, NDCL, CCL and Kmeans clustering pseudo-label generation to guide training for HEGCL, we conducted ablation experiments on the model under the best setting. The experimental results are shown in Table 4.

[0397] Table 4 Ablation experiment

[0398]

[0399] As shown in the experimental results in Table 4, it can be seen that the introduction of Kmeans clustering to generate pseudo-label strategies can effectively guide the training process of HEGCL, so that the model classification can be optimized towards the global optimal loss under the condition of insufficient labeled data. Among the other three modules, NDCL and CCL have the same contribution to the model. When the GECL module is removed, the classification score of the model on all datasets decreases by 1%-2%, which is a higher decrease than NDCL and CCL. The experimental results show that the HEGCL model that uses Kmeans clustering for training guidance and introduces fine-tuned NDCL and CCL and GECL for heterogeneous dual-view contrast learning has a positive impact on the performance improvement and robustness of the STC task.

[0400] 5 Conclusion

[0401] This patent addresses the challenges currently faced by short text classification in terms of semantic sparsity and model robustness, and proposes a new heterogeneous multi-view efficient STC method (HEGCL). Specifically, the multiple information sources of the original text are used to construct a heterogeneous graph, and two feature graphs are constructed using GCNs and MD matrices. Then, the graph enhancement operation and the approximate selection r operation after SVD decomposition are introduced to extract enhanced views of the two feature graphs to form a heterogeneous multi-information source text view. Subsequently, the short text information of the multiple views is used for graph comparative learning in modules such as GECL, NDCL, and CCL for Kmeans clustering, so as to mine multi-source short text information to expand its semantics and enhance the classification prediction ability of the model. Finally, a large number of experiments were carried out on 6 short text datasets. The experimental results show that compared with the current mainstream deep network methods, the HEGCL model has achieved great improvements in the semantic mining ability and model robustness of short texts.

[0402] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. An efficient short text classification method based on heterogeneous multi-view graph contrast learning, characterized by: The following steps are involved: S1, constructing a multi-source text graph; S2, learning heterogeneous text graph representations; S3, comparative learning of heterogeneous multi-views; S4, output text classification results.

2. The efficient graph contrast learning short text classification method based on heterogeneous multi-views according to claim 1 is characterized in that Step S1 includes: Among them, A w It is an adjacency matrix based on word co-occurrence statistics in a corpus generated within a text dataset; [A w ] ij Represents the connection weight between different words i and j; max(,) means taking the larger value; PMI(,) represents the PMI value between word i and word j in point mutual information calculation; Represents the real value of word i in the adjacency matrix; Represents the real value of word j in the adjacency matrix; Among them, PMI(,) represents the PMI value between word i and word j in point mutual information calculation; Indicates the number of words and their corresponding sliding windows; represents the count of word i sliding window; represents the count of word j sliding window; Among them, A e represents the entity adjacency matrix; [A e ] ij Represents the adjacency value between entity i and entity j; max(,) means taking the larger value; cos(,) represents cosine similarity; represents entity i; Represents entity j.

3. The efficient graph contrast learning short text classification method based on heterogeneous multi-views according to claim 1 is characterized in that: Step S2 includes: in, Indicates the Embedded features in layer GCN; σ() represents the activation function; express The degree matrix represents the self-loop adjacency matrix; Represents the embedded features in the l-th layer GCN; represents the trainable hyperparameter matrix; Among them, M e (i,j) represents the relationship between text i and entity j; Indicates that if text i is contained in entity j; M e (i,j)=1 means that text i is contained in entity j; M e (i,j)=0 means that text i is not contained in entity j; Z o =M o H o ,o∈{w,e,m} (6) Among them, Z o Represents high-dimensional aggregate features for extracting multi-source information from short texts; M o Represents the constructed text MD matrix; H o Represents the original semantic embedding features; WITH org =Z w ||From e ||From m (7) Among them, Z org represents the first contrastive view used for contrastive learning; Z w Indicates word contrast attempt; Z e Indicates entity comparison attempt; Z m Represents the POS tag comparison view; || represents feature concatenation operation.

4. The efficient graph contrast learning short text classification method based on heterogeneous multi-views according to claim 1 is characterized in that Step S3 includes the following steps: S31, graph-enhanced contrastive learning; S32, contrastive learning for noise dimensionality reduction; S33, Cluster Contrastive Learning.

5. The efficient graph contrast learning short text classification method based on heterogeneous multi-views according to claim 1 is characterized in that Step S31 includes: Among them, H o,λ represents the enhanced feature map; λ represents the intensity of the introduced random noise; H o Represents the original semantic embedding features; |||| means arbitrary modulus; Z o,λ =M o H o,λ ,o∈{w,e,m} (9) Z o,λ Represents the embedded features after graph augmentation operation; M o Represents the constructed text MD matrix; H o,λ represents the enhanced feature map; WITH emg =Z w,λ ||From e,λ ||From m,λ (10) Among them, Z emg Indicates enhanced contrast view; Z w,λ Representation word enhanced contrast view; Z e,λ Represents an entity enhanced contrast view; Z m,λ Indicates POS tag enhancement comparison attempt; || represents feature concatenation operation; represents contrast loss; exp() represents the exponential function with the natural base e as the base; P i Represents a positive sample pair (P i ,P j ) in the left sample; P j Represents a positive sample pair (P i ,P j ) in the right sample; P k Represents a positive sample pair (P i ,P j ) in the sample; τ represents the temperature parameter of contrastive learning; N represents the total number of short texts; represents the indicator function; in, represents the total loss of the GECL module; N represents the total number of short texts; represents contrast loss.

6. The efficient graph contrast learning short text classification method based on heterogeneous multi-views according to claim 1 is characterized in that Step S32 includes: Among them, Δ rmax represents the reversible diagonal matrix after full rank decomposition; Σ p×q is a singular real-valued matrix; M o,r =U o,r Σ o,r V o,r Τ ,o∈{w,e,m} (14) Among them, M o,r Represents the characteristic matrix after MD matrix noise decomposition; Σ o,r Represents the first r feature matrices after singular value decomposition; V o,r represents a right singular column vector; Indicates V o,r The transpose of Z o,r =M o,r H o ,o∈{w,e,m} (15) Among them, Z o,r Represents the text representation vector after noise decomposition; M o,r Represents the characteristic matrix after MD matrix noise decomposition; H o Representation represents the original semantic embedding features; WITH aug =Z w,r ||From e,r ||From m,r (16) Among them, Z aug represents the noise decomposition contrast view; Z w,r represents the noise decomposed word contrast view; Z e,r Represents a noise decomposition entity comparison view; Z m,r Represents the noise decomposition POS tag comparison view; || represents feature concatenation operation; in, represents the contrast loss function of NDCL; N represents the total number of short texts; represents contrast loss; γ represents the regularization coefficient; W nd Represents the trainable hyperparameter matrix of the NDCL module; represents the square of L2 regularization.

7. The efficient graph contrast learning short text classification method based on heterogeneous multi-views according to claim 1 is characterized in that Also includes: in, represents the optimization objective function of CCL; N represents the total number of short texts; S i represents the set with the same predicted label as text i and does not contain text i; |S i | represents the set S i size; Q i Represents a positive sample pair (Q i ,Q j ) in the left sample; Q j Represents a positive sample pair (Q i ,Q j ) in the left sample; Q k Represents a positive sample pair (Q i ,Q j ) in the sample; τ represents the temperature parameter of contrastive learning; χ represents the adaptive sliding loss weight; Represents the indicator function.

8. The efficient graph contrast learning short text classification method based on heterogeneous multi-views according to claim 1 is characterized in that: Also includes: in, represents the cross entropy loss; T lab Represents a collection of label texts; c weak represents a weak label set; y ij represents the pseudo label of text j; σ() represents the activation function; W cls represents the trainable hyperparameter matrix for the cross entropy loss function; Υ() represents the feature mapping function; Z org represents the original text embedding vector; in, Represents the total loss function of the HEGCL model; represents the cross entropy loss; represents the optimization objective function of CCL; represents the contrast loss function of NDCL; represents the total loss of the GECL module; η and Indicates control parameters.

9. A computer system, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the efficient graph contrast learning short text classification method based on heterogeneous multi-views as described in any one of claims 1 to 8 when executing the executable instructions.

10. A computer-readable storage medium, characterized in that include: a memory having a computer program stored thereon; A processor is used to execute the program in the memory to implement the efficient short text classification method based on heterogeneous multi-view comparative learning according to any one of claims 1 to 8.

Images