Short text classification method based on uncertainty perception heterogeneous graph attention network

Abstract

This invention discloses a short text classification method based on an uncertainty-aware heterogeneous graph attention network. The method includes: acquiring a labeled dataset and an unlabeled dataset, wherein the number of labeled data is less than the number of unlabeled data; training a heterogeneous graph attention network model based on the labeled dataset; using the trained heterogeneous graph attention network model to predict the unlabeled dataset and calculating the uncertainty value of the prediction result; using both the predicted value and the uncertainty value as confidence levels to assign pseudo-labels to the unlabeled dataset and selecting positive and negative pseudo-labels; mixing the selected pseudo-labeled data and the labeled dataset to obtain a mixed dataset, and further training the heterogeneous graph attention network model using this mixed dataset; repeating the above steps until a set termination condition is met. This invention can accurately classify common short texts even with limited data sources.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically, to a short text classification method based on an uncertainty-aware heterogeneous graph attention network. Background Technology

[0002] In recent years, deep learning-based natural language processing (NLP) technologies have developed rapidly and are widely used in fields such as text classification, word vector representation, semantic similarity, opinion extraction, and sentiment analysis. With the rapid development of online social media and e-commerce, short text information such as online news, comments, and tweets is becoming increasingly common on the internet. Deep learning-based methods have already achieved great success in extracting useful information from low-information-content data.

[0003] The immense success of deep learning methods is largely attributed to advancements in learning algorithms and the availability of large-scale labeled datasets. However, in some real-world scenarios, labeled data is scarce, and manual labeling is extremely time-consuming. Semi-supervised learning is one of the most prominent approaches to address this problem, aiming to leverage large unlabeled datasets and small labeled datasets. Semi-supervised learning classification tasks primarily employ two core methods: consistency regularization and pseudo-labeling. Consistency regularization-based methods aim to position the decision boundary in low-density regions by making the network output unaffected by small input perturbations. However, this method often relies on a wide range of data augmentation techniques, such as affine transformations, cropping, and color jittering in images, which limits its effectiveness in some domains where augmentation is less effective. Pseudo-labeling selects unlabeled samples with high-confidence predictions to move the decision boundary to low-density regions, but due to poor neural network calibration, many of these selected predictions are incorrect, and for poorly calibrated networks, erroneous predictions can have high confidence.

[0004] In summary, making full use of limited labeled data and a large amount of unlabeled data is a key issue in short text classification. Furthermore, effectively capturing the importance of different information, integrating information at multiple granular levels to address sparsity issues, and reducing the weight of noisy information to obtain more accurate classification results remains a challenge. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a short text classification method based on an uncertainty-aware heterogeneous graph attention network. This method includes the following steps:

[0006] Step S1: Obtain labeled datasets and unlabeled datasets, wherein the number of labeled data is less than the number of unlabeled data. The labeled datasets reflect the correspondence between short text data and label types.

[0007] Step S2: Train a heterogeneous graph attention network model based on the labeled dataset;

[0008] Step S3: Use the trained heterogeneous graph attention network model to predict the unlabeled dataset and calculate the uncertainty value of the prediction result;

[0009] Step S4: Using the predicted value and the uncertainty value as confidence levels, pseudo-labels are assigned to the unlabeled dataset, and positive and negative sample pseudo-labels are selected.

[0010] Step S5: Mix the selected pseudo-labeled data and the labeled dataset to obtain a mixed dataset, and use the mixed dataset to further train the heterogeneous graph attention network model;

[0011] Step S6: Repeat steps S3 to S5 until the set termination condition is met.

[0012] Compared with existing technologies, the advantages of this invention are that, taking into account the sparsity of short texts, it integrates topic information from heterogeneous information network structures and entity information from knowledge graphs, deeply mines the internal structural information between texts, dynamically integrates the contextual information features of massive amounts of text, and makes full use of unlabeled data to calibrate the model, thus significantly improving the accuracy and generalization ability of the model classification.

[0013] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0015] Figure 1 This is a schematic diagram of a short text heterogeneous information network structure according to an embodiment of the present invention;

[0016] Figure 2 This is a structural diagram of a short text classification model based on an uncertainty-aware heterogeneous graph attention network according to an embodiment of the invention;

[0017] Figure 3 This is a schematic diagram of a heterogeneous graph attention network structure according to an embodiment of the present invention;

[0018] Figure 4 This is a flowchart of an uncertainty-aware pseudo-label selector according to an embodiment of the present invention. Detailed Implementation

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0021] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] This invention addresses the semantic sparsity, ambiguity, and lack of context characteristics of short texts by designing a heterogeneous information network structure and further constructing a short text classification model based on an uncertainty-aware heterogeneous graph attention network. In the following description, the uncertainty-aware heterogeneous graph attention network is also referred to as a heterogeneous graph attention network or a heterogeneous graph attention network model.

[0025] Figure 1 This is the constructed heterogeneous information network structure for short texts. The left side illustrates the topic, the middle section represents the short text content, and the right side represents the knowledge graph. This heterogeneous information network structure integrates topic information, short text information, and entity information from the knowledge graph, enabling it to capture the rich relationships between text and supplementary information, thus alleviating the sparsity problem of short texts. Specifically, the heterogeneous information network structure includes three types of nodes and three types of relationships. Node types include topic information nodes, short text information nodes, and entity information nodes; relationship types include relationships between entities, relationships between text and entities, and relationships between text and topics.

[0026] In one embodiment, the topic information node representation uses LDA (Latent Dirichlet Distribution) to mine potential topics and represents them with the probability distribution of words. Then, each short text is assigned to, for example, the top two topics with the highest probability, and edges are constructed between topics and texts.

[0027] For short text information nodes, the sentence vector representation of the short text can use pre-trained language models, including but not limited to Word2Vec, GloVe, fastText, and BERT. In one embodiment, a BERT-based short text vector embedding representation is used, specifically employing the pre-packaged library bert-as-service, where the maximum sentence length max_seq_len is set to 256. The BERT-based short text vector embedding dynamically fuses the contextual information features of massive amounts of text, providing high-quality feature vector input representations for short texts.

[0028] Entity information nodes represent entities that are associated with the knowledge graph from the text, and add the neighboring nodes of the associated entities to the heterogeneous information network structure. For example, the added neighboring nodes are nodes that are one hop away from the related entity nodes. At the same time, edges between entities and text, and edges between entities are constructed.

[0029] Furthermore, a short text classification model based on an uncertainty-aware heterogeneous graph attention network is constructed based on the designed heterogeneous information network structure. Figure 2 This is a schematic diagram of a short text classification model based on an uncertainty-aware heterogeneous graph attention network. It includes a heterogeneous graph attention network, a two-layer attention mechanism, and a pseudo-label selector.

[0030] Heterogeneous graph attention networks fully leverage the advantages of knowledge graphs and abundant unlabeled data to enable information propagation along the graph and effectively capture the relationships between adjacent nodes to solve the semantic sparsity problem of short texts. Among them, graph neural networks are used to mine the internal structural information between texts.

[0031] A two-layer attention mechanism is used to extract important feature information from neighboring nodes and assign different weights to different types of nodes to reduce the weight of noisy information. Figure 3 As shown, α1, α2, and α3 represent the importance of different types of nodes to node i (i.e., type-level attention), while β1 and β2 represent the importance of node N when the node types are the same. i 1 N i 2 The importance of node i (i.e., node-level attention), where N i 1 and N i 2 They are nodes of the same type, N i 4 and N i 5 They are nodes of the same type, N i 3 and N i 6They are nodes of the same type.

[0032] The pseudo-label selector is used to select positive and negative pseudo-labels for samples based on computational uncertainty. Figure 4 This is a flowchart of an uncertainty-aware pseudo-label selection framework. This framework creates positive and negative pseudo-labels from a large amount of unlabeled data, and uses uncertainty awareness to reduce noise and improve the accuracy of the pseudo-labels. This results in more training data being generated, and model performance is improved through continuous iteration.

[0033] The following section focuses only on the short text classification model based on an uncertainty-aware heterogeneous graph attention network designed in this invention; the construction of the heterogeneous information network structure will not be elaborated upon further. See also Figure 4 As shown, the short text classification method based on uncertainty-aware heterogeneous graph attention network includes the following steps:

[0034] Step S110: Construct a heterogeneous information network using the acquired short text data.

[0035] For example, 6000 labeled data points are randomly selected from the publicly available news dataset AGNews and evenly distributed into four categories. For each category, 100 labeled short texts are randomly selected, with half used as the training set, half as the validation set, and the remainder as the test set and unlabeled data. The labeled data can be labeled as D. L Unlabeled data is marked as D U .

[0036] Step S120: Use a heterogeneous graph attention network to perform the first stage of training on the labeled data.

[0037] During training, the dropout parameter (dropout rate) can be set to 0.5.

[0038] Step S130: Use the model trained in the first stage to generate pseudo-labels for the unlabeled data.

[0039] This step involves using a trained model to predict the category of unlabeled data. For example, during the model prediction phase, dropout is used multiple times to predict the same input (i.e., the same unlabeled data), while these output values ​​are averaged and their variances calculated in parallel to determine the uncertainty value for each unlabeled data point. See also... Figure 1 As shown, μ1, μ2, and μ3 represent the mean, and σ1, σ2, and σ3 represent the variance.

[0040] Step S140: Use the output predicted value and uncertainty value as confidence levels simultaneously, and assign positive or negative pseudo-labels to values ​​that satisfy both confidence levels.

[0041] Specifically, when both the prediction probability and the model output prediction value are stable, positive and negative labels are assigned. For example, when the predicted mean of a sample is greater than 0.7 and the predicted variance of the sample is less than 0.05, it can be selected as a positive sample. When the predicted mean of a sample is less than 0.1 and the predicted variance of the sample is less than 0.05, it can be selected as a negative sample.

[0042] For example, let P represent the set of probabilities that a sample belongs to class A after ten consecutive predictions by the model, then:

[0043] The condition for selecting it as a positive sample is:

[0044] μ(P)>0.7 &&σ(P)>0.05

[0045] The condition for selecting it as a negative sample is:

[0046] μ(P) < 0.1 && σ(P) > 0.05

[0047] The selected positive and negative sample pseudo-label data are labeled Dselected. Figure 2 The mean threshold of the positive samples is labeled T. μ The variance threshold is denoted as T. σ The mean threshold for negative samples is denoted as F. μ The variance threshold is denoted as F. σ It should be understood that T μ and F μ T can be set to the same or different values. σ and F σ It can be set to the same value or different values.

[0048] Step S150: Mix the selected pseudo-labeled data and labeled dataset to obtain a mixed dataset and use it as the training set for the model to perform the second stage of training.

[0049] Figure 4 In this context, the training set of the model is represented as D = D L ∪D selected .

[0050] Step S160, repeat steps S130-S150 until the set termination conditions are met.

[0051] For example, training terminates when the number of iterations reaches a given maximum number of iterations (e.g., MaxIteration = 50) or when the model's loss value meets a set condition.

[0052] It should be noted that, without departing from the spirit and scope of this invention, those skilled in the art can make appropriate changes or modifications to the above embodiments. For example, the threshold for assigning pseudo-labels to positive and negative samples in the pseudo-label selector, the number of topics selected in the heterogeneous information network structure, and the order of selecting neighbors of relevant entities in the knowledge graph can be set according to actual needs or device limitations. This invention does not limit the specific structure.

[0053] In summary, this invention utilizes a heterogeneous information network structure to model short texts, topic information, and entity knowledge from knowledge graphs. It employs BERT for sentence vector embedding representation of short texts and inputs the fused network structure into an uncertainty-aware heterogeneous graph attention network model to capture the relationships between them for short text classification. The uncertainty-aware heterogeneous graph attention network comprises three parts: a heterogeneous graph attention network, a two-layer attention mechanism, and a pseudo-label selector. The heterogeneous graph attention network fully leverages the advantages of knowledge graphs and abundant unlabeled data to achieve information propagation along the graph, effectively capturing the relationships between adjacent nodes to address the semantic sparsity problem of short texts. The graph neural network is used to mine the internal structural information between texts; the two-layer attention mechanism extracts important feature information from neighboring nodes and assigns different weights to different types of nodes to reduce the weight of noise information; the knowledge graph and topics provide additional information for the short text; and the BERT-based short text vector embedding dynamically fuses the contextual information features of massive amounts of text to provide high-quality feature vector input representations for the short text. The pseudo-label selector creates positive and negative pseudo-labels on a large amount of unlabeled data and uses uncertainty awareness to reduce noise and improve the accuracy of the pseudo-labels. This is used to generate more training data and improves model performance through continuous iteration. This invention can fully utilize unlabeled data to calibrate the model even with low data sources, significantly improving the model's classification accuracy and generalization ability. Experiments have verified that the heterogeneous graph attention network, after adding the uncertainty-aware pseudo-label selector, significantly improves the accuracy of short text classification and is applicable to various common types of short text.

[0054] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0055] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0056] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0057] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0058] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0059] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0060] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0061] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0062] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A short text classification method based on uncertainty-aware heterogeneous graph attention network, comprising the following steps: Step S1, obtaining a labeled data set and an unlabeled data set, wherein the number of labeled data is less than that of unlabeled data, and the labeled data set reflects the corresponding relationship between short text data and label types; Step S2, training a heterogeneous graph attention network model based on the labeled data set; Step S3, predicting the unlabeled data set using the trained heterogeneous graph attention network model, and calculating the uncertainty value of the prediction result; Step S4, using the prediction value and the uncertainty value as the confidence at the same time, pseudo-labeling the unlabeled data set, and selecting positive and negative sample pseudo-labels; Step S5, mixing the selected pseudo-label data and the labeled data set to obtain a mixed data set, and further training and optimizing the heterogeneous graph attention network model using the mixed data set; Step S6, repeating steps S3 to S5 until the set termination condition is met; Wherein, the heterogeneous graph attention network model includes a heterogeneous graph attention network, a double-layer attention mechanism module and a pseudo-label selector, the heterogeneous graph attention network uses knowledge graph and unlabeled data to realize information propagation along the graph to capture the relationship between adjacent nodes; the double-layer attention mechanism module is used to extract important feature information of neighbor nodes and give different weights to different types of nodes; the pseudo-label selector uses the prediction value and the uncertainty value to select high-accuracy and high-stability positive and negative samples for pseudo-labeling, so as to expand the training set in the training iteration process, and further iteratively train and optimize the heterogeneous graph attention network model; Wherein, the heterogeneous information network structure is used to model short text, topic information and entity knowledge in the knowledge graph, and the heterogeneous graph network node vector representation is obtained as the input of the heterogeneous graph attention network model; Wherein, the heterogeneous graph attention network model takes the heterogeneous information network structure as input, and projects different types of information into an implicit common space to realize a heterogeneous graph convolutional neural network, and introduces a double-layer attention mechanism to extract important feature information of neighbor nodes and give different weights to different types of nodes; Wherein, the heterogeneous information network structure is set to integrate multiple additional information and capture the relationship between short text and additional information; Wherein, the additional information includes topic information, entity information and short text information, wherein the topic information is mined by an implicit Dirichlet distribution clustering method; the entity information is obtained from a domain knowledge graph or a Wikipedia vector representation; and the short text information is vector represented using a pre-trained language model to extract language features from short text data; Wherein, the uncertainty value is obtained by using dropout rate to predict multiple times for the same input in the training process of the heterogeneous graph attention network model, and then averaging and calculating the statistical variance of the multiple prediction values in parallel to obtain the uncertainty of the input.

2. The method of claim 1, wherein, The double-layer attention mechanism module is provided with a type-level attention mechanism and a node-level attention mechanism, the type-level attention mechanism is used for representing the importance of different types of nodes to a specific node, and the node-level attention mechanism is used for representing the importance of nodes of the same type to the specific node.

3. A computer readable storage medium having stored thereon a computer program, wherein, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1-2.

4. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the method according to any one of claims 1-2.

Images