Short text classification method fusing theme features

Through LDA theme model, word vector weighting and the method of constructing multiple feature maps, the problem of label indexing difficulties in short text classification is solved, and the expansion of short text features and classification performance improvement is achieved.

CN118939799APending Publication Date: 2024-11-12SICHUAN POLICE COLLEGE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410757788.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Difficulty in classification of short texts. Due to the lack of context information in text content and high manual annotation costs, traditional methods are difficult to effectively solve the label indexing problem.

Method used

The theme features are obtained using LDA theme model and word vector weighting, and a word graph, part-of-speech label graph, and entity graph are constructed. Short text graphs are embedded and dynamically learned through GCN learning nodes, and the theme features and text features are integrated to expand short text features to achieve more Effective tag propagation.

Benefits of technology

By integrating the theme features and text features, the semantic information of short text is enriched, the effect of classification tasks is improved, the lack of context information is made up for, and the classification performance of short text is promoted.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939799A_ABST
    Figure CN118939799A_ABST
Patent Text Reader

Abstract

The invention discloses a short text classification method fusing theme features in the field of computer natural language processing. According to the method, firstly, topic features are extracted from a short text, and weighted summation is carried out on the topic features obtained by an LDA model and word vectors generated by Word2vec to capture a core topic of the short text. Secondly, modeling the short text data set into a word-level component graph consisting of three feature graphs, namely a word graph, a part-of-speech tag graph and an entity graph, so as to obtain text features of the short text; and finally, fusing the theme features and the text features, and then dynamically learning a short text graph to encode the similarity between the short documents, so that the short documents with similar connection between more effective labels are propagated. According to the method, the text feature information and the theme feature information of the short text are considered at the same time, the semantics of the short text is enriched, and the classification task is greatly promoted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of computer natural language processing, and specifically is a short text classification method integrating subject features. Background Art

[0002] With the increase in Internet users and the rapid development of social networks, a large amount of short text data is generated in life, such as product descriptions, comment information, and microblogs. More and more short texts need to be classified. Maron published the first article on automatic text classification. However, traditional text classification methods are not suitable for short text classification. Therefore, how to use short text classification technology to mine hidden information and apply it to practical tasks such as topic tracking, sentiment analysis, public opinion monitoring, and user personalized recommendations is one of the hot topics in the current research field of natural language processing.

[0003] Compared with long text classification, short text classification is more difficult. One reason is that short texts contain fewer terms, which leads to a lack of sufficient context information in the text content. Another reason is that the cost of manual annotation is too high, so a lot of data lacks annotation information. Therefore, understanding short texts requires auxiliary knowledge, examples of which include concepts that can be found in common sense knowledge graphs, latent topics extracted from short text datasets, and entities in knowledge graphs. However, simply enriching auxiliary knowledge cannot solve the lack of labeled data. Moreover, the commonly used deep models require large-scale labeled data to be well trained.

[0004] So far, graph neural networks have achieved state-of-the-art performance in short text classification. Among them, the best classification effect is the SHINE algorithm. This algorithm models the short text dataset as a hierarchical heterogeneous graph composed of word-level component graphs, which introduce more semantic and syntactic information. Then a short document graph is dynamically learned to promote effective label propagation between similar short texts. However, it does not perform particularly well in solving the problem of sparse short text features and label indexing of short texts. Summary of the invention

[0005] In order to solve the problem of short text tag indexing, the present invention aims to provide a short text classification method integrating topic features.

[0006] In order to achieve the above-mentioned purpose, the technical scheme of the present invention is as follows: a short text classification method integrating topic features, which obtains topic features and text features of short texts, utilizes the weighted sum of LDA topic model and word vector in topic feature acquisition, utilizes the construction of feature graph in text feature acquisition, and finally integrates topic features with text features to achieve the purpose of short text feature expansion, so as to realize the classification of short texts.

[0007] Furthermore, the topic feature acquisition includes the following steps:

[0008] S1, first preprocess the short text to obtain the topic features of the short text;

[0009] S2, constructs a feature graph for the short text at the same time. The constructed feature graph includes word graph, part-of-speech tag graph and entity graph;

[0010] S3, obtained by connecting the representations of these three feature maps with the topic features, records the index of the labeled short text.

[0011] The preprocessing in S1 first performs corresponding word segmentation on the short text i, and then cleans the result of the word segmentation, including redundant punctuation marks and redundant stop words. The cleaned result is then used as the input of the Word2Vec model to obtain the word vector set of text i;

[0012] Secondly, the first m words under the topic with the highest probability in the short text are selected, and the probabilities corresponding to these m words are normalized, and the results are used as the weight values ​​corresponding to these m words;

[0013] Finally, a weighted sum is performed to obtain the short text topic features.

[0014] Furthermore, the word vector set of text i is shown in formula (1):

[0015] V i = {v i (c1),v i (c2),...,v i (c n )} (1)

[0016] Among them, V i is the set of word vectors of all words in short text i, n is the number of words, v i (c1) is represented as the word vector representation of vocabulary c1.

[0017] Furthermore, the topic with the highest probability in the current short text i corpus Select the first m words under the topic with the highest probability The probability corresponding to these m words Probability Perform normalization operation and use the result as the weight value corresponding to the m words. The normalization formula (2) is as follows:

[0018]

[0019] Furthermore, the steps of weighted summation of short text topic features are as follows: express The corresponding value after normalization is k = {1, 2, 3, ..., m}. The corresponding word vector {v i (c1),v i (c2),...,v i (c m )} perform weighted summation to obtain the short text topic features As shown in formula (3):

[0020]

[0021] Furthermore, the node embedding of the feature graph in step S2 includes: Z T = {N T ,M T} represents a word-level component graph of type T, where N T Represents a node, represents the adjacency matrix, Represents the features corresponding to the word-level component graph, node features E T Through two layers of GCN learning, we can get:

[0022]

[0023] Where [ReLu(x)] i =max([x] i ,0), [B T ] ij =∑ j [M T ] ij , and It is a trainable parameter, and the definition of node N in the graph depends on the integrated features.

[0024] Furthermore, the adjacency matrix of the feature graph is constructed by:

[0025] ①PMI

[0026] PMI is used to measure the correlation between two variables. The calculation formula of PMI is as follows:

[0027]

[0028] Where x and y are two variables, the larger the value of PMI, the higher the correlation between the two variables;

[0029] Cosine Similarity:

[0030] Cosine similarity is the cosine value of the angle between two vectors in the vector space. It is used to measure the similarity between two vectors. The calculation formula of cosine similarity is as follows:

[0031]

[0032] in and are two vectors. In trigonometric functions, the function value of the cosine function is [-1,1]. Therefore, the cosine similarity between two vectors ranges from [-1,1]. When the angle between the two vectors is 0°, the similarity is 1, and when the angle is 180°, the similarity is -1.

[0033] Furthermore, the word graph G W = {N W ,M W}, N is the set of words in the dataset, and PMI is calculated between any two words in the entire dataset to construct M;

[0034]

[0035] Then use formula (4) to learn the node representation function E based on the graph W (w is the type of graph), so that the representation of each node can be calculated, that is, the embedding of each word, and the initial representation uses onehot encoding;

[0036] Part-of-speech tag graph G P = {N P ,M P}, the construction and node representation of this graph are similar to the word graph. The only difference is that the words in the original dataset are replaced with part-of-speech tags. e = {N e ,M e}, the node set N comes from the entity types defined in the NELL knowledge base, and the adjacency matrix M is:

[0037]

[0038] x e It is the embedding of entity type, which is learned using the classic knowledge graph embedding method TransE.

[0039] In general, the NELL knowledge base is used to identify the entities contained in each text, and then the TransE method is used to learn the embedding of each entity. Finally, the cosine distance is used to calculate the similarity between two entities to fill M.

[0040] Furthermore, in the feature map G S = {N S ,M S}, where N represents the short text dataset, M represents the adjacency matrix, and hierarchical pooling is used to learn the similarity between short texts M S , and perform label propagation learning;

[0041] We first use the above three feature graphs to represent each node, and then calculate the similarity by the inner product of two nodes;

[0042]

[0043] Among them, the subscript T indicates the type of feature map, E T is the node embedding of the corresponding graph, u is the L1 norm,

[0044] For word graphs and part-of-speech tag graphs, S T represents the TF-IDF value of each word or part-of-speech tag in document i; for an entity graph, if the entity in the graph exists in the current document i, then S T It is 1 if it is not, otherwise it is 0;

[0045] Finally, a text i is obtained by connecting the representations of these three feature maps with the topic features, as shown in formula (10):

[0046]

[0047] Then use the inner product to represent the relationship matrix M:

[0048]

[0049] Among them, δ is the threshold for measuring the similarity between texts;

[0050] Then GCN is used to learn the node embedding of the graph, as shown in formula (12):

[0051]

[0052] in, and is a trainable parameter,

[0053] Its classification loss is shown in formula (13):

[0054]

[0055] in, Record the index of the short text with the label, is a one-hot vector of 0.

[0056] The following beneficial effects are achieved by adopting the above scheme:

[0057] We propose a method to better compensate for the lack of contextual information in short text classification. First, the topic features of short texts are obtained by weighted summing of the LDA model and Word2vec word vectors. Then, the short text dataset is modeled as a word-level component graph consisting of three feature graphs: word graph, part-of-speech tag graph, and entity graph to obtain the text features of short texts. The topic features are fused with the text features, and then a short text graph is dynamically learned to encode the similarity between short documents, thereby enabling more effective label propagation between connections of similar short documents. This method considers both the text feature information and the topic feature information of the short text itself, enriching the semantics of the short text, which greatly facilitates the classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a schematic diagram of an embodiment of the present invention; Figure 2 It is a word-level component graph. DETAILED DESCRIPTION

[0059] The following is further described in detail through specific implementation methods:

[0060] The embodiment is basically as shown in the attached Figure 1 As shown: A short text classification method integrating topic features is specifically implemented as follows:

[0061] 1. Topic feature extraction

[0062] First, the short text i is segmented accordingly, and then the segmentation results are cleaned. Text cleaning mainly removes many useless parts in the short text, such as unnecessary punctuation marks, stop words, etc. Finally, the cleaned results are used as the input of the Word2Vec model to obtain the word vector set of text i as shown in formula (1):

[0063] V i = {v i (c1),v i (c2),...,v i (c n )} (1)

[0064] Among them, V i is the set of word vectors of all words in short text i, n is the number of words, v i (c1) is represented as the word vector representation of vocabulary c1.

[0065] The LDA model can obtain the topic with the highest probability in the current short text i corpus. Select the first m words under the topic with the highest probability The probability corresponding to these m words Probability Perform normalization operation and use the result as the weight value corresponding to the m words. The normalization formula (2) is as follows:

[0066]

[0067] in express The corresponding value after normalization is k = {1, 2, 3, ..., m}. The corresponding word vector {v i (c1),v i (c2),...,v i (c m )} perform weighted summation to obtain the short text topic features As shown in formula (3):

[0068]

[0069] (II) Word-level component diagram

[0070] The word-level component graph consists of word graph, part-of-speech tag graph, and entity graph, such as Figure 2 shown.

[0071] 1. Node Embedding

[0072] Z T = {N T ,M T} represents a word-level component graph of type T, where N T Represents a node, represents the adjacency matrix, Represents the features corresponding to the word-level component graph, node features E T Through two layers of GCN learning, we can get:

[0073]

[0074] Where [ReLu(x)] i =max([x] i ,0), [B T ] ij =∑ j [M T ] ij , and It is a trainable parameter, and the definition of node N in the graph depends on the integrated features.

[0075] 2. Graph Construction

[0076] According to the definition of the graph: G T = {NT ,M T}, the construction of the graph needs to solve two problems:

[0077] (1) How to define node N

[0078] This needs to be defined according to the features you want to incorporate. If you want to incorporate part-of-speech information, V is the part-of-speech tag. If you want to incorporate entity information, V is the entity tag.

[0079] (2) How to construct the adjacency matrix

[0080] ①PMI

[0081] PMI is used to measure the correlation between two variables. The calculation formula of PMI is as follows:

[0082]

[0083] Where x and y are two variables.

[0084] The larger the value of PMI, the higher the correlation between the two variables.

[0085] ② Cosine similarity

[0086] Cosine similarity is the cosine value of the angle between two vectors in the vector space, which is used to measure the similarity between two vectors. The calculation formula of cosine similarity is as follows:

[0087]

[0088] in and are two vectors. In trigonometric functions, the function value of the cosine function is [-1,1]. Therefore, the cosine similarity between two vectors ranges from [-1,1]. When the angle between the two vectors is 0°, the similarity is 1. When the angle is 180°, the similarity is -1. It can be seen that the larger the cosine value, the more similar the two vectors are.

[0089] Next, we will describe in detail how to construct word graphs, part-of-speech tag graphs, and entity graphs.

[0090] (1) Word graph

[0091] We constructed a word graph G W = {N W ,M W}, N is the set of words in the dataset, and PMI is calculated between any two words in the entire dataset to construct M.

[0092]

[0093] Then use formula (4) to learn the node representation function E based on the graph W (w is the type of graph), so that the representation of each node can be calculated, that is, the embedding of each word, and the initial representation uses onehot encoding.

[0094] (2) Part-of-speech tag graph

[0095] We constructed a part-of-speech tag graph G P = {N P ,M P}, the construction and node representation of this graph are similar to word graph. The only difference is that the words in the original dataset are replaced with part-of-speech tags.

[0096] (3) Entity graph

[0097] We construct an entity graph G e = {N e ,M e}, the node set N comes from the entity types defined in the NELL knowledge base. Adjacency matrix M:

[0098]

[0099] x e It is the embedding of entity type, which is learned using the classic knowledge graph embedding method TransE.

[0100] In general, the NELL knowledge base is used to identify the entities contained in each text, and then the TransE method is used to learn the embedding of each entity. Finally, the cosine distance is used to calculate the similarity between two entities to fill M.

[0101] (III) Short text map

[0102] We dynamically learn a short text graph G S = {N S ,M S}, where N represents the short text dataset and M represents the adjacency matrix. Using hierarchical pooling, learn the similarity M between short texts S , and perform label propagation learning.

[0103] We first use the above three feature graphs to represent each node, and then calculate the similarity by the inner product of two nodes.

[0104]

[0105] Among them, the subscript T indicates the type of feature map, E T is the node embedding of the corresponding graph, u is the L1 norm,

[0106] For word graphs and part-of-speech tag graphs, S T represents the TF-IDF value of each word or part-of-speech tag in document i. For an entity graph, if the entity in the graph exists in the current document i, then S T It is 1 if yes, otherwise it is 0.

[0107] Finally, a text i is obtained by connecting the representations of these three feature maps with the topic features, as shown in formula (10):

[0108]

[0109] Then use the inner product to represent the relationship matrix M:

[0110]

[0111] Among them, δ is the threshold for measuring the similarity between texts.

[0112] Then GCN is used to learn the node embedding of the graph, as shown in formula (12):

[0113]

[0114] in, and is a trainable parameter,

[0115] Its classification loss is shown in formula (13):

[0116]

[0117] in, Record the index of the short text with the label, is a one-hot vector of 0.

[0118] The pseudo code of this method is as follows:

[0119]

[0120] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and / or characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A short text classification method integrating topic features, characterized by: The topic features and text features of short texts are obtained. The weighted sum of LDA topic model and word vector is used for topic feature acquisition, and the construction of feature graph is used for text feature acquisition. Finally, the topic features are fused with the text features to achieve the purpose of short text feature expansion and realize the classification of short texts.

2. The short text classification method integrating topic features according to claim 1 is characterized in that: The subject feature acquisition comprises the following steps: S1, first preprocess the short text to obtain the topic features of the short text; S2, constructs a feature graph for the short text at the same time. The constructed feature graph includes word graph, part-of-speech tag graph and entity graph; S3, obtained by connecting the representations of these three feature maps with the topic features, records the index of the labeled short text.

3. The short text classification method integrating topic features according to claim 2 is characterized in that: The preprocessing in S1 first performs corresponding word segmentation on the short text i, and then cleans the result of the word segmentation, including redundant punctuation marks and redundant stop words. The cleaned result is then used as the input of the Word2Vec model to obtain the word vector set of text i; Secondly, the first m words under the topic with the highest probability in the short text are selected, and the probabilities corresponding to these m words are normalized, and the results are used as the weight values ​​corresponding to these m words; Finally, a weighted sum is performed to obtain the short text topic features.

4. The short text classification method integrating topic features according to claim 3 is characterized by: The word vector set of text i is shown in formula (1): V i ={v i (c1),v i (c2),...,v i (c n )} (1) Among them, V i is the set of word vectors of all words in short text i, n is the number of words, v i (c1) is represented as the word vector representation of vocabulary c1.

5. The short text classification method integrating topic features according to claim 4 is characterized in that: The normalization operation is as follows: the topic with the highest probability in the current short text i corpus Select the first m words under the topic with the highest probability The probability corresponding to these m words Probability Perform normalization operation and use the result as the weight value corresponding to the m words. The normalization formula (2) is as follows:

6. The short text classification method integrating topic features according to claim 5 is characterized in that: The steps of weighted summation of short text topic features are as follows: express The corresponding value after normalization is k = {1, 2, 3, ..., m}. The corresponding word vector {v i (c1),v i (c2),...,v i (c m )} to perform weighted summation to obtain the short text topic feature T t i As shown in formula (3):

7. The short text classification method integrating topic features according to claim 2 is characterized in that: The node embedding of the feature graph in step S2 includes: Z T = {N T ,M T } represents a word-level component graph of type T, where N T Represents a node, represents the adjacency matrix, Represents the features corresponding to the word-level component graph, node features E T Through two layers of GCN learning, we can get: Where [ReLu(x)] i =max([x] i ,0), [B T ] ij =∑ j [M T ] ij , and It is a trainable parameter, and the definition of node N in the graph depends on the integrated features.

8. The short text classification method integrating topic features according to claim 7 is characterized in that: The adjacency matrix construction of the feature graph includes: ①PMI PMI is used to measure the correlation between two variables. The calculation formula of PMI is as follows: Where x and y are two variables, the larger the value of PMI, the higher the correlation between the two variables; Cosine Similarity: Cosine similarity is the cosine value of the angle between two vectors in the vector space. It is used to measure the similarity between two vectors. The calculation formula of cosine similarity is as follows: in and are two vectors. In trigonometric functions, the function value of the cosine function is [-1,1]. Therefore, the cosine similarity between two vectors ranges from [-1,1]. When the angle between the two vectors is 0°, the similarity is 1, and when the angle is 180°, the similarity is -1.

9. The short text classification method integrating topic features according to claim 2 is characterized in that: Word Graph G W = {N W ,M W }, N is the set of words in the dataset, and PMI is calculated between any two words in the entire dataset to construct M. Then use formula (4) to learn the node representation function E based on the graph W (w is the type of graph), so that the representation of each node can be calculated, that is, the embedding of each word, and the initial representation uses onehot encoding; Part-of-speech tag graph G P = {N P ,M P }, the construction and node representation of this graph are similar to the word graph. The only difference is that the words in the original dataset are replaced with part-of-speech tags. e = {N e ,M e }, the node set N comes from the entity types defined in the NELL knowledge base, and the adjacency matrix M is: x e It is the embedding of entity type, which is learned using the classic knowledge graph embedding method TransE. In general, the NELL knowledge base is used to identify the entities contained in each text, and then the TransE method is used to learn the embedding of each entity. Finally, the cosine distance is used to calculate the similarity between two entities to fill M.

10. The short text classification method integrating topic features according to claim 9 is characterized in that: In the feature map G S = {N S ,M S }, where N represents the short text dataset, M represents the adjacency matrix, and hierarchical pooling is used to learn the similarity between short texts M S , and perform label propagation learning; We first use the above three feature graphs to represent each node, and then calculate the similarity by the inner product of two nodes; Among them, the subscript T indicates the type of feature map, E T is the node embedding of the corresponding graph, u is the L1 norm, For word graphs and part-of-speech tag graphs, S T represents the TF-IDF value of each word or part-of-speech tag in document i. For an entity graph, if the entity in the graph exists in the current document i, then S T It is 1 if yes, otherwise it is 0. Finally, a text i is obtained by connecting the representations of these three feature maps with the topic features, as shown in formula (10): Then use the inner product to represent the relationship matrix M: Among them, δ is the threshold for measuring the similarity between texts; Then GCN is used to learn the node embedding of the graph, as shown in formula (12): in, and is a trainable parameter, Its classification loss is shown in formula (13): in, Record the index of the short text with the label, is a one-hot vector of 0.