Text clustering method and system

Abstract

The invention relates to a text clustering method and system. The text clustering method comprises the following steps: keywords of to-be-classified texts are extracted when the to-be-classified texts are received; the keywords of the to-be-classified texts are matched according to the obtained keywords in a final word bag, and the type tag of the to-be-classified text is obtained; the final word bag is obtained by sorting and screening the key words in various type tag word bags according to preset selection rules; the type tag word bags are sets of key words generated after key word extraction from texts corresponding to type tags. The key words corresponding to each tag are extracted through records of existing tags, the final word bag is obtained, to-be-classified texts are classified according to the key words in the final word bag, good adaptability to noise data is realized, and the condition that the accuracy is reduced substantially under the condition of more noise is avoided; an approximate string matching effect is improved greatly through large-range thresholding of a centroid.

Description

technical field

[0001] The invention relates to the technical field of digital text mining, in particular to a text clustering method and system. Background technique

[0002] The traditional text clustering technology is mainly based on the Rocchio algorithm of TFIDF (Term frequency–inverse document frequency). The Rocchio algorithm comes from the vector space model theory. The basic idea of ​​the vector space model Vectorspace model is to use a vector to represent a text, and the subsequent processing process is Can be converted to operations on vectors in space. Rocchio based on TFIDF is an implementation method of this idea, in which an N-dimensional vector is used to represent the text, the vector dimension N is the number of features, and the vector component is a certain weight of the feature. The calculation method representing the weight is called TFIDF method. Through the TFIDF method, the text in the training set is first expressed as a vector, and then the cate...

Examples

Embodiment 1

[0024] In order to solve the problems of low accuracy and poor matching effect of traditional text clustering methods, the present invention provides a text clustering method embodiment 1; figure 1 It is a schematic flow diagram of Embodiment 1 of the method for text clustering of the present invention; as figure 1 As shown, the following steps may be included:

[0025] Step S110: When receiving the text to be classified, extract the keywords of the text to be classified

[0026] Step S120: According to the keywords in the obtained final bag of words, match the keywords of the text to be classified to obtain the type label of the text to be classified; wherein, the final bag of words is a collection of words from all kinds of word bags according to the preset selection rules It is obtained after sorting and filtering the keywords; the class tag word bag is a set of keywords generated after keyword extraction of each text corresponding to each type of tag.

[0027] Specifical...

Embodiment 2

[0052] In order to solve the problems of low accuracy and poor matching effect of traditional text clustering methods, the present invention also provides a text clustering method embodiment 2; image 3 It is a schematic flow diagram of Embodiment 2 of the method for text clustering of the present invention; as image 3 As shown, the following steps can be included, generating keywords→constructing word bag through keywords→adjusting word bag→using word bag classification, including:

[0053] Step S310: extract keywords according to TFIDF;

[0054] TF can be calculated based on the following formula: (the number of times the word appears in the document) / (the total number of words in the document), the larger the value, the more important the word, that is, the greater the weight.

[0055] For example: after a document is segmented, there are a total of 500 word segments, and the word segment "Hello" appears 20 times, then the TF value is: tf=20 / 500=2 / 50=0.04;

[0056] IDF c...

technical field

[0001] The invention relates to the technical field of digital text mining, in particular to a text clustering method and system. Background technique

[0002] The traditional text clustering technology is mainly based on the Rocchio algorithm of TFIDF (Term frequency–inverse document frequency). The Rocchio algorithm comes from the vector space model theory. The basic idea of ​​the vector space model Vectorspace model is to use a vector to represent a text, and the subsequent processing process is Can be converted to operations on vectors in space. Rocchio based on TFIDF is an implementation method of this idea, in which an N-dimensional vector is used to represent the text, the vector dimension N is the number of features, and the vector component is a certain weight of the feature. The calculation method representing the weight is called TFIDF method. Through the TFIDF method, the text in the training set is first expressed as a vector, and then the cate...

Examples

Embodiment 1

[0024] In order to solve the problems of low accuracy and poor matching effect of traditional text clustering methods, the present invention provides a text clustering method embodiment 1; figure 1 It is a schematic flow diagram of Embodiment 1 of the method for text clustering of the present invention; as figure 1 As shown, the following steps may be included:

[0025] Step S110: When receiving the text to be classified, extract the keywords of the text to be classified

[0026] Step S120: According to the keywords in the obtained final bag of words, match the keywords of the text to be classified to obtain the type label of the text to be classified; wherein, the final bag of words is a collection of words from all kinds of word bags according to the preset selection rules It is obtained after sorting and filtering the keywords; the class tag word bag is a set of keywords generated after keyword extraction of each text corresponding to each type of tag.

[0027] Specifical...

Embodiment 2

[0052] In order to solve the problems of low accuracy and poor matching effect of traditional text clustering methods, the present invention also provides a text clustering method embodiment 2; image 3 It is a schematic flow diagram of Embodiment 2 of the method for text clustering of the present invention; as image 3 As shown, the following steps can be included, generating keywords→constructing word bag through keywords→adjusting word bag→using word bag classification, including:

[0053] Step S310: extract keywords according to TFIDF;

[0054] TF can be calculated based on the following formula: (the number of times the word appears in the document) / (the total number of words in the document), the larger the value, the more important the word, that is, the greater the weight.

[0055] For example: after a document is segmented, there are a total of 500 word segments, and the word segment "Hello" appears 20 times, then the TF value is: tf=20 / 500=2 / 50=0.04;

[0056] IDF c...

Images