Chinese text parallel data mining method based on hierarchy

A data mining and text technology, applied in the field of information processing, can solve the problems of low mining efficiency and large amount of original data, and achieve the effects of improving calculation speed, high word segmentation accuracy, and high efficiency

Active Publication Date: 2012-09-12
UESTC COMSYS INFORMATION
View PDF5 Cites 35 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention aims at the shortage of too much original data in the current Chinese text data mining process, the mining efficiency is too low, and the clustering process can only deal with circular clusters, and proposes a hierarchical Chinese text parallel data mining method

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Chinese text parallel data mining method based on hierarchy
  • Chinese text parallel data mining method based on hierarchy
  • Chinese text parallel data mining method based on hierarchy

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0013] The present invention will be further described below in conjunction with the drawings and specific implementations.

[0014] A hierarchical-based parallel data mining method for Chinese text. Its characteristic is that it includes the following steps:

[0015] Step 1: Establishment of the Chinese text vector space model: By segmenting the entire Chinese text set, the word segmentation form of each text and the feature entry set containing all the deduplication entries in the text set are obtained, and then each feature entry set is used to count each The term frequency inverse document frequery (TFIDF) of the text is used to establish a text vector space model based on the term frequency inverse document frequency (TFIDF).

[0016] The definition of term frequency inverse document frequency (TFIDF): it refers to an index that a term represents the amount of text information that contains the term. The calculation formula is: TFIDF ij =TF ij *IDF i

[0017] TF ij Refers ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a Chinese text parallel data mining method based on hierarchy, comprising the steps of: step 1: a establishing vector space model of Chinese texts: performing work segmentation regarding to the entire Chinese text set to obtain a word segmentation form and a feature term set containing all removed duplicated terms in the text set of each text, then using the feature term set to count the term frequency-inverse document frequency (TFIDF) of each text, and establishing the text vector space model according to the TFIDF; step 2: performing dimension reduction regarding to a feature item vector of the text vector space model; and step 3: clustering texts using DCURE algorithm based on hierarchy. The method is efficient in word segmentation of Chinese texts with high accuracy, requires no input of parameters like radius of neighborhood for the clustering process, can mine irregular cluster and is insensitive to noise, employs distributed calculating, has high efficiency in mining mass texts and improves calculating speed of feature weight.

Description

Technical field [0001] The invention belongs to the field of information processing technology, and specifically relates to the field of computer data mining and machine learning technology. It can be used for the improvement of search engine search results, a personalized content recommendation system, a question and answer system for aggregating similar questions, and news websites grouping news and other information by category The field of processing technology plays a role. Background technique [0002] With the development of the Internet, the text information on web pages has grown rapidly. How to index, retrieve, manage, and mine the massive text information on web pages has become a huge challenge in the field of computer science. Chinese text clustering technology is also constantly developing and mature. Distributed technology has been more and more widely used, and distributed clustering technology is an important research content in the field of distributed data mini...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F17/27
Inventor 唐雪飞罗石唐先萍
Owner UESTC COMSYS INFORMATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap