Chinese text parallel data mining method based on hierarchy

A text and Chinese technology, applied in the field of information processing, can solve problems such as low mining efficiency and large amount of original data, and achieve the effects of improving calculation speed, high word segmentation accuracy, and high efficiency

Active Publication Date: 2015-04-15
UESTC COMSYS INFORMATION
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention aims at the shortage of too much original data in the current Chinese text data mining process, the mining efficiency is too low, and the clustering process can only deal with circular clusters, and proposes a hierarchical Chinese text parallel data mining method

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Chinese text parallel data mining method based on hierarchy
  • Chinese text parallel data mining method based on hierarchy
  • Chinese text parallel data mining method based on hierarchy

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0013] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0014] A layer-based Chinese text parallel data mining method is characterized in that it comprises the following steps:

[0015] Step 1: Establishment of the Chinese text vector space model: By segmenting the entire Chinese text collection, the word segmentation form of each text and the feature entry set containing all deduplicated entries in the text set are obtained, and then the feature entry set is used to count each The term frequency inverse document frequency (TFIDF) of the text, and the text vector space model is established according to the term frequency inverse document frequency (TFIDF).

[0016] Definition of Term Frequency Inverse Document Frequency (TFIDF): It refers to an index that an entry represents the amount of text information that contains the entry. Its calculation formula is: TFIDF ij =TF ij *IDF i

[0017] TF i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a Chinese text parallel data mining method based on hierarchy, comprising the steps of: step 1: a establishing vector space model of Chinese texts: performing work segmentation regarding to the entire Chinese text set to obtain a word segmentation form and a feature term set containing all removed duplicated terms in the text set of each text, then using the feature term set to count the term frequency-inverse document frequency (TFIDF) of each text, and establishing the text vector space model according to the TFIDF; step 2: performing dimension reduction regarding to a feature item vector of the text vector space model; and step 3: clustering texts using DCURE algorithm based on hierarchy. The method is efficient in word segmentation of Chinese texts with high accuracy, requires no input of parameters like radius of neighborhood for the clustering process, can mine irregular cluster and is insensitive to noise, employs distributed calculating, has high efficiency in mining mass texts and improves calculating speed of feature weight.

Description

technical field [0001] The invention belongs to the technical field of information processing, specifically relates to the technical field of computer data mining and machine learning, and can be used for improving search results in search engines, personalized content recommendation systems, aggregation of similar questions in question answering systems, and aggregation of news by category on news websites and other information The processing technology field plays a role. Background technique [0002] With the development of the Internet, the text information on the webpage is increasing rapidly. How to index, retrieve, manage, and mine the massive text information on the webpage has become a huge challenge in the field of computer science. Chinese text clustering technology is also developing and maturing, distributed technology has been more and more widely used, and distributed clustering technology is an important research content in the field of distributed data minin...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30G06F17/27
Inventor 唐雪飞罗石唐先萍韩春梅
Owner UESTC COMSYS INFORMATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap