An implementation method of an analysis model supporting massive long text data classification

Abstract

The invention provides a realization method for an analysis model supporting massive long text data classification, and belongs to the technical field of big data text analysis. According to the method, standard word segmentation in an HanLP word segmentation tool and an improved CHI algorithm are adopted, so that on one hand, the dimension of a word vector space of each article during text classification is effectively reduced, the time complexity of text classification computing is lowered, the algorithm efficiency is improved, and the performance demand during massive long text classification under the background of big data is met; and meanwhile, the situation of reduced classification accuracy due to reduced vector space dimension is reduced to the maximum extent. A barrier between a text and a vector can be effectively eliminated by adopting a TFIDF algorithm, and finally the text can be accurately subjected to better training by adopting a Naive Bayesian classification algorithm, so that accurate classification of the long text is realized. The method can effectively solve the problem of contradictoriness of a performance index and an accuracy index of long text classification in a big data environment, and has a wide application prospect.

Description

technical field [0001] The invention belongs to the technical field of big data text analysis, and specifically relates to an analysis model for classifying a large amount of long text data that uses the CHI algorithm to extract the characteristic words of each type of text, the TFIDF algorithm to realize the vectorized representation of the text, and the naive Bayesian method to train and classify implementation method. Background technique [0002] Today's era is an era of rapid development of information technology. With the development of information technology, scientific knowledge has experienced rapid and explosive growth in a short period of time. A large amount of information is produced every day. 500,000 books are published in the world every year, and a new book is published every minute. On average, 13,000 to 14,000 papers containing new knowledge are published every day; more than 300,000 invention-creation patents are registered every year, and an average of ...

AI Technical Summary

Problems solved by technology

[0004] Aiming at these two problems of the statistical classification model, the present invention provides a corresponding solution to deal with the current actual classification needs, and proposes to use a CHI algorithm to extract the feature words of each type of text, and calculate the feature words of all categories The set intersection operation of the set can obtain the word vector space of the subsequent text classification, which can effectively reduce the dimension of the word vector space of each article during text classification, reduce the time complexity of text classification calculation, improve the algorithm efficiency, and meet the requirements of big data. Performance requirements for massive long text classification

Images