A text processing method and system

Abstract

The invention provides a text processing method and system. The text processing method comprises the steps of S1, establishing a classification hyperplane function; and S2, performing prediction on a newly-input text via the classification hyperplane function. The step S1 specifically comprises the sub-steps of S10, performing work segmentation processing on a text and establishing an entry document matrix ; S20, extracting features from the entry document matrix via the decision tree algorithm; S30, constructing the classification hyperplane function. The method and the system have the advantages that after word segmentation of a stored text, the sentence features of the text are extracted; the features are extracted according to the decision tree algorithm, so that the number of dimensions of model training points in a support vector machine is reduced and the training time is shortened. The feature vectors of texts are extracted according to decision tree training and text classification is performed by using a multi-core support vector machine according to the feature vectors, so that the method and the system have the advantages of accurate calculation, fewer model training samples, short training time and high text classification accuracy.

Description

technical field [0001] The invention relates to the technical field of intelligent text information processing, in particular to a text processing method and system. Background technique [0002] 80% of social big data is unstructured data, and unstructured big data processing is the biggest challenge big data faces. Structured data analysis cannot fully mine and discover semantics in big data. The challenge of unstructured text mining lies in: maintenance challenges brought about by language diversity, including the variety of language expressions in the text, irregular usages such as abbreviations and abbreviations are common, and it is necessary to exhaustively enumerate all language expressions. Difficulty in maintenance due to language expression details; maintenance challenges brought about by many business categories and fast-changing rules: There are many business categories and fast-changing categories. Every time a category changes, it is necessary to reorganize t...

AI Technical Summary

Problems solved by technology

Structured data analysis cannot fully mine and discover semantics in big data

The challenge of unstructured text mining lies in: maintenance challenges brought about by language diversity, including the variety of language expressions in the text, irregular usages such as abbreviations and abbreviations are common, and it is necessary to exhaustively enumerate all language expressions. Difficulty in maintenance due to language expression details; maintenance challenges brought about by many business categories and fast-changing rules: There are many business categories and fast-changing categories. Every time a category changes, it is necessary to reorganize the language rules of all related categories, and the maintenance workload is huge. , maintenance efficiency is low; challenges brought by multilingual synchronous processing: mining of different languages ​​needs to be analyzed at the same time, and rules need to be established for each language separately, requiring maintenance business personnel to master multiple languages, and the requirements for maintenance personnel are too high; noise in the text The classification challenge brought by big data: the length of the text is different, and the correlation among them is intricate, so it is impossible to use the method of keyword statistics to achieve a good mining effect

[0003] However, the existing technologies generally use statistical methods for text mining, without considering the needs of business personnel, and only provide mining algorithms, which has brought a lot of trouble to business personnel

The problem faced by text mining technology is how to analyze and mine valuable information that users care about from a piece or a large amount of unstructured text, so that business personnel can define mining requirements and mining rules from a business perspective, regardless of the language expression in the text Linguistic Ambiguity Caused by the Diversity of Habits

Images