System and method for automatically classifying text

Abstract

A method is provided for automatically classifying text into categories. In operation, a plurality of tokens or features are manually or automatically associated with each category. A weight is then coupled to each feature, wherein the weight indicates a degree of association between the feature and the category. Next, a document is parsed into a plurality of unique tokens with associated counts, wherein the counts are indicative of the number of times the feature appears in the document. A category score representative of a sum of products of each feature count in the document times the corresponding feature weight in the category for each document is then computed. Next, the category scores are sorted by perspective, and a document is classified into a particular category, provided the category score exceeds a predetermined threshold.

Description

RELATED APPLICATIONS [0001] The following application is relied upon and hereby incorporated by reference in this application: [0002] U.S. Provisional Application No. 60 / 206,975, entitled “System and Method for Providing a Text Classifier,” bearing attorney docket no. 07569-6004-00000.TECHNICAL FIELD [0003] The present invention is directed to text classification and, more particularly, to a computer-based system for text classification that provides a resource that can be utilized by external applications for text classification. BACKGROUND [0004] Throughout the entire period of recorded history, people have memorialized their thoughts, actions, hopes and dreams on a daily basis. Prior to the latter part of the 20th century, this recorded history was typically written for exchange between human beings without any expectation that the information would be stored in a machine or otherwise converted into a machine-readable format. At that time, archives of this information resided in ...

AI Technical Summary

Problems solved by technology

While this approach is efficient, it suffers from problems relating to the inaccuracy of the retrieved information.

While deep linguistic processing improves accuracy based upon an analysis of the meaning of input text, speed of execution is slow and range of coverage is limited.

This is especially problematic when such techniques are applied to large volumes of text

Text classification systems which rely upon rule-base techniques also suffer from a number of drawbacks.

The most significant drawback is that such systems require a significant amount of knowledge engineering to develop a working system appropriate for a desired text classification application.

It becomes more difficult to develop an application using rule-based systems because individual rules are time-consuming to prepare, and require complex interactions.

Exacerbating the problem is the fact that in most applications, training data is hard to locate, often does not provide adequate coverage of the categories, and is difficult and time-consuming for people to categorize, requiring manual effort by experts in the subject area (who are usually scarce and expensive resources).

Further, badly categorized training data or correctly categorized training data with extraneous or unusual vocabulary degrades the statistical model, causing the resulting classifier to perform poorly.

Of the prior art systems that utilize training data, most do not have the capability to interactively take advantage of human knowledge.

History has shown that a person will often know what results from sound training data, what results from poor training data, and what may not be adequately expressed in the training data.

Those prior art systems that do utilize user input, do not allow users to directly affect the quantified relationship between vocabulary features and classification categories, but simply allow the user to change the training data.

These classifiers cannot interact, consequently one cannot benefit from the other.

Images