Method and device for extracting hot word phrases from document set

Abstract

The invention discloses a method and a device for extracting hot word phrases from a document set. The method comprises performing word segmentation on every clause in the document set through a word segmentation unit; judging the phrase boundary distinctness degree and or the closeness degree of the relation between words in every phrase which is formed by less than K continuous words in every clause through a judgment unit, wherein K is a positive integer and the boundary distinctness degree is used for indicating the matching freedom degree of phrases and words located on the left sides and the right sides of the phrases; at least extracting a part of phrases from the phrases which are formed by the less than K continuous words based on a judgment result of the phrase boundary distinctness degree and or the closeness degree of the relation between the words in every phrase through a hot word phrase extraction unit to serve as the hot word phrases to be output. Compared with the prior art, the hot word phrases can be accurately extracted from various corpuses.

Description

technical field [0001] The invention relates to information processing technology, in particular to a method and device for extracting hot word phrases from document collections. Background technique [0002] With the explosive growth of Internet information, people have a higher and higher demand for hot topics to obtain hot information, such as the topic "Development and Reform Commission", "Security Regulatory Commission", "Ye Bao", etc. to obtain related objects and events Therefore, how to better extract hot word phrases from various corpus resources has become an important topic in the field of natural language processing. As far as the existing hot word phrase extraction technology is concerned, it usually relies on the word segmentation system. If the word segmentation effect of the word segmentation system is not good, it will directly lead to the poor effect of hot word phrase extraction. Segmentation of sentences in regular documents such as newspapers has a bett...

AI Technical Summary

Problems solved by technology

As far as the existing hot word phrase extraction technology is concerned, it usually relies on the word segmentation system. If the word segmentation effect of the word segmentation system is not good, it will directly lead to the poor effect of hot word phrase extraction. Segmentation of sentences in regular documents such as newspapers has a better effect of word segmentation, but the effect of word segmentation for irregular words on the Internet is poor, which further leads to the failure of most Internet hot word phrases to be extracted well; and , the existing hot word phrase extraction technology can usually only extract phrases containing fewer words, such as two words or three words, and the extraction of phrases is more dependent on linguistic rules (such as grammar and syntax rules), flexible In addition, the existing hot word phrase extraction technology usually adopts a strategy of expanding shorter words into longer words, resulting in the inability to extract most of the longer and noise-containing words (such as "的", " "Let" and "To") were successfully extracted, which means that the existing hot word phrase extraction technology is less robust

Images