Dictionary-based lucene Chinese word segmentation method

Abstract

The invention discloses a dictionary-based Chinese word segmentation method. The method comprises the steps of collecting linguistic data; establishing a terminological dictionary, wherein the establishing method comprises the steps of removing stop words firstly, dividing the linguistic data into text fragments, exacting candidate words from the text fragments, obtaining the appearance probability of the candidate words and each individual character in all the text fragments through statistics, calculating the mutual information of two Chinese characters in each candidate word, keeping the candidate words if mutual information is larger than a preset mutual information threshold value, deleting the candidate words otherwise, combining the candidate words obtained after screening, matching and filtering the combined candidate words by means of a general dictionary, and adding the candidate words obtained after filtration into the terminological dictionary; conducting word segmentation on a text with words to be segmented by means of the terminological dictionary firstly, and then conducting word segmentation on the rest of texts by means of the general dictionary. The terminological dictionary is established by extracting terminologies from the linguistic data through statistics, universality is high, and requirements of the professional field can be effectively met by conducting word segmentation with the terminological dictionary.

Description

technical field [0001] The invention belongs to the technical field of Chinese word segmentation, and more specifically relates to a dictionary-based Lucene Chinese word segmentation method. Background technique [0002] There is an obvious difference between Chinese information and English information. English words are separated by spaces; in Chinese text, there is no obvious separator between words, and Chinese words are mostly composed of two or more Chinese characters. , and the sentences are written consecutively. This means that before automatic analysis of Chinese text, a whole sentence must be cut into small lexical units, namely Chinese word segmentation. Chinese word segmentation is a difficult point in today's Chinese information processing and retrieval, and it is an inevitable problem in the research field. Now, Chinese word segmentation has achieved some results, and has been widely used in many aspects such as information retrieval. [0003] With the rapid ...

AI Technical Summary

Problems solved by technology

But currently Lucene only provides Chinese single-character and double-character word segmentation mechanisms, and these two Chinese word segmentation modules cannot well support Lucene Chinese analysis and processing

Images