Chinese word segmentation method based on Hash algorithm

Abstract

The invention discloses a Chinese word segmentation method based on a Hash algorithm, and relates to the field of natural language processing. The method comprises the following steps of S1, configuring a word segmentation device on a search engine and establishing a dictionary structure; s2, monitoring the return operation of the user, and obtaining the first character in an input box; s3, inputting the first character into a dictionary for primary searching and screening; s4, forming a tree by all words with the same first characters in the dictionary; s5, placing a second word in the word on a second layer of the tree, and creating a Hash index table; s6, carrying out Hash searching on the remaining characters; s7, after an IK reads the new lexicon, notifying the search engine to update; and S8, updating the dictionary information in the memory by the search engine. According to the invention, the Hash search is carried out on the first character by creating a dictionary storage mechanism, the dictionary structure and the algorithm of carrying out Hash search on the remaining characters via the tree result are established, and the search engine is updated by using IK word segmentation, so that the Chinese word segmentation efficiency is improved, the system complexity is reduced, and the index redundancy degree is reduced.

Description

technical field [0001] The invention belongs to the field of natural language processing, in particular to a Chinese word segmentation method based on a hash algorithm. Background technique [0002] With the continuous development of the Internet, the number of web pages has increased dramatically, and web pages have become the largest and most extensive source of information for people. A lot of useful information is submerged in the vast number of Web pages. Faced with massive information, people can no longer simply rely on manual processing of all the information. Text search is one of the important application technologies in the field of large-scale information processing, and it is also an important research direction in the field of information processing. With the in-depth study of text classification search technology, text search technology is more and more widely used in information technology. The word segmentation technology is the "soul" of any search engine...

AI Technical Summary

Problems solved by technology

[0010] The object of the present invention is to provide a kind of Chinese participle method based on Hash algorithm, carry out Hash search to first word by creating dictionary storage mechanism, set up the dictionary structure and algorithm that tree form result carries out Hash search to remaining word, and utilize IK Word segmentation updates the search engine, which solves the problems of slow matching speed, low efficiency and redundant indexes in the existing Chinese word segmentation method

Images