Method for optimizing word segmentation of search engine through precomputation and word segmenting device of search engine

Abstract

The invention provides a method for optimizing the word segmentation of a search engine through precomputation. The method comprises the following steps of: coding a character in a word segmentation dictionary according to the number of sub nodes of a Trie tree, so as to generate a sequence code, wherein the character with many numbers of the sub nodes of the Trie tree is coded preferentially; carrying out the precomputation according to the sequence code, so as to generate a first array and a second array of a double-array Trie tree; and carrying out word segmentation inquiry in the word segmentation dictionary according to the sequence code, the first array and the second array. By using the method, the spatial utilization rate of the word segmentation of the search engine is improved; the loading speed of a word segmenting module is quickened; and the stability of an online service is enhanced. The invention further discloses a word segmenting device of the search engine.

Description

technical field [0001] The invention relates to the technical field of computer science, in particular to a method for optimizing search engine word segmentation through pre-calculation and a search engine word segmentation device. Background technique [0002] Word segmentation is the most basic function of search engines, and it is a technology that search engines use various methods to match according to the keyword strings submitted by users. In order to achieve a better word segmentation effect in existing search engines, word segmentation dictionaries generally have relatively large capacity, and use plaintext word segmentation dictionaries to generate internal data structures through online calculations. Existing search engines generally adopt data structures such as linear index table, inverted list, hash (hash) and search tree. Both the linear index table and the inverted list are static index structures. Hash (Hash) maps characters to a certain storage location t...

AI Technical Summary

Problems solved by technology

[0004] (1) The search time of static search is related to the size of the word segmentation dictionary. When the dictionary capacity is large, the response time is long, and it is difficult to reach the order of O(1), and the static search needs to be changed when the dictionary is changed.

[0005] (2) The hash search speed is fast, and the time complexity can be optimized to O(1), but the space utilization rate is low, the memory usage is large, and the query time also depends on the design of the Hash conflict handling mechanism

[0006] (3) The large word segmentation dictionary leads to a long loading time, which makes the response of the search product to the failure not quick enough, which is easy to cause large-scale loss of traffic. At the same time, the long startup time of the module will also affect the efficiency of daily product operation and maintenance.

[0007] (4), a large number of calculations and frequent IO operations during the loading process are likely to have an impact on the stability of online services

Images