An Orderly Construction and Retrieval Method of String Data Dictionary

Abstract

The invention discloses a method for orderly constructing and retrieving a string data dictionary. The method includes S1, inputting string data into a temporary bursting tree one by one; S2, when data quantity in the temporary bursting tree reaches a threshold value condition, combing the data into a final bursting tree; S3, converting the final bursting tree into a six-tuple structure finite state converter; S4, compiling the six-tuple structure finite state converter to be in a three-array structure form; S5, according to application needs, utilizing the three-array structure finite state converter after being compiled to realize retrieving or order transversing of the data dictionary. By the method, high-efficiency dictionary construction can be performed on millions of data items, and retrieval needs in different environments and application can be met.

Description

technical field [0001] The invention relates to the fields of information retrieval, natural language processing and pattern recognition and matching, in particular to an orderly construction and retrieval method of a dictionary suitable for string data of any scale. Background technique [0002] Dictionary construction and retrieval for string data has always been an important technical link in many applications in the fields of information retrieval, natural language processing, and pattern recognition and matching. The construction and retrieval speed of dictionaries largely determine the overall performance of the application system. For example, the positioning of inverted index items in search engines, word segmentation and synonym replacement in text processing, spell checking in text editors, and text association in input methods all have very high requirements on the construction of corresponding dictionaries and retrieval performance. [0003] Due to the key positi...

AI Technical Summary

Problems solved by technology

In the research results of perfect hash function design, the time overhead for generating a perfect hash function is usually very high. Even the most famous perfect hash function generator Gperf under the Linux system cannot guarantee that it can still be used under large-scale data. Generate a perfect hash function. When the dictionary entries exceed 15,000, its hash performance is poor, especially for multi-byte large character set languages ​​​​like Chinese.

On the other hand, since the design goal of the hash function is uniform distribution, the order of the dictionary entries is usually not guaranteed. When performing ordered traversal, all dictionary entries must be sorted first, which is almost impossible for practical application in terms of efficiency.

Images