Method and system for Chinese full-text search in database

Abstract

The invention discloses a method and system for Chinese full-text retrieval in a database. The method comprises the following steps: receiving text to be retrieved; performing binary word segmentation processing on each two Chinese characters of the to-be-retrieved text to obtain a plurality of binary words. metaphrase, and insert the data table file; create an inverted index for the data table file, the inverted index contains the position index of each of the binary phrases, and is used to write the corresponding phrase in the database during the retrieval process. The position information in each text data in , the position information includes the row containing the phrase, and the position in the row; according to the plurality of bigrams, the full text of the to-be-retrieved text is performed in the database. retrieve. The retrieval method of the present invention does not need to construct a dictionary, has better retrieval effect on new words, and has higher retrieval efficiency by introducing a multi-level index mechanism.

Description

technical field [0001] The disclosure belongs to the technical field of data retrieval, and in particular relates to a method and system for Chinese full-text retrieval of a database. Background technique [0002] Full-text retrieval technology is a very common information query application, and one of the core technologies of various search engines on the Internet is full-text retrieval. A full-text search product is essentially a database product with embedded full-text search technology. Chinese word segmentation is involved in the Chinese full-text search process. [0003] At present, the main Chinese word segmentation can be divided into: word segmentation method based on string matching and word segmentation method based on statistics. The word segmentation method based on string matching needs to match the Chinese character string to be analyzed with the entry in a dictionary. If a certain string is found in the dictionary, it is considered that a word has been reco...

AI Technical Summary

Problems solved by technology

The word segmentation method based on string matching needs to match the Chinese character string to be analyzed with the entry in a dictionary. If a certain string is found in the dictionary, it is considered that a word has been recognized. This word segmentation method requires a "complete enough "Dictionary, but due to the rapid update of new words on the Internet, the update of the dictionary is difficult to adapt to the update speed of new words

If the text to be retrieved contains a new word on the Internet but not in the dictionary, the word cannot be segmented and processed correctly, so that the text containing the new word cannot be retrieved, resulting in missed detection

[0004] The word segmentation method based on statistics uses the frequency or probability of co-occurrence of adjacent words in the text to perform word segmentation. This method only needs to count the frequency of word groups in the corpus and does not require a dictionary. However, this method often extracts some co-occurrences. Common word groups with high occurrence frequency but not words have a certain recognition effect on new words, but the recognition accuracy of common words is poor, and the calculation is time-consuming, and the amount of data generated by word segmentation is also relatively large, which affects subsequent retrieval s efficiency

[0005] On the basis of the word segmentation, in order to speed up the retrieval efficiency, the inverted index is commonly used in database products for data processing. Specifically, after the database receives the data file to be inserted, it first reads the data file for Chinese word segmentation. After the word segmentation, it needs Read it again, get the position of each phrase in the data file and write it into the inverted index, that is, read the data file twice. In the case of a large data file or a large amount of data inserted into the database, this The processing method has a large amount of computation and low efficiency; moreover, when the general inverted index stores the word position, it only stores the row position where the phrase is located. The data is read out, and then the similarity is calculated, and the retrieval efficiency is low

Images