Distributed real-time search engine

Abstract

The invention relates to the technical field of search engines, specifically relating to a distributed real-time search engine. A system construction and operation method of the search engine at least comprises the following steps: A, designing a functional structure of a system; B, designing a data index structure of the system; C, creating an index; D, updating the index; and E, searching the index. The distributed real-time search engine can construct an updating index and a combining index simultaneously in the memory of the system, and can access the updating index and the combining index simultaneously while searching the index; when the number of the documents of the updating index is accumulated to a threshold value, the updating index is submitted to a disk index and changed as a combining index, and the original combining index is changed as a new updating index; and therefore, the updating data can be searched, and the real time property of the retrieval data of the search engine can be improved.

Description

technical field [0001] The invention relates to the technical field of search engines, in particular to a distributed real-time search engine. Background technique [0002] With the advent of the era of knowledge economy, the information on the Internet is growing explosively. At this stage, what people are facing is not the lack of information, but the flood of information, and there is no way to filter it. Therefore, how to obtain the required information accurately, quickly and in time is a must Problems that search engines need to solve. [0003] A search engine refers to a system that uses specific computer programs to collect information from a specific network, such as the Internet, according to a certain strategy. After organizing and processing the information, it provides users with retrieval services and displays relevant information to users. [0004] Traditional search engines, such as Google, Baidu, Yahoo, etc., although the amount of data processed has reache...

AI Technical Summary

Problems solved by technology

When the amount of data reaches terabytes, there is a great contradiction between the frequency of data update and the speed of query response, because when the amount of accumulated data is large and the amount of updated data is also large, it will lead to a long time for index construction and maintenance As a result, the real-time performance cannot be guaranteed, that is, when the existing search engine solution adopts the incremental indexing mechanism, the index construction and retrieval process are carried out separately, and the index construction logic is only when the number of documents accumulated in the new segment reaches the threshold ( Such as 10000) or after the interval reaches the threshold (such as 5 minutes), the new segment is submitted to the index fragment for index retrieval logic

Therefore, there will be a certain time delay between the submission of a document and the ability to retrieve the document, usually in the range of several minutes to tens of minutes, and in real-time retrieval, such a long delay is intolerable

Images