System and method for rapidly searching unstructured data

Abstract

The invention discloses a system and a method for rapidly searching unstructured data. The system comprises a query processing module, a query condition analyzing module, a large data storage module, a Bloom filter filtering module, an index establishment module and a Bloom filter building module, wherein the query processing module is used for receiving the search request of an application client, acquiring the query conditions from the search request, calling the query condition analyzing module to analyze and transform the query conditions, using a Bloom filter to replace the word segmentation condition in the original query conditions and generating a new query condition based on Bloom filter filtering record identification; the Bloom filter and other query conditions are tested by a condition query data record sheet; the data record which is in accordance with all the query conditions is taken as the search result; the search result is returned to the application client. The system and the method can more quickly judge whether a data record is in accordance with the word segmentation condition or not, and consumes less resource.

Description

technical field [0001] The application belongs to the field of information technology, and in particular relates to a system and method for fast retrieval of unstructured data in large data volume storage. Background technique [0002] With the rapid development of the Internet, applications such as webpages, blogs, social networks, and instant messaging software have become popular rapidly, and a large amount of unstructured text data has been generated. How to quickly and effectively retrieve these text data has become a research hotspot in the industry. Internet search engines establish reverse indexes from word segmentation to text identification through word segmentation operations on text data, and through these indexes, text retrieval based on word segmentation is realized. Although text data can be retrieved through the reverse index, when the user searches with multiple search conditions, or the search conditions contain multiple search terms, a large number of text...

AI Technical Summary

Problems solved by technology

Although text data can be retrieved through the reverse index, when the user searches with multiple search conditions, or the search conditions contain multiple search terms, a large number of text identification comparisons are required, which consumes a lot of system resources. The consumption is relatively large, and the performance is low

Images