Distributed framework-based log data storing and retrieving method

Abstract

The invention provides a distributed framework-based log data storing and retrieving method and aims at solving the problem of high performance storage of mass user internet log data and the problem of slow data retrieval under multi-dimension and multi-keyword conditions. On one hand, the invention provides a method for storing user internet log data; HBase and Solr are mainly utilized to achieve distributed data storage and index construction; the method comprises the method for storing the user internet log data into the HBase in an entry form and building a unique identification, the method for carrying out dimension splitting and dimension word segmentation on the user internet log data and building an index relationship between the enquiry dimension and a word segmentation keyword and a data identification, the method for carrying out dimension splitting on the log data, the method for carrying out word segmentation on the dimension, and the method for building an index according to the dimension and the word segmentation keyword; on the other hand, the invention provides a method for retrieving the user internet log data; and the method mainly comprises an organization and data accessing method under the multi-dimension and multi-keyword conditions.

Description

technical field [0001] The present invention relates to the field of mobile communication, in particular to a method for storing and retrieving online log data of massive users in 2G, 3G and 4G data networks in a mobile communication network, and in particular to a method for storing and retrieving log data based on a distributed architecture. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. The core design of the Hadoop framework is: HDFS and MapReduce. HDFS provides storage for massive data, and MapReduce provides calculation for massive data. [0003] HBase is a NoSQL database system based on the Hadoop distributed system. It is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. [0004] Solr is a high-performance, Lucene-based full-text search server. At the same time, it has been extended to provide a richer query language than Lucene. At the same time, it is co...

AI Technical Summary

Problems solved by technology

[0009] The present invention provides a method for constructing an efficient distributed data storage and data retrieval system based on HBase and Solr, which solves the problem of high-performance storage of massive user log data and the problem of slow data retrieval under multi-dimensional and multi-keyword conditions; Including using HBase and Solr to realize the storage method of massive log data and the retrieval method of log data

Images