Massive search query log calculation and analysis system based on cloud platform

Abstract

The invention provides a massive search query log calculation and analysis system based on a cloud platform. The invention aims at characteristics and requirements of an internet e-commerce platform,designs a structured log of a search engine, and provides the structured log data for a log analysis system to be applied, so the space required by traditional log analysis and the workload of log cleaning work are greatly reduced; an efficient e-commerce platform search query log analysis system is realized in combination with a Hadoop distributed computing platform and a big data processing algorithm, thus mining data back value; key points and excavation potential are fully grasped by feature selection of the e-commerce platform, so massive log key extraction information currently needed bythe e-commerce platform can be clearly and visually displayed, enough expansion capacity can be provided for mining of various interested features of logs, website optimization and precision marketing are achieved by analyzing behavior logs of users, personalized services are provided, and the invention has huge utilization value.

Description

technical field [0001] The invention relates to a search log calculation and analysis system, in particular to a cloud platform-based massive search query log calculation and analysis system, which belongs to the technical field of log calculation and analysis. Background technique [0002] With the rapid development of computers and informatization all over the world, it has penetrated into all aspects of social life, and the Internet has become a huge global information business center, closely connecting people all over the world and providing people with It has brought about huge changes in all areas of life. Especially in the past few decades, China has gradually embarked on the track of rapid Internet development. With the gradual popularization of Internet devices, the improvement of the network environment, the increasingly rich Internet application scenarios, the increasingly convenient logistics and the huge number of Internet users, China's electronic The busines...

AI Technical Summary

Problems solved by technology

[0007] One is the explosive growth of network data behind the current prosperity of the Internet and e-commerce. In the face of massive information and data, it is becoming more and more difficult for current technologies to choose and find useful information quickly and efficiently.

The existing technical information acquisition solution is to use classified directories and search engines. The classified directory is to classify the website URLs commonly used by users according to the main content and functional characteristics of the website. However, with the rapid development of the Internet, a large number of newer website information does not have Appearing in the classified directory, because the information is too rich, it is difficult to find it, and gradually cannot meet people's needs

Search engine If the user has clear search needs, the efficiency of the search engine is high, but many times the user does not know what they want to find, and it is difficult to find the desired information at this time. Such needs are found in the search engines of e-commerce platforms. More obviously, the existing technology cannot recommend high-quality products that users like based on user habits, and the user experience is not good, which creates great inconvenience for both users and merchants

[0008] The second is that whether it is a classified directory or a search engine in the prior art, if you want to count user information, analyze user habits, and improve the analysis system, the method is to analyze the log according to the log of the WEB site, which is mainly divided into three steps: one is pre-processing , Most of the WEB logs are unstructured or semi-structured data. The existing data mining algorithms cannot be directly applied to the original log data, and complex pre-processing must be performed to obtain valuable information, which is time-consuming and laborious. And the effect is not good; the second is pattern recognition, by adopting appropriate data mining techniques and algorithms to process the data files generated in the pre-processing stage, find out the hidden data patterns that can reflect the user's specific behaviors, sessions, resources and concise data, but The existing technology does not have a targeted, efficient and practical method for these conveniences; the third is pattern analysis, which further analyzes the pattern information unearthed in the previous step, finds out the pattern of interest and then performs a visual output. Due to the existing technology log calculation The analysis and processing are not ideal, the information mining of logs is almost impossible, and there is no suitable solution for the method of interest points and visual output

[0009] The third is that most of the log analysis in the existing technology adopts a centralized method. The data analysis system is deployed on a single server node, and a series of data collection, storage, pre-processing and data mining are completed through this node. For complex work, when the amount of data processing is not large and the complexity of analysis work is not high, the work efficiency of a single node can basically meet the requirements

However, with the further expansion of the network scale, the amount of data that needs to be stored and analyzed by the e-commerce platform is very large, and the compressed data is at the terabyte level. The centralized log analysis and processing methods of the existing technology cannot solve the problem at all.

Considering the data size alone, single-node analysis can no longer meet the requirements of large-scale log processing

[0010] Fourth, the existing technology still has a massive log analysis system for the e-commerce platform. The general analysis system cannot complete the task of analyzing the massive data of Internet e-commerce, nor can it extract specialized analysis tools for the log data of the e-commerce platform, resulting in The analysis of massive data cannot be completed on the hardware, and the log data of the e-commerce platform cannot be adapted on the software. Contains a huge amount of potentially important information

Images