Massive search query log calculation and analysis system based on cloud platform

Abstract

The invention provides a massive search query log calculation and analysis system based on a cloud platform. The invention aims at characteristics and requirements of an internet e-commerce platform,designs a structured log of a search engine, and provides the structured log data for a log analysis system to be applied, so the space required by traditional log analysis and the workload of log cleaning work are greatly reduced; an efficient e-commerce platform search query log analysis system is realized in combination with a Hadoop distributed computing platform and a big data processing algorithm, thus mining data back value; key points and excavation potential are fully grasped by feature selection of the e-commerce platform, so massive log key extraction information currently needed bythe e-commerce platform can be clearly and visually displayed, enough expansion capacity can be provided for mining of various interested features of logs, website optimization and precision marketing are achieved by analyzing behavior logs of users, personalized services are provided, and the invention has huge utilization value.

Description

technical field [0001] The invention relates to a search log calculation and analysis system, in particular to a cloud platform-based massive search query log calculation and analysis system, which belongs to the technical field of log calculation and analysis. Background technique [0002] With the rapid development of computers and informatization all over the world, it has penetrated into all aspects of social life, and the Internet has become a huge global information business center, closely connecting people all over the world and providing people with It has brought about huge changes in all areas of life. Especially in the past few decades, China has gradually embarked on the track of rapid Internet development. With the gradual popularization of Internet devices, the improvement of the network environment, the increasingly rich Internet application scenarios, the increasingly convenient logistics and the huge number of Internet users, China's electronic The busines...

AI Technical Summary

Problems solved by technology

[0007] One is the explosive growth of network data behind the current prosperity of the Internet and e-commerce. In the face of massive information and data, it is becoming more and more difficult for current technologies to choose and find useful information quickly and efficiently.

The existing technical information acquisition solution is to use classified directories and search engines. The classified directory is to classify the website URLs commonly used by users according to the main content and functional characteristics of the website. However, with the rapid development of the Internet, a large number of newer website information does not have Appearing in the classified directory, because the information is too rich, it is difficult to find it, and gradually cannot meet people's needs

Search engine If the user has clear search needs, the efficiency of the search engine is high, but many times the user does not know what they want to find, and it is difficult to find the desired information at this time. Such needs are found in the search engines of e-commerce platforms. More obviously, the existing technology cannot recommend high-quality products that users like based on user habits, and the user experience is not good, which creates great inconvenience for both users and merchants

[0008] The second is that whether it is a classified directory or a search engine in the prior art, if you want to count user information, analyze user habits, and improve the analysis system, the method is to analyze the log according to the log of the WEB site, which is mainly divided into three steps: one is pre-processing , Most of the WEB logs are unstructured or semi-structured data. The existing data mining algorithms cannot be directly applied to the original log data, and complex pre-processing must be performed to obtain valuable information, which is time-consuming and laborious. And the effect is not good; the second is pattern recognition, by adopting appropriate data mining techniques and algorithms to process the data files generated in the

Images