Commodity dynamic prediction method and system based on web crawler

Abstract

The invention relates to a commodity dynamic prediction method and system based on a web crawler. A user can customize an analysis operation flow according to a specific service scene through a complete network data analysis method of data crawling, cleaning, processing, querying, modelling and visually presenting, the maximization of the working efficiency is realized, the data of multiple websites can be crawled by using various targeted anti-crawling technologies, the automatic acquisition of the back-end work data is supported, and the data leaning, the normalization, the smooth processingand like method can be performed by providing multiple statistics method; and then the actual service scene is combined to perform modelling and analysis so as to display in various visual diagrams;the data quality is improved, and the system can help user making decision.

Description

technical field [0001] The invention relates to the technical field of network information processing, in particular to a method and system for commodity dynamic prediction based on a web crawler. Background technique [0002] In systems that obtain information from massive commodity information data sources on the Internet, most of them are to improve query and retrieval efficiency. Their respective characteristics often form their own update modes, and dynamically predict the bandwidth required by each crawler as a basis for dynamic bandwidth allocation. That is, the web crawler systems on the market are all technical research aimed at improving crawling efficiency or retrieval efficiency, and lack data-based statistical processing, presentation, and mining of deep value of data. The above crawling technologies lack the support of business scenarios, and have little improvement in actual work, which is not conducive to promotion. In fact, with the popularization of web c...

AI Technical Summary

Problems solved by technology

The above crawling technology lacks the support of business scenarios, and the improvement effect on actual work is small, which is not conducive to promotion

In fact, with the popularization of web crawler technology, more and more websites will formulate various anti-crawling strategies based on the protection of data assets, resulting in crawling failure and even IP blocking. In the form of text, tables, etc., it will be stored in the form of pdf, word, etc., and for some websites, the data update time is relatively frequent, and the data level is as high as tens of millions, which has a great impact on the entire data retrieval

Images