Universal internet data acquisition method

Abstract

The invention provides a universal internet data acquisition method. The method comprises the steps of executing transaction scheduling, judging the type of an acquisition transaction, and if the acquisition transaction is a media or a file link, executing corresponding document acquisition processing; if the access address of the webpage acquisition transaction is not in a history grasping library, conducting acquisition according to a newly found webpage; obtaining the last acquisition information of the webpage address from the history grasping library if the acquisition transaction is in the history grasping library; comparing the amount of page content of a current webpage address to the amount of the last webpage content if an internal time exceeds a renewing frequency, if the amount of the webpage content of the current webpage address is not equal to that of the last webpage content, obtaining a webpage source code of the webpage link, renewing acquisition information of the webpage address in a history access library, and executing webpage washing and extraction. The invention provides a universal internet data acquisition method. According to the universal internet data acquisition method, by utilizing a transaction control strategy to conduct efficient data acquisition, data mining is conducted aiming at a coupling relation among multi-dimensional objects.

Description

technical field [0001] The invention relates to data retrieval, in particular to a general Internet data collection method. Background technique [0002] With the continuous development of Web technology, network information resources are growing geometrically. How to quickly retrieve useful data related to users from the massive information on the Internet has become an urgent problem to be solved. Search engines are developed on the basis of information retrieval technology. The search engine helps the invention better express and store essential information in the real world, and by analyzing the connection information in the search engine, it can be used as a useful tool for mining hidden information. Existing search engines simply rely on limited search words to express user needs, which has the problem of incomplete expression. Even for the same search term, different users may expect different results. For example, the Weibo system, if considering the relationship...

AI Technical Summary

Problems solved by technology

Existing search tools do not consider the complex environment formed by the above multi-dimensional objects for data mining

Images