Webpage analysis container and method

Abstract

The invention discloses a webpage analysis container and method. The webpage analysis container comprises a webpage downloading module, a detection module, a script analysis module and a page rendering module. The webpage downloading module is used for sending requests to a web server repeatedly to obtain one html text of one webpage. The detection module is used for detecting the version of html in the html text, the version of a dynamic script of at least one dynamic script triggering event, and the classification of the version. The script analysis module is used for calling a script engine analysis the same as the version of the dynamic script of the dynamic script triggering event and the classification of the version, and operating at least one dynamic script triggering event. The page rendering module is used for calling a page rendering engine the same as the version of the detected html to render the webpage, and adding an operation result of the script engine to the webpage. According to the webpage analysis container and method, collection and analysis on complicated webpages including dynamic scripts of client sides are achieved, all contents in the webpage can be obtained, and precision and success rates of webpage collection are improved.

Description

technical field [0001] The present invention relates to a web page analysis container and method, in particular to a web page analysis container capable of collecting and analyzing web pages including client dynamic scripts and a web page analysis method realized by using the web page analysis container. Background technique [0002] With the rapid development of the Internet, various websites have appeared, and many websites include many webpages with very beautiful display effects and good user experience. These webpages use javascript, vbscript, jscript ( Above-mentioned javascript, vbscript, jscript all are the client dynamic script technologies such as client script language commonly used in the prior art, and these dynamic script technologies are widely used, also make original simple html (hypertext markup language) webpage become Very complex and very difficult to extract. [0003] The traditional web page information collection technology simulates http (hypertext ...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to overcome the defect that the traditional web page collection method in the prior art cannot collect the running results and content including the client dynamic script after running, and to provide a method that can collect and parse the webpage including the client Webpage parsing container for webpage with dynamic script and webpage parsing method realized by using said webpage parsing container

Images