Webpage data capturing method and device, storage medium and equipment

Abstract

The embodiment of the invention provides a webpage data capturing method and device, a storage medium and equipment. The method comprises the steps that after a target webpage is successfully logged in through a headless browser, if the headless browser monitors an AJAX request of the target webpage, authorization authentication information carried in the monitored AJAX request is stored in a cache through the headless browser; the authorization authentication information is read from the cache through the data capture script, and the read authorization authentication information is added intoan access request for capturing webpage data; and webpage data returned by the server is captured after passing the authentication based on the authorization authentication information through the data capture script and based on the access request containing the authorization authentication information. Therefore, the webpage data can be effectively captured.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and equipment for capturing web page data. Background technique [0002] With the rapid development of Internet information technology, more and more websites have emerged. Corresponding further processing can take place based on the data captured from the website. For example, data services such as data analysis and intelligent recommendation can be provided by capturing relevant data of the website for analysis and processing. [0003] At present, in large-scale web application clusters, Single Sign On (SSO) is often used as a user authentication method. This authentication method allows users to log in only once with the same account in multiple application systems. , you can access other mutually trusted application systems. However, under such a complex login process, current web crawlers such as Scrapy (crawling) framework...

AI Technical Summary

Problems solved by technology

However, under such a complex login process, current web crawlers such as Scrapy (crawling) frameworks have very weak ability to redirect (redirect) between different domain names, and it is inconvenient to handle cookies under different domain names (stored on the user's local terminal) data) sharing problem, when encountering some content that requires user authentication and authorization or triggering the anti-robot detection of the website, it will cause the web crawler to stop grabbing web page data

Images