A resource link acquisition method, device, electronic equipment and storage medium

Abstract

The present application provides a resource link acquisition method, device, electronic device, and storage medium, the method including: acquiring the webpage to be processed corresponding to the access link; finding the events of all document nodes in the webpage to be processed, and storing the event of the existence of the document node Store in the pending queue; use a headless browser to simulate triggering events in the pending queue in a multi-threaded manner; intercept resource requests generated during the triggering process of events in the pending queue, and obtain resource links in resource requests. In the above implementation process, by storing the events existing in the web page in the queue to be processed, and then triggering and intercepting the events in the queue to be processed, it effectively avoids multiple page jumps, page re-rendering, and multiple pop-ups. Pages, etc., while avoiding the excessive consumption of computing resources and bandwidth resources caused by these situations, thereby effectively saving computing resources and bandwidth resources.

Description

technical field [0001] The present application relates to the technical fields of network security and network communication, and specifically relates to a resource link acquisition method, device, electronic equipment and storage medium. Background technique [0002] At present, in the process of using crawlers to grab resource links in public web pages, multiple page jumps, page re-rendering, and multiple pop-ups of new pages often occur, which cause the browser to run many unnecessary processes resources or thread resources, and at the same time, multiple loading and jump requests to obtain web pages will also cause a waste of bandwidth resources. Therefore, there is a problem of excessive consumption of computing resources and bandwidth resources in the process of using existing crawlers to grab resource links in public web pages. Contents of the invention [0003] The purpose of the embodiments of the present application is to provide a resource link acquisition meth...

AI Technical Summary

Problems solved by technology

[0003] The purpose of the embodiments of the present application is to provide a resource link acquisition method, device, electronic device, and storage medium, which are used to improve the problem of excessive consumption of computing resources and bandwidth resources in the process of grabbing resource links from public web pages

Images