A resource link acquisition method, device, electronic equipment and storage medium

A technology for obtaining methods and resources, which is applied in the field of network security and network communication, and can solve problems such as excessive consumption of computing resources and bandwidth resources

Active Publication Date: 2021-09-14
BEIJING TOPSEC NETWORK SECURITY TECH +2
View PDF12 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The purpose of the embodiments of the present application is to provide a resource link acquisition method, device, electronic device, and storage medium, which are used to improve the problem of excessive consumption of computing resources and bandwidth resources in the process of grabbing resource links from public web pages

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A resource link acquisition method, device, electronic equipment and storage medium
  • A resource link acquisition method, device, electronic equipment and storage medium
  • A resource link acquisition method, device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0026] Before introducing the resource link acquisition method provided by the embodiment of this application, some concepts involved in the embodiment of this application are first introduced:

[0027] A headless browser refers to a browser without a graphical user interface; a headless browser provides automatic control of web pages in an environment similar to popular web browsers, but performed through a command line interface or using network communications.

[0028] The WebDriver tool is an open source software. WebDriver can control different browsers (such as Firefox, Chrome, Safari, IE) by defining the driving engine. WebDriver can open URLs and interact with rendered pages; the goal of WebDriver is to provide A well-designed object-oriented application programming ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present application provides a resource link acquisition method, device, electronic device, and storage medium, the method including: acquiring the webpage to be processed corresponding to the access link; finding the events of all document nodes in the webpage to be processed, and storing the event of the existence of the document node Store in the pending queue; use a headless browser to simulate triggering events in the pending queue in a multi-threaded manner; intercept resource requests generated during the triggering process of events in the pending queue, and obtain resource links in resource requests. In the above implementation process, by storing the events existing in the web page in the queue to be processed, and then triggering and intercepting the events in the queue to be processed, it effectively avoids multiple page jumps, page re-rendering, and multiple pop-ups. Pages, etc., while avoiding the excessive consumption of computing resources and bandwidth resources caused by these situations, thereby effectively saving computing resources and bandwidth resources.

Description

technical field [0001] The present application relates to the technical fields of network security and network communication, and specifically relates to a resource link acquisition method, device, electronic equipment and storage medium. Background technique [0002] At present, in the process of using crawlers to grab resource links in public web pages, multiple page jumps, page re-rendering, and multiple pop-ups of new pages often occur, which cause the browser to run many unnecessary processes resources or thread resources, and at the same time, multiple loading and jump requests to obtain web pages will also cause a waste of bandwidth resources. Therefore, there is a problem of excessive consumption of computing resources and bandwidth resources in the process of using existing crawlers to grab resource links in public web pages. Contents of the invention [0003] The purpose of the embodiments of the present application is to provide a resource link acquisition meth...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/951G06F16/958H04L12/741H04L29/08H04L45/74
CPCG06F16/951G06F16/958H04L45/74H04L67/56H04L67/63
Inventor 熊毅
Owner BEIJING TOPSEC NETWORK SECURITY TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap