Resource link acquisition method and device, electronic equipment and storage medium

An acquisition method and storage medium technology, applied in the field of devices, resource link acquisition methods, electronic equipment and storage media, can solve problems such as excessive consumption of computing resources and bandwidth resources

Active Publication Date: 2021-04-09
BEIJING TOPSEC NETWORK SECURITY TECH +2
View PDF12 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The purpose of the embodiments of the present application is to provide a resource link acquisition method, device, electronic device, and storage medium, which

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Resource link acquisition method and device, electronic equipment and storage medium
  • Resource link acquisition method and device, electronic equipment and storage medium
  • Resource link acquisition method and device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0026] Before introducing the resource link acquisition method provided by the embodiment of this application, some concepts involved in the embodiment of this application are first introduced:

[0027] A headless browser refers to a browser without a graphical user interface; a headless browser provides automatic control of web pages in an environment similar to popular web browsers, but performed through a command line interface or using network communications.

[0028] The WebDriver tool is an open source software. WebDriver can control different browsers (such as Firefox, Chrome, Safari, IE) by defining the driving engine. WebDriver can open URLs and interact with rendered pages; the goal of WebDriver is to provide A well-designed object-oriented application programming ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a resource link acquisition method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-processed webpage corresponding to an access link; searching for events existing in all document nodes in the to-be-processed webpage, and storing the events existing in the document nodes into a to-be-processed queue; simulating and triggering events in the to-be-processed queue in a multi-thread manner by using a headless browser; intercepting a resource request generated by the event in the to-be-processed queue in the triggering process, and obtaining a resource link in the resource request. In the implementation process, the events existing in the webpage are stored in the to-be-processed queue, and then the events in the to-be-processed queue are triggered and intercepted, so that the situations that the webpage jumps for multiple times, the page is rendered again, a new page pops up for multiple times and the like are effectively avoided; and meanwhile, excessive consumption of computing resources and bandwidth resources caused by the conditions is avoided, so that the computing resources and the bandwidth resources are effectively saved.

Description

technical field [0001] The present application relates to the technical fields of network security and network communication, and specifically relates to a resource link acquisition method, device, electronic equipment and storage medium. Background technique [0002] At present, in the process of using crawlers to grab resource links in public web pages, multiple page jumps, page re-rendering, and multiple pop-ups of new pages often occur, which cause the browser to run many unnecessary processes resources or thread resources, and at the same time, multiple loading and jump requests to obtain web pages will also cause a waste of bandwidth resources. Therefore, there is a problem of excessive consumption of computing resources and bandwidth resources in the process of using existing crawlers to grab resource links in public web pages. Contents of the invention [0003] The purpose of the embodiments of the present application is to provide a resource link acquisition meth...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/951G06F16/958H04L12/741H04L29/08H04L45/74
CPCG06F16/951G06F16/958H04L45/74H04L67/56H04L67/63
Inventor 熊毅
Owner BEIJING TOPSEC NETWORK SECURITY TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap