Edge computing cloud environment distributed Web page extraction and analysis system and method

Abstract

The invention belongs to the technical field of network communication. Cloud computing has security risks such as data loss and leakage, sharing technical vulnerabilities, unsafe application program interfaces and the like, the webpage crawling efficiency is low, a large amount of time is required for subsequent vulnerability scanning, the invention provides an edge computing cloud environment distributed Web page extraction and analysis system and method. The central management node schedules the computing node to complete the work task according to the historical crawling efficiency, the reasonable allocation of resources is improved, and the crawling strategy and depth are analyzed and deduplicated; on the premise that a vulnerability scanning result is correct, the crawling speed of awebsite is effectively increased, transverse expansion of scanning capacity and reasonable utilization of computing resources are facilitated, higher transmission and response speed is brought by edgecomputing, the problem of computing resource waste of nodes in a traditional cloud computing system is solved, the productivity ratio is increased, and the resource utilization rate is obviously increased.

Description

technical field [0001] The present invention relates to the technical field of network communication, and more specifically, relates to a cloud environment distributed Web page extraction and analysis system and method for edge computing. Background technique [0002] Based on cloud computing technology, the main job function is to provide daily automated office support services for relevant work departments and related staff, comprehensively empower social development, improve intelligent management capabilities, improve work execution efficiency, integrate urban planning, and improve service efficiency. Efficiency; the cloud computing software service model has the characteristics of integrating software and hardware resources, lower client requirements and a unified maintenance platform. Applying the cloud computing model to the construction of the e-government platform can maximize the sharing of data resources and save construction and operation Cost, increase the load ...

AI Technical Summary

Problems solved by technology

The web vulnerability scanning methods in the prior art basically use scanning tools or hardware devices to scan the crawled website vulnerabilities. For common government affairs office systems (the average page files exceed 20,000), the processing speed is too low, generally from It will take hours or even days from scanning to the end of the analysis, so the premise of scanning each website needs to crawl all the pages under the website

Images