Web data acquisition method, web server and web data acquisition system based on docker

A data acquisition system and data acquisition technology, applied in the direction of digital transmission system, transmission system, data exchange network, etc., can solve the problems of unsatisfactory scalability of Web data acquisition system, save task allocation time, improve robustness and performance. Universality, speed-up effect

Active Publication Date: 2019-08-13
SHANDONG UNIV
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The present invention can effectively solve the problem of unsatisfactory scalability of the Web data collection system existing in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Web data acquisition method, web server and web data acquisition system based on docker
  • Web data acquisition method, web server and web data acquisition system based on docker
  • Web data acquisition method, web server and web data acquisition system based on docker

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. The Docker of the present invention is a lightweight virtual machine.

[0037] figure 1 It is a flow chart of Embodiment 1 of the Web data collection method based on Docker of the present invention, and the method is completed in the Web server, such as figure 1 As shown, it specifically includes the following steps:

[0038] Step 1: Create a mirror container based on Docker, and construct a data collection master node and several data collection work nodes from the mirror container; the data collection master node communicates with the data collection work nodes.

[0039] Step 2: The data collection master node receives the Web data collection task, and starts a pre...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Docker-based Web data collection method, a Web server and a Web data collection system, wherein the Web data collection method is completed in the Web server, including creating a mirror container based on Docker, and constructing a data collection master node from the mirror container and several data collection work nodes; the data collection master node communicates with the data collection work nodes; the data collection master node receives the Web data collection task, and starts a preset amount of data collection according to the number of URLs in the Web data collection task Work nodes; Web data collection tasks include data source IP addresses and URLs; after data collection work nodes are started, the data collection master node assigns data source IP addresses and URLs to each data collection work node, and the data collection work nodes collect corresponding Web Data; the data collection master node receives the data sent by each data collection work node, and recycles all data collection work nodes to complete the collection of Web data.

Description

technical field [0001] The invention belongs to the field of Internet Web data processing, and in particular relates to a Docker-based Web data collection method, a Web server and a Web data collection system. Background technique [0002] With the rapid development of network technology, the Internet has become the main carrier of information, and fully and effectively extracting this information is the focus and difficulty of today's Internet information collection work. Data acquisition technology emerges at the historic moment, which can focus on solving the problem of extracting key information from data sources. At present, large Internet companies and related research institutions at home and abroad have provided some relatively mature solutions, and some have been put into use. However, most of these solutions are realized by establishing a master node and deploying a fixed number of working nodes. Extremely unstable in terms of resource utilization. [0003] As we...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08H04L12/24
CPCH04L41/06H04L67/1031H04L67/1001
Inventor 边俊峰钱进闵新平郭伟崔立真
Owner SHANDONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap