Unlock instant, AI-driven research and patent intelligence for your innovation.

A distributed dynamic scheduling method and system for large-scale network data acquisition

A technology of dynamic scheduling and network data, applied in the direction of resource allocation, program startup/switching, multi-program installation, etc., can solve the problems of upgrading, incompatibility, difficulty in collecting and upgrading, etc.

Inactive Publication Date: 2019-05-28
INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF6 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this has led to difficulties in collection upgrades, and each collection upgrade needs to be adapted to the scheduling
At the same time, when a new news source is added or a new collector is added, it is very likely that there will be incompatibility
Unable to perform flexible deployment upgrades

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A distributed dynamic scheduling method and system for large-scale network data acquisition
  • A distributed dynamic scheduling method and system for large-scale network data acquisition
  • A distributed dynamic scheduling method and system for large-scale network data acquisition

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the method and system for sensing and acquiring large-scale network data proposed by the present invention will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described here are only used to explain the present invention, and are not intended to limit the present invention.

[0028] The following terms are included in the specification of the present invention:

[0029] "Information source" refers to the source of Internet information; in the framework of "media-information cluster-information source", "media" refers to different information dissemination media in cyberspace such as news, forums, blogs, news APP, Weibo, WeChat, social media, etc. "Information cluster" refers to a collection of specific network data of a type of media, such as Sina News website in news websi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a distributed dynamic scheduling method for large-scale network data acquisition. The method comprises the steps of obtaining an information source where network data is located; registering the nodes participating in the data acquisition as acquisition nodes or scheduling nodes; acquiring a scheduling strategy of data acquisition; generating an acquisition task accordingto the scheduling strategy and the information source information of the information source; Transmitting the collection task to a collector of the collection node so as to configure and start the collector; And executing the collection task through the collector to obtain a collection result. The distributed scheduling method is a universal scheduling method irrelevant to a collector and an information source, supports multiple heterogeneous collectors, supports heterogeneous nodes, and supports hot plug and dynamic expansion of the collection nodes and the collector.

Description

technical field [0001] The invention belongs to the field of data perception and acquisition, in particular to a distributed dynamic task scheduling system for large-scale collection of network data. Background technique [0002] The scheduling management of distributed collection tasks is the core component of distributed collection technology. [0003] With the development of the Internet, the amount of data in the network continues to increase. Correspondingly, the gradual reduction of computing resources makes distributed collection a trend of Internet data collection. [0004] However, the development of the Internet has not only brought about an increase in the amount of data, but also the diversity of data carriers, not only traditional web data, but also streaming data such as Weibo, Toutiao, and mobile phones. Various carriers such as applications. This has led to the failure of the traditional general collection framework. It is necessary to analyze and design a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/50
Inventor 孟剑俞晓明程学旗史存会郭岩贺广福周秀花余智华刘悦
Owner INST OF COMPUTING TECH CHINESE ACAD OF SCI