Distributed file acquisition monitoring method

A collection and monitoring, distributed technology, applied in the direction of electrical components, digital transmission systems, transmission systems, etc., can solve the problems of few developers, few documents and use, etc., and achieve the effect of strong compatibility

Active Publication Date: 2018-11-23
上海新炬网络技术有限公司
View PDF4 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] Facebook Scribe: 1. Developed in C++, relatively

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed file acquisition monitoring method
  • Distributed file acquisition monitoring method

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0023] The present invention will be further described below in conjunction with the drawings and embodiments.

[0024] figure 1 It is a flowchart of a method based on distributed file collection and monitoring in an embodiment of the present invention.

[0025] See figure 1 , The distributed file collection and monitoring method provided by the present invention includes the following steps:

[0026] S1: Use Mysql cluster and Redis cluster to build the server, and deploy WEBAPP on the server;

[0027] S2: WEBAPP provides task definition, each type of business or each file path is a task;

[0028] S3: Save the task in the server, scan the file information and save it to the server;

[0029] S4: The client obtains task information by reading the Redis cluster each time, and records the file read and write information at the same time for task processing;

[0030] S5: When the task starts, the monitoring program checks the integrity of the collection process. If it finds that the service is...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed file acquisition monitoring method, comprising the following steps: S1, constructing a server by using a Mysql cluster and a Redis cluster, and arranging a WEBAPPat the server; S2, providing a task definition by the WEBAPP; S3, saving the task in the server; S4, a client acquiring task information by reading the Redis cluster every time, and processing the task; S5, the task beginning, a monitoring program checking the integrity of the acquisition process, and performing compensation acquisition; S6, the monitoring program recording a ftp login status anda network status; S7, the client acquiring data and storing the state in a remote storage cluster; and S8, the WEBAPP controlling the downloading, uploading or stopping of the acquisition task. The distributed file acquisition monitoring method implements decentralization, and does not have primary nodes or child nodes, so that the acquisition application can be extended at any time; the providedWEBAPP visualized operation can be used intuitively and conveniently; and the Docker can allow the coexistence of acquisition of multiple versions without conflicts and has strong compatibility.

Description

technical field [0001] The invention relates to a file collection method, in particular to a distributed file collection and monitoring method. Background technique [0002] In the period when big data is becoming more and more important, data acquisition is the most indispensable part of big data development / analysis. In the early stage, hardware or network equipment is required for basic data collection, and data files collected by various manufacturers or equipment need a centralized data center for transmission or maintenance. Due to the diversity and complexity of documents, a stable and monitorable document collection and monitoring method needs to be specially designed to liberate labor costs and quickly locate the collection process and the cause of errors. [0003] At the same time, when the amount of data is increasing, it is necessary to ensure the stability and scalability of the method, the horizontal expansion of the acquisition method, and the need for self-t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/08H04L12/26H04L12/24
CPCH04L41/06H04L43/12H04L67/025H04L67/06H04L67/10H04L67/1097
Inventor 程永新谢涛梁超
Owner 上海新炬网络技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap