A monitoring system and method for multi-link real-time data stream transmission

A real-time data and monitoring system technology, applied in the field of big data transmission, can solve the problems of low sensitivity of the monitoring system, accurate statistics, inability to collect data per minute and the number of landing records, etc.

Active Publication Date: 2020-11-24
SHANGHAI HODE INFORMATION TECH CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] At present, the industry pays more attention to big data integration in real-time data transmission, but in data monitoring, it only relies on counter statistical data to report, and it is impossible to collect data per minute and accurately count the number of landing records, resulting in monitoring system The sensitivity is not high, and it is not easy to quickly perceive and solve online problems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A monitoring system and method for multi-link real-time data stream transmission
  • A monitoring system and method for multi-link real-time data stream transmission
  • A monitoring system and method for multi-link real-time data stream transmission

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] For a large number of data collection tasks (such as tens of billions of data collection per day), it is necessary to set up multiple nodes in the entire data flow to carry different responsibilities through a layered architecture. It is very difficult to count data loss in such a real-time stream. The data is sent from the client to the final landing link is very long. It is unknowable whether the data has been lost, because the data will be considered lost only when the data has not landed in the end (maybe hours or days). The so-called "loss" mainly means that there is a delay in the internal transmission of data. Under normal circumstances, the data will be processed normally.

[0039] For this reason, the data loss strategy defined in the present invention is: when a piece of data is received by a certain gateway node, if it is not written into other frameworks within a set time (such as within 5 minutes), it is considered that this piece of data is lost . Simult...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a multi-link real-time data stream transmission monitoring system and method. The method comprises the following steps: sequentially arranging a plurality of monitoring pointsin a data integration component that performs real-time transmission on data streams based on a layered architecture; according to the receiving time of any piece of data at a first monitoring point,adding a timestamp corresponding to the receiving time of the piece of data; respectively making a statistics of the number of data flowing through the monitoring point according to set minute-levelintervals by each monitoring point through a counter, and correspondingly recording the data in a plurality of partitions matched with the number of minute-level intervals; and correspondingly comparing the statistical value of each monitoring point according to the partitions, and when the statistical values of the counters of any two monitoring points in the same partition are inconsistent, judging whether data transmitted between the two monitoring points are lost or not. The multi-link real-time data stream transmission monitoring method provided by the invention can realize accurate statistics in real-time transmission of big data, and find the number of data lost in the current system in a minute level, so that the monitoring system can quickly respond to problems and give an alarm.

Description

technical field [0001] The invention relates to the field of big data transmission, in particular to a monitoring system and method for multi-link real-time data stream transmission. Background technique [0002] The rapid development of big data has spawned a huge ecosystem. A simple summary of big data scenarios can be roughly divided into: big data integration, big data distributed storage, big data distributed computing, big data visualization, and big data applications. AI (artificial intelligence), data mining, machine learning, recommendation and other upper-layer applications are all based on the current rich big data ecology, and the computing scenarios also range from CPU, to GPU, to TPU. [0003] As the basic field of big data integration, it is necessary to gather a large amount of raw data from different sources into the same platform for processing by upper-level modules. For the integration of big data, the main solutions in the industry now include Flume, K...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L12/26H04L12/24
CPCH04L41/06H04L43/0829H04L43/0852H04L43/106H04L43/16
Inventor 叶强盛
Owner SHANGHAI HODE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap