Data collection and transmission method, device and electronic equipment

A technology of data acquisition and transmission method, applied in the field of data processing, can solve problems such as poor reliability, and achieve the effect of alleviating poor reliability and improving stability

Active Publication Date: 2022-02-01
华青融天(北京)软件股份有限公司
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a data collection and transmission method, device and electronic equipment to alleviate the technical problem of poor reliability in the data collection and transmission method in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data collection and transmission method, device and electronic equipment
  • Data collection and transmission method, device and electronic equipment
  • Data collection and transmission method, device and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0029] figure 1 A flow chart showing a data collection and transmission method, such as figure 1 As shown, the method specifically includes the following steps:

[0030] Step S12, counting the total number of bytes of data collected by the source component in the Flume system within a preset time period.

[0031] Conventional Flume data acquisition frameworks such as figure 2As shown, Flume uses agent as the smallest independent operating unit. An agent is mainly composed of three components: source component, channel component and sink component. The source component is mainly responsible for collecting data from the data source and placing the collected data in the channel component. Among them, the channel component simply caches the data, the sink component obtains data from the channel component, and sends the acquired data to an external storage device, and the source component encapsulates each piece of data into an event during the process of collecting data Messag...

Embodiment 2

[0070] The embodiment of the present invention also provides a data collection and transmission device, the data collection and transmission device is mainly used to implement the data collection and transmission method provided in the first embodiment above, and the data collection and transmission device provided by the embodiment of the present invention will be described in detail below.

[0071] Figure 4 It is a functional block diagram of a data collection and transmission device provided by an embodiment of the present invention, as shown in Figure 4 As shown, the device mainly includes: a statistical module 10, a judgment module 20, a first control module 30, and a sending module 40, wherein:

[0072] The statistical module 10 is used to count the total number of bytes of data collected by the source component in the Flume system within a preset time period.

[0073] A judging module 20, configured to judge whether the sum of the bytes exceeds a preset threshold.

...

Embodiment 3

[0094] see Figure 5 , the embodiment of the present invention provides an electronic device, the electronic device includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used to execute executable modules, such as computer programs, stored in memory 61 .

[0095] Wherein, the memory 61 may include a high-speed random access memory (RAM, RandomAccessMemory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local network, metropolitan area network, etc. can be used.

[0096] The bus 62 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be d...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data collection and transmission method, device and electronic equipment, and relates to the technical field of data processing, including counting the sum of the bytes of data collected by the source component in the Flume system within a preset time period; When the sum of the numbers exceeds the preset threshold, the total number of bytes of the event message sent by the source component to the channel component within the preset time period is not greater than the preset threshold; the sink component obtains the event message from the channel component and will get the The event message is sent to the external storage device. This method achieves the effect of controlling the data flow sent by the sink component externally by limiting the total number of bytes of the event message flowing to the channel component, improves the stability of the server business system, and effectively alleviates the data problem in the prior art. The technical problem of poor reliability in the collection and transmission method.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data collection and transmission method, device and electronic equipment. Background technique [0002] People have generated a large amount of data due to excessive use and reliance on Internet-related technologies. Enterprises can analyze these data and obtain data value from them. Collecting and using data is difficult. In the existing technology, Flume (log collection system) is mostly used as the data collection framework for data collection and transmission, but in the process of data collection and transmission, the response of the server business system often times out or even the server goes down. [0003] To sum up, the data acquisition and transmission method in the prior art has the technical problem of poor reliability. Contents of the invention [0004] The object of the present invention is to provide a data collection and transmission method,...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L43/12H04L43/16H04L43/0876H04L47/20H04L9/40
CPCH04L43/12H04L43/16H04L43/0876H04L47/20H04L63/0428
Inventor 蔡伟培
Owner 华青融天(北京)软件股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap