Data acquisition and transmission method and device and electronic equipment

A technology of data acquisition and transmission method, applied in the field of data processing, can solve problems such as poor reliability, and achieve the effect of alleviating poor reliability and improving stability

Active Publication Date: 2020-09-11
华青融天(北京)软件股份有限公司
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a data collection and transmission method, device and electronic equipment to alleviate the technical problem of poor reliability in the data collection and transmission method in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data acquisition and transmission method and device and electronic equipment
  • Data acquisition and transmission method and device and electronic equipment
  • Data acquisition and transmission method and device and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0029] figure 1 A flow chart of a data acquisition and transmission method is shown, such as figure 1 As shown, the method specifically includes the following steps:

[0030] Step S12, count the total number of bytes of data collected by the source component in the Flume system within a preset time period.

[0031] Conventional Flume data collection frameworks such as figure 2As shown, Flume uses agent as the smallest independent running unit. An agent is mainly composed of three components: source component, channel component and sink component. The source component is mainly responsible for collecting data from the data source and placing the collected data in the channel component. , the channel component simply caches the data, the sink component obtains data from the channel component, and sends the obtained data to an external storage device, where the source component encapsulates each piece of data into an event during the process of collecting data A message, an e...

Embodiment 2

[0070] An embodiment of the present invention further provides a data acquisition and transmission device, which is mainly used to execute the data acquisition and transmission method provided in the above-mentioned first embodiment.

[0071] Figure 4 is a functional block diagram of a data acquisition and transmission device provided by an embodiment of the present invention, such as Figure 4 As shown, the device mainly includes: a statistics module 10, a judgment module 20, a first control module 30, and a transmission module 40, wherein:

[0072] The statistics module 10 is configured to count the total number of bytes of data collected by the source component in the Flume system within a preset time period.

[0073] The judgment module 20 is used for judging whether the sum of the number of bytes exceeds a preset threshold.

[0074] The first control module 30, if yes, controls the source component to send an event message to the channel component in the Flume system, ...

Embodiment 3

[0094] see Figure 5 , an embodiment of the present invention provides an electronic device, the electronic device includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used to execute executable modules, such as computer programs, stored in memory 61 .

[0095] The memory 61 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), which may use the Internet, a wide area network, a local area network, a metropolitan area network, and the like.

[0096] The bus 62 may be an ISA bus, a PCI bus, an EISA bus, or the l...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data acquisition and transmission method, a data acquisition and transmission device and electronic equipment, which relate to the technical field of data processing and comprise the following steps of: counting the sum of byte numbers of data acquired by a source component in a Flume system in a preset time period; when the total number of bytes exceeds a preset thresholdvalue, controlling the total number of bytes of an event message sent to the channel component by the source component within a preset time period to be not greater than the preset threshold value; and enabling the sink component to obtain the event message from the channel component, and send the obtained event message to an external storage device. The method limits the sum of bytes of event messages flowing to a channel componentto achieve the effect of controlling the flow of the data sent by the sink component to the outside, the stability of the service system of the server is improved,and the technical problem of poor reliability of a data acquisition and transmission method in the prior art is effectively alleviated.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a method, device and electronic device for data acquisition and transmission. Background technique [0002] People have generated a large amount of data due to excessive use and reliance on Internet-related technologies. Enterprises can analyze these data and obtain data value from it. Collecting and using data is difficult. In the prior art, Flume (log collection system) is often used as a data collection framework for data collection and transmission. However, in the process of data collection and transmission, the response of the server-side business system often times out or even the server is down. [0003] To sum up, the data acquisition and transmission method in the prior art has the technical problem of poor reliability. SUMMARY OF THE INVENTION [0004] The purpose of the present invention is to provide a data acquisition and transmission method, dev...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/26H04L12/813H04L29/06H04L47/20
CPCH04L43/12H04L43/16H04L43/0876H04L47/20H04L63/0428
Inventor 蔡伟培
Owner 华青融天(北京)软件股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap