Transmission method and device of data

A transmission method and data technology, applied in the Internet field, can solve problems such as memory overflow, large amount of stored real-time log data, large storage space, etc., and achieve the effect of improving performance

Active Publication Date: 2018-05-25
BEIJING GRIDSUM TECH CO LTD
View PDF5 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The second type, FileChannel, is a channel established in the external storage of Flume; since the channel of the FileChannel type is located in the external storage, it has a large storage space and stores a large amount of real-time log data.
[0004] In the process of implementing the above invention, the inventor found that when using Flume in the prior art, in order to maintain the high efficiency of Flume transmission performance, the first type of MemoryChannel above can be used to process real-time log data; however, due to the limited server memory The number of real-time log data stored in MemoryChannel is limited. When the speed of receiving real-time log data is faster than the

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Transmission method and device of data
  • Transmission method and device of data
  • Transmission method and device of data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0058] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] The embodiment of the present invention provides a data transmission method, which is applied in the Flume system, such as figure 1 As shown, the method includes:

[0060] 101. Monitor the writing speed of data in the preset storage space and the moving speed of data.

[0061] The Flume system supports various types of data senders for collecting log data. After the log data is collected, the log data can be simply processe...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a transmission method and device of data, relates to the field of internet technology, and mainly aims to solve problems that in the prior art, a Flume system collects log data, and if MemoryChannel is adopted for storing the data, an overflow anomaly exists when speed of writing of the received log data is greater than speed of removing log data and overstocked data in theMemoryChannel are too much; and if FileChannel is adopted for storing the data, performance of log-data writing or removing is lower. The technical solution of the invention includes: monitoring writing speed of data in preset storage space and removing speed of data; respectively transferring data in the preset storage space to first storage space and second storage space if the writing speed ofthe data is greater than the removing speed of the data and data occupancy space in the preset storage space exceeds an overflow risk threshold value; and transmitting the data in the first storage space to an external memory, and transmitting the same to the second storage space by the external memory.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a data transmission method and device. Background technique [0002] With the rapid development of Internet technology, more and more users are more inclined to use the Internet for daily life. For example: the user plays a video through a video player or directly uses a web page, and so on. At present, during the video playing process, the video player or the webpage will record the real-time data of the currently playing video, and send the generated real-time log data to the server, so that the server can analyze the currently playing video in real time according to the real-time log data. Due to the large and scattered data volume of real-time log data, in specific applications, the server often collects real-time log data through Flume, among which Flume is a highly available, highly reliable, distributed massive log collection, aggregation and transmission platform s...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F3/06
CPCG06F3/0614G06F3/0646G06F3/067
Inventor 王晓涛
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap