Data transmission method and system based on Kafka component

A data transmission method and a data transmission system technology, applied in the field of communication, can solve problems such as packet loss, memory overflow, and affecting normal system functions, and achieve the effects of meeting data quality requirements, increasing stability, and no memory pressure

Pending Publication Date: 2021-10-29
BEIJING INST OF RADIO MEASUREMENT
View PDF19 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Among them, the traditional real-time data transmission system piles up buffers to temporarily store data when the amount of received data exceeds the processing capacity, which is likely to cause packet loss or memory overflow, affecting the normal function of the system; due to data delays in non-real-time transmission systems, many data applications cannot be supported. Inability to make full use of data
[0004] Therefore, the existing data transmission methods cannot satisfy the stable transmission of large batches of data.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data transmission method and system based on Kafka component
  • Data transmission method and system based on Kafka component

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] The principles and features of the present invention will be described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0023] Kafka is a distributed publish-subscribe messaging system written in the Scala language and later became part of the Apache project. The Kafka message system is mainly composed of Producer, Broker and Consumer. The Producer is responsible for generating messages and sending them to the Broker. The Broker temporarily stores the messages and then forwards them to the Consumers. The Consumers receive and process the messages.

[0024] Based on this, the present application proposes a data transmission method based on Kafka components, which will be further described below.

[0025] Such as figure 1 As shown, the process provided for the embodiment of the data transmission method of the present invention, t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data transmission method and system based on a Kafka component, and relates to the technical field of communication. The method comprises the steps: a data acquisition device acquires original data to be transmitted; the data processing device extracts key information of the original data; the Producer component sends the original data and the key information to different themes of the Broker component of the corresponding type for caching according to the type of the original data; and the user terminal obtains the key information from the corresponding theme of the Broker component by using the Consumer component, and judges whether to obtain the original data or not according to the key information. The method is suitable for efficient and high-quality transmission of a large amount of data, the real-time requirement during large-scale data transmission can be met, key information extraction is equivalent to efficient simplification of original data, the data quality requirement is met, and the system stability is improved due to shortening of a data transmission link.

Description

technical field [0001] The present invention relates to the technical field of communications, in particular to a data transmission method and system based on Kafka components. Background technique [0002] With the development of large data-volume experimental environments and higher-performance sensors, a large amount of data needs to be processed and transmitted. [0003] Traditional data transmission systems are divided into two types, real-time data transmission and non-real-time data transmission. Among them, the traditional real-time data transmission system piles up buffers to temporarily store data when the amount of received data exceeds the processing capacity, which is likely to cause packet loss or memory overflow, affecting the normal function of the system; due to data delays in non-real-time transmission systems, many data applications cannot be supported. Data cannot be fully utilized. [0004] Therefore, the existing data transmission methods cannot satis...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/06H04L67/34H04L67/568
Inventor 崔迪段晓杰张磊
Owner BEIJING INST OF RADIO MEASUREMENT
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap