Streaming data processing system and method

A streaming data and processing system technology, applied in the computer field, can solve the problems of no API interface to access data streams, inability to dynamically perceive back-end pressure and elastic scaling, etc., to achieve the effect of flexible data processing

Active Publication Date: 2020-07-17
GEOVIS CO LTD
View PDF6 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003]However, the above-mentioned Kubernetes-based system has the following defects: it cannot dynamically sense the back-end pressure and elastically scale; there is no API interface to access the data stream

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Streaming data processing system and method
  • Streaming data processing system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] In order to understand the above-mentioned purpose, features and advantages of the present invention more clearly, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0030] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described here. Therefore, the protection scope of the present invention is not limited by the specific details disclosed below. EXAMPLE LIMITATIONS.

[0031] The present invention improves on the basis of the existing Kubernetes platform, adds a front-end data access program to access the data stream and performs classification processing, realizes...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a streaming data processing system and method, and the system comprises a leading system which is used for receiving data, forwarding the data to a Kafka platform, and transmitting a notification to a scheduling system; the Kafka platform is used for storing the data forwarded by the leading system in a pre-distributed topic; the scheduling system is used for generating a scheduling notice according to the information of the Kafka platform and the state information in the etcd system and sending the scheduling notice to the Kubernetes platform after receiving the noticeof the leading system; the etcd system is used for updating and storing the state information according to the report of the Kubernetes platform; and the Kubernetes platform is used for controlling, generating or adjusting pods according to the scheduling notification and consuming the data in the corresponding topics. According to the invention, the unified API interface for receiving the data flow is realized, and the scheduling system can be started, closed, expanded and contracted in a targeted manner according to the load condition of the container, so that elastic and flexible data processing is realized.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a streaming data processing system and method. Background technique [0002] As an open source container orchestration engine, Kubernetes is used to manage containerized applications on multiple hosts in the cloud platform. It provides a mechanism for application deployment, planning, updating, and maintenance. The goal is to make deploying containerized applications simple and efficient. efficient. In a Kubernetes-based system, the client sends the Deployment to the ApiServer, and the ApiServer receives the client's request and stores the resource content in the database (etcd); the Controller component (including scheduler, replication, and endpoint) monitors resource changes and responds; ReplicaSet Check the database changes and create the expected number of pod instances; the Scheduler checks the database changes again and finds the Pods that have not been assigne...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08H04L12/24
CPCH04L41/0896H04L67/60
Inventor 毕俊张敬亮傅伟康马银龙王千一王晓东杨光辉张家旺
Owner GEOVIS CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap