Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data balancing method and system based on streaming processing platform

A data balancing and stream processing technology, applied in the field of computer technology applications, can solve problems such as cluster resource waste, load distribution strategy data skew, and computing units that cannot be dynamically allocated, and achieve the effect of improving processing efficiency

Active Publication Date: 2022-03-18
厦门美亚亿安信息科技有限公司
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In order to solve the technical problems of cluster resource waste in the operator allocation process in the prior art, data skew in the load allocation strategy, and computational units on the computing nodes that cannot be dynamically allocated during the execution process, the present invention proposes a stream-based The data balancing method and system of the type processing platform are used to solve the above-mentioned technical problems,

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data balancing method and system based on streaming processing platform
  • Data balancing method and system based on streaming processing platform
  • Data balancing method and system based on streaming processing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, rather than to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

[0032] It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0033] figure 1 A flowchart of a data balancing method based on a stream processing platform according to an embodiment of the present application is shown, as figure 1 As shown, the method includes the following steps:

[0034] S101: Divide the state data of each computin...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data balancing method and system based on a streaming processing platform, and the method comprises the steps: dividing the state data of each calculation unit into a plurality of independent and migratable execution blocks, and dynamically adjusting the execution blocks to idle nodes of a distributed cluster; the quasi real-time heartbeat obtains the computing power of the node and synchronizes the computing power of the node to the tuner; the execution block is subjected to data packaging, the packaged execution block is put into a tuner, and the structure of the packaged execution block comprises a message queue data source sub-library, a partition, operator information, the total amount of execution data, a corresponding data path, predicted execution time and a distribution node; and the tuner explores the resource information of each computing node, matches the computing node suitable for the execution block, and sends an instruction to the migration device to migrate the execution block. According to the method, efficient utilization of resources is achieved through balanced distribution of the data, and the data processing efficiency is improved.

Description

technical field [0001] The invention relates to the application field of computer technology, in particular to a data balancing method and system based on a stream processing platform. Background technique [0002] The computing model of a stream processing system is usually a DAG model. Each data processing job can be represented as a directed acyclic graph. In this directed acyclic graph, there are various data processing nodes, that is, operators. Operators will Allocation to each computing node in the cluster for calculation does not take into account the computing power of each node in the cluster and the overall running time of the DAG model during the allocation process, which often leads to a waste of cluster resources. [0003] The traditional load distribution strategy focuses on computing load, focusing on the balance between computing nodes from the perspective of computing load, ignoring the data balance between computing tasks. In streaming computing, due to d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
CPCG06F9/5088Y02D10/00
Inventor 庄晓明吴少华吴江煌宋峥晨吴琛华卢振业
Owner 厦门美亚亿安信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com