Configuration method for distributed statistical analysis system and distributed statistical analysis system

A statistical analysis and configuration method technology, applied in transmission systems, electrical components, etc., can solve problems such as underutilization of processing resources, unallocated statistical analysis tasks, etc., and achieve high stability, high scalability, and improved statistical efficiency Effect

Active Publication Date: 2017-07-07
九次方大数据信息集团有限公司
View PDF12 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the current distributed statistical system does not distribute statistical analysis tasks among device nodes according to the actual operation of processing resources.
As a result, processing resources may not be fully utilized

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Configuration method for distributed statistical analysis system and distributed statistical analysis system
  • Configuration method for distributed statistical analysis system and distributed statistical analysis system
  • Configuration method for distributed statistical analysis system and distributed statistical analysis system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0035] This embodiment provides a method for configuring a distributed statistical analysis system, wherein, such as figure 1 As shown, the distributed statistical analysis system is mainly composed of three parts: ZooKeeper cluster, service node and computing node cluster. The ZooKeeper cluster is used for state management of the computing node cluster. The service node is responsible for the decomposition and integration of statistical services and data update control. The service node receives the statistical analysis request from the front end and parses it into two parts: content search and statistical analysis. The tasks of the content search part are run by the content search engine, and the tasks of the statistical analysis part are run by the computing node cluster. Computing node clusters are used for data fragmentation and backup, splitting of computing tasks and merging of results, and load of computing tasks.

[0036] Explanation of terms: If there is no special...

Embodiment 2

[0123] This embodiment provides a distributed statistical analysis system, the system is configured by the configuration method in Implementation 1, and a corresponding module for executing the configuration method is obtained. Specifically include: cluster management module, data storage and migration module, statistical analysis query module and statistical task load sharing module (such as Figure 6 shown).

[0124] The cluster management module is used to establish and maintain the connection between each computing node and the ZooKeeper cluster, specifically for electing the overseer node (that is, the leader node) in the computing node cluster, and to shard data in each computing node according to the principle of data sharding, and Elect the leading shard (ie shard leader) in the copy of the data shard.

[0125] When the device status changes, the distribution of shard replicas needs to be adjusted. Device status changes mainly include two situations: device failure a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a configuration method for a distributed statistical analysis system. The distributed statistical analysis system comprises a ZooKeeper cluster, a service node and a computational node cluster. The method comprises the following steps of electing a leader node in the computational node cluster, fragmenting data in each computational node according to a data fragmentation principle and electing a leader fragment in a copy of the data fragment; after the service node receives a statistical analysis request, applying to the leader node for the computational node and making the leader node feed the computational node with the minimum task load back to the service node; after the service node obtains the feedback computational node, sending the statistical request to the computational node; and making the computational node search the leader fragment, apply to the leader fragment for an idle copy of the data fragment and distributing a statistical task to the copy of the data fragment to execute the statistical task. The invention also provides the distributed statistical analysis system based on the configuration method.

Description

technical field [0001] The present invention relates to a configuration method of a distributed statistical analysis system, in particular to a configuration method for configuring cluster management, data storage and migration, statistical analysis query, and statistical task load sharing functions, and the distributed statistics obtained by the configuration method analysis system. Background technique [0002] A distributed system is a computer system in which multiple processing resources are interconnected. These processing resources can also be called node devices, and execute the same role under unified control. For example, Chinese patent CN102497280 discloses a distributed system, which can realize mutual perception among multiple device nodes. Improved management efficiency. However, it does not disclose the management and configuration of specific execution tasks of each device node. [0003] Distributed systems usually need to have the function of statistical...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/1034H04L67/1001
Inventor 何毅荣龚朕郑建全
Owner 九次方大数据信息集团有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap