Distributed data processing method and system

A technology for distributed data and processing systems, applied in the distributed field, can solve the problems of reducing the data processing efficiency and shortcomings of distributed database systems, and achieve the effects of reducing network IO, improving efficiency, and avoiding short-board effects.

Active Publication Date: 2017-04-19
CHINA UNITED NETWORK COMM GRP CO LTD
View PDF5 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] A large number of network input and output (IO) of distributed database systems come from cross-node data processing requirements. In this cross-node data processing, the delay of any node will greatly reduce the data processing efficiency of distributed database systems. resulting in a short-board effect

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data processing method and system
  • Distributed data processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] In order to make the purpose, technical solution and advantages of the present invention more clear, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

[0034] The steps shown in the flowcharts of the figures may be performed in a computer system, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

[0035] The inventor found that the distribution and application of telecommunication business data has its own characteristics. Firstly, the management mode of telecommunication enterprises is based on the branch responsibility system based on geographical...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed data processing method and system. The method comprises the following steps: a service agent unit divides a task requesting processing into a plurality of sub-tasks, each of which only relates to a calculation task in one dimension; and a data node carries out data processing of a received sub-task in a corresponding network card. By dividing data nodes by different dimensions to process data of different dimensions respectively, the method reduces the network IO between the data nodes, improves the data processing efficiency across the data nodes, and thereby avoids the short board effect.

Description

technical field [0001] The invention relates to distributed technology, in particular to a distributed data processing method and system. Background technique [0002] In a distributed database system, data needs to be scattered and stored on multiple data nodes. Multiple data nodes have the ability to calculate and load data. Through distributed algorithms, the speed of database query or calculation in specific scenarios can be accelerated. However, for scenarios such as cross-node data analysis, because in the cross-node query or calculation, the operation of each data node must be completed before the results of the query or calculation can be combined, which will reduce the efficiency of data processing, thus forming a certain the bottleneck. For example: Take the query of users whose packages expire within one month in Shanghai as an example. Since the data is irregularly stored on 9 data nodes, it is necessary to query the users whose packages expire within one month ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2471G06F16/27
Inventor 郭志斌张云勇雷磊陈晓明
Owner CHINA UNITED NETWORK COMM GRP CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap