Distributed data batch processing system and method

A distributed data and batch processing technology, applied in the field of data processing, can solve the problems of waste of resources and the inability of server resource utilization to reach the highest level at the same time.

Active Publication Date: 2016-08-31
BANK OF CHINA
View PDF5 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Different servers are isolated from each other and do not communicate, which will cause the server r

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data batch processing system and method
  • Distributed data batch processing system and method
  • Distributed data batch processing system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0076] In order to make the object, technical solution and technical effect of the present invention more clear and complete, the specific implementation manners of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0077] figure 1 It is a schematic diagram of the architecture of the distributed data batch processing system provided by the embodiment of the present invention. Such as figure 1 As shown, the distributed data batch processing system architecture is a cluster system architecture, and its physical structure includes a master node 11 and a plurality of service nodes 12. TCPIP protocol or database DB13 can be used for communication and communication between the master node 11 and service nodes 12. Data collection and sharing. When using the TCPIP protocol for communication and data collection and sharing, the service node 12 directly transmits the data to the master node 11 through the TCPIP protocol.

[0078] ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention discloses a distributed data batch processing system and method. In the system and method, a task scanning module in a main node can calculate expendable weight of each task, a task scheduling distribution module in the main node can calculate and obtain specific task expendable weight distribution proportion among tasks distributed to each service node according to the task expendable weight in a single batch task queue and queue length description and expendable weight rates in service node queues, and the specific task expendable weight distribution proportions can enable sources of each service node to be fully utilized and basically uniformize the time consumed by processing the task distributed to each service node by the service node. Accordingly, tasks are distributed to each service node according to the specific task expendable weight distribution proportions, the sources of the service nodes can be fully utilized when the service nodes are processing the tasks, resource waste is prevented, and each service node can basically process the distributed task at the same time.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a distributed data batch processing system and method. Background technique [0002] With the expansion of the company's business and the construction of new application systems, the data in the enterprise information system is increasingly concentrated, and the amount of data is increasing rapidly. For the batch processing of data, the demand for statistical analysis is also emerging. On the other hand, in order to ensure the normal operation of banks and other enterprises, the time window for batches is limited. That is, in order to ensure normal business operations, data can only be processed within a specified time period. This phenomenon is called the tidal character of the data. [0003] In order to deal with the above two problems, there is an urgent need for a data processing system with a faster processing rate. [0004] In the prior art, in order to speed up the proces...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50G06F9/48
CPCG06F9/4843G06F9/5083
Inventor 罗焱学王宝义安莉
Owner BANK OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap