Multicenter data task processing method, multicenter data task processing device and multicenter data task processing system

A task processing and multi-center technology, applied in the direction of electrical digital data processing, special data processing applications, multi-programming devices, etc., can solve the problem of inapplicable cross-data center data calculation, redundant data storage in multiple places, and inability to form a unified view and other issues to achieve intuitive and accurate task allocation, reduce pressure, and avoid massive migration and redundancy

Active Publication Date: 2013-06-19
CHINA MOBILE COMM GRP CO LTD
View PDF5 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] 1. The existing dynamic allocation algorithm is suitable for calculations that are irrelevant to data or not related to massive data, but not suitable for calculation scenarios with a huge amount of data, which will cause a large amount of data migration and data redundancy;
[0012] 2. The existing fixed allocation method is usually divided by geographical application, and is usually based on long-term statistical information. It is difficult to ensure optimal allocation, which will lead to redundant data storage in multiple places, large data transmission volume, repeated calculations, and inability to A series of issues such as forming a unified view;
[0013] 3. The method of merging based on the client is not suitable for calculations with a huge amount of data;
[0014] 4. The distribution and parallel processing of data calculation based on a single center is not suitable for data calculation across data centers because it does not consider data synchronization and data migration between data centers

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multicenter data task processing method, multicenter data task processing device and multicenter data task processing system
  • Multicenter data task processing method, multicenter data task processing device and multicenter data task processing system
  • Multicenter data task processing method, multicenter data task processing device and multicenter data task processing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0075] Such as figure 1 As shown, the embodiment of the multi-center data task processing method of the present invention includes:

[0076] Step 102, collect the information of the data task, including: the source of the data processed each time for each data processing task, the amount of single-processed data of each data source, the processing time of the processing task, and the calculation amount of the processing task, which can be used The CPU time of the data task is used as the basis for measurement, and the cost (cost) of the database statistics can also be used as the basis;

[0077] Step 104, generate a data task model graph, wherein, the nodes in the data task model graph are each data task, and generate node information of each node and connection relationships (edges) between nodes, and the edges are nodes The calculation relationship is the data flow direction;

[0078] The node information includes: the single calculation output data volume v of the data ta...

Embodiment 2

[0118] Such as Figure 9 As shown, another embodiment of the multi-center data task processing method of the present invention includes:

[0119] Step 202, collecting the information of the data task

[0120] Step 204, when monitoring whether the imbalance degree of multiple data centers is greater than the preset threshold, if yes, execute step 206 and step 208, if not, return to step 202;

[0121] Step 206, alarm;

[0122] Step 208, generating a data task model diagram;

[0123] Step 210, simplifying the model diagram;

[0124] Step 212, cutting the model diagram;

[0125] Step 214 , deploy the grouped data tasks to corresponding data centers, and then return to step 202 .

[0126] Specifically, there are several ways to monitor the imbalance of multiple data centers:

[0127] (1) Obtain the current task processing capacity and the maximum task processing capacity of each data center, and calculate the load degree of each data center, that is, divide the current task p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a multicenter data task processing method, a multicenter data task processing device and a multicenter data task processing system. The processing method comprises the steps: A, collecting data task information; B, generating a data task model graph according to the data task information, cutting the data task model graph, namely grouping all nodes in the data task model graph, wherein each node of the data task model graph is one data task; and C, allocating data tasks of different groups into corresponding data centers. By means of the multicenter data task processing method, the device and the system, through the establishment of a model reflecting the correlations among data tasks, multicenter task allocation is performed based on the model, and thus the task allocation is visual, accurate and balanced. Transfer and redundancy of large amount of data are avoided, data volume transmitted among each center is small, therefore pressure of network load is reduced, overall data processing efficiency is improved, and loads in all the data centers are basically the same.

Description

technical field [0001] The invention relates to a business support technology, in particular to a multi-center data task processing method, device and system. Background technique [0002] For security reasons, enterprises usually establish two or more data processing centers to achieve disaster recovery. In daily life, task allocation is used to distribute the load among data processing centers. Each data processing center handles different tasks and stores different data, especially complex intermediate data that is not directly oriented to applications. However, due to the complex relationship between data, often a data processing needs to rely on multiple data sources, which is very likely to be the result of data processed by another data center. In this way, the data needs to be exported from a certain center, then transmitted through the network, and then imported to A computing center is required, and this process consumes a lot of time, which greatly reduces the ov...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/46G06F9/50G06F17/30
Inventor 尚晶袁向阳孙少陵
Owner CHINA MOBILE COMM GRP CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap