Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and data processing device

A data processing and streaming data technology, applied in the field of data processing, can solve the problems of slow startup and slow data loading, and achieve the effect of improving the speed and the startup speed.

Active Publication Date: 2015-07-01
ALIBABA GRP HLDG LTD
View PDF4 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The main purpose of this application is to provide a data processing method and device to solve the problem of slow start-up caused by slow data loading at start-up in the distributed flow computing system existing in the prior art, wherein:

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and data processing device
  • Data processing method and data processing device
  • Data processing method and data processing device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] The main idea of ​​this application is that in the distributed flow computing system, the intermediate data generated by each computing node is written into the primary data table and the secondary data table of the database with different keys. When the distributed system is restarted, it can Find and load the intermediate data corresponding to each node in the secondary data table with the corresponding key, so that the speed of loading data can be improved. Moreover, according to the solution of the present application, the corresponding intermediate data of each computing node can be loaded immediately when the system is started, so the solution has wide applicability and is not limited by application scenarios.

[0022] The technical solution of this application can be applied to distributed stream computing systems, refer to Image 6 , the distributed flow computing system 600 may include one or more computing nodes 610-1, . . . , 610-i, . In the process of data ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data processing method and a data processing device. The data processing method includes: subjecting received streaming data to streaming data processing through one or more computational nodes; taking a streaming data processing result as intermediate data to store in a main data list and an auxiliary data list of a database; when the one or more computational nodes restart, loading the corresponding intermediate data of the computational nodes for the computational nodes from the auxiliary data list according to node identifications of the computational nodes, and continuing subjecting streaming data received subsequently to streaming data processing on the basis of the intermediate data. By adoption of the technical scheme, query of the intermediate data corresponding to each computational node in starting of a distributive stream computing system can be accelerated, and accordingly data loading speed is increased to further increase starting speed of the distributive stream computing system.

Description

technical field [0001] The present application relates to the field of data processing, and in particular to a data processing method and device in a distributed stream computing system. Background technique [0002] During the operation of a distributed flow computing device, a large amount of intermediate process calculation data is usually stored in the memory. This part of the data is essential for calculating the final result data. Therefore, the intermediate process calculation data is generally persisted during operation. to the disk in case the device is interrupted due to various reasons and restarts. For the storage of computing data in the middle process of distributed stream computing, traditional relational databases are an option. However, traditional relational databases are not suitable for storing massive amounts of data. When the amount of stored data reaches more than 100 million, most of the traditional The query performance of the relational database wi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 刘健男
Owner ALIBABA GRP HLDG LTD