Method, device and system for processing mass data
A massive data processing and data technology, applied in electrical digital data processing, special data processing applications, transmission systems, etc., can solve the problems of data copy rollback, one-time write failure, affecting the efficiency of massive data processing, etc., to achieve processing performance reduced effect
Inactive Publication Date: 2013-01-30
杭州勒卡斯广告策划有限公司
View PDF0 Cites 14 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] In view of this, the present application provides a massive data processing method, device, and system to solve the problem that the system needs to perform multiple writes when performing one-time write processing on multiple data copies that store massive data at the same time in the prior art. input operation, which reduces the processing efficiency of massive data, and is used to solve the existing massive data processing scheme. Once a write fails, all data copies are rolled back, which obviously reduces the processing performance of the system for massive data. Technical issues that seriously affect the processing efficiency of massive data
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0057] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.
[0058] Embodiment 1 of the present application provides a massive data processing method, the method is applied to data nodes storing massive data in a preset distributed data replica cluster, the data replica cluster includes at least one data node, refer to figure 1 , which shows a schematic structural diagram of the data replica cluster 101 , the data replica cluster includes n data nodes: data node 111 , data node 112 , and data node 113 . . . data node n. Each data node in t...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a method, a device and a system for processing mass data. The method for processing mass data is applied to data nodes storing mass data in data copy clusters. A data node obtains different data of another data node which is connected with the data node at a preset time interval, and writes the obtained data in the another data node to trigger the another data node to update a data version of the another data node. The method comprises that the current data node receives data to be updated and writes the data to be updated in the data node, and the data to be updated are extracted in update request information received by a data copy cluster; a data version of the current data node is updated; and if current system time achieves the preset time interval, the current data node obtains the data which are different from the another data node in the current data node and writes the obtained data in the another data node to trigger the another data node to update a data version of the another data node. By means of the method, the device and the system for processing mass data, the mass data processing efficiency is improved.
Description
technical field [0001] The present application relates to the field of software technology, in particular to a massive data processing method, device and system. Background technique [0002] Massive data generally refers to data with a data volume of more than PB level. The processing of massive data generally adopts a distributed processing scheme, that is, the massive data is fragmented and distributed to different nodes for processing. In order to ensure data security, a data redundancy scheme is usually adopted, that is, a piece of data exists on multiple nodes , at this time, the redundant data is called a data copy. [0003] At present, when performing a write process on massive data, the system needs to perform a write operation on each data copy that stores the massive data, that is, send the data to be updated to each data copy, and each data copy writes the data to be updated Into this copy of the data to ensure the consistency of massive data. For example, the...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F17/30H04L29/08
Inventor 李晨马向晖
Owner 杭州勒卡斯广告策划有限公司