Unlock instant, AI-driven research and patent intelligence for your innovation.

Big data consistency comparison method and system

A data comparison and consistency technology, applied in the computer field, can solve problems such as low efficiency, and achieve the effects of improving comparison efficiency, solving consistency comparison, and efficient consistency comparison

Active Publication Date: 2016-04-13
SHENZHEN TENCENT COMP SYST CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, traditional comparison tools can usually only achieve consistent comparison of small data volumes, but the efficiency is very low when comparing large data volumes (PB-level data volumes)

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data consistency comparison method and system
  • Big data consistency comparison method and system
  • Big data consistency comparison method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0028] A big data consistency comparison method provided by the embodiment of the present invention can be applied to such as figure 1 in the system shown. refer to figure 1 As shown, the big data A and big data B that need to be compared for consistency are the data in the computer cluster. For example, they can be respectively the big data before and after the replacement when the system of the big data cluster is The calculation results before optimization and the calculation results after optimization corresponding to the algorithm optimization can also be respectively the data before migrati...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a big data consistency comparison method and system. The method includes: converting first to-be-compared data into first structured data and converting second to-be-compared data into second structured data; utilizing a distributed parallel computing framework to subject the first structured data and the second structured data to hierarchical comparisons including comparison of data statistic information in the first structured data and the second structured data and comparison of content in the first structured data and the second structured data in different hierarchies; if the data statistic information in the first structured data is different from the data statistic information in the second structured data, directly returning a result of comparison inconsistency; if the data statistic information in the first structured data is as same as that in the second structured data, returning a result of comparison consistency. By the method and system, efficient consistency comparison of disorderly big data can be realized.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a big data consistency comparison method and system. Background technique [0002] Big data, also known as massive data, refers to data with a data volume of more than PB (PB refers to petabyte, which is a relatively advanced storage unit, which is 2 to the 50th power byte). Due to the huge amount of data, big data cannot be captured, managed, processed, and sorted into information that can help enterprises make more positive business decisions within a reasonable time through the current mainstream software tools. It usually requires thousands or even tens of thousands of computers to pass through Computer clusters (that is, big data clusters) composed of network connections that jointly complete specific data storage and computing tasks for processing. [0003] With the advent of the era of big data, the value of big data has been developed, and the application and processin...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/2365G06F16/258
Inventor 徐天华贺波梁栋蔡伟岗张宝亮
Owner SHENZHEN TENCENT COMP SYST CO LTD