System and method for ensuring eventual consistency of mass data caches

Abstract

The invention discloses a system and a method for ensuring the eventual consistency of mass data caches. The system disclosed by the invention comprises a log monitoring module and a data cache eventual consistency batch-processing task module, wherein the log monitoring module is used for monitoring a binary data operation log of a practical data storage module and extracting change data records from the binary data operation log; the change data records comprise data records before and after changing; each data record is composed of a plurality of data columns; the data cache eventual consistency batch-processing task module is used for receiving the change data records, splitting the change data records into a plurality of change data columns according to columns, deleting unchanged data columns, only reserving changed data columns, combining the changed data columns by utilizing a reducer, and finally correcting the changed data columns; duplication of the change data records of the practical data storage module in a target data cache module is realized; and mass data cache consistency is realized. The system disclosed by the invention has high concurrency, batch-processing, high availability and the like; and the data cache consistency is realized by mapping, reducing and correcting batch-processing tasks.

Description

technical field [0001] The invention relates to a data cache consistency method, in particular to a system and method for ensuring the final consistency of mass data caches. Background technique [0002] Mass data cache consistency refers to the process of ensuring that the actual data storage is consistent with the target data cache during the actual data storage change process. In order to meet the challenges brought by users’ high concurrent requests for massive data and solve the big data access and extreme transaction bottlenecks faced by traditional relational databases, the massive data cache consistency method has become an important means to improve the performance of new applications that carry big data. Massive data usually has a large amount of data and a high rate of data change, making it difficult to achieve strong consistency in data caching. [0003] On the other hand, according to the theory of consistency / availability / partition fault tolerance, in a distr...

AI Technical Summary

Problems solved by technology

[0009] The above methods have different defects in the scope of application: the efficiency of single-task batch execution is the lowest, and the cache consistency time of massive data is longer; compared with multi-task parallelism, multi-thread concurrency occupies less system resources and high CPU utilization, but Poor reliability, deadlock of a single thread will affect the entire process

[0010] Although the improved method of dividing the changed data record into multiple fragments for parallel execution can be used for massive data cache consistency, it cannot stably meet the real-time requirements. The performance is as follows: the processing delay is proportional to the length of the data fragment and the overhead of the initial processing task; A segment with a small length will reduce latency, increase additional overhead, and increase the complexity of segment dependencies; a segment with a large length will increase delay

Images