Efficient data processing system

Abstract

The invention discloses an efficient data processing system. The system comprises a data processing module, a data storage module and a data communication module, wherein the data communication module is used for transmitting data between the data processing module and the data storage module; the data processing module comprises a data caching sub-module; and the data caching sub-module is used for storing data with the processing frequency higher than a predetermined frequency threshold. The invention aims to provide the efficient data processing system which can efficiently perform data screening, reduce IO rate difference between a memory and a disk, increase a data transmission pipeline quantity, increase data transmission pipeline bandwidth, and improve shared storage processing capability and redundancy protection capability.

Description

technical field [0001] The invention belongs to the technical field of high-efficiency data processing system design, and in particular relates to a high-efficiency data processing system capable of improving data processing speed. Background technique [0002] In the field of databases, the traditional hardware systems generally used to process massive data are mostly in two modes: stand-alone (using local storage), stand-alone or cluster plus shared storage (single storage device). Before the high-efficiency data processing system is submitted for use, various components need to be integrated, including cabinets, power supplies, node machines, storage, and networks. The hardware system currently used for data processing services has high requirements for the use environment and personnel, such as dedicated computer rooms, dedicated refrigeration equipment, independent power systems, networks, etc., and at the same time brings many inconveniences to personnel, such as manag...

AI Technical Summary

Problems solved by technology

The hardware system currently used for data processing services has high requirements for the use environment and personnel, such as dedicated computer rooms, dedicated refrigeration equipment, independent power systems, networks, etc., and at the same time brings many inconveniences to personnel, such as management, skills, Brand compatibility etc.

[0003] In the existing technology, it is usually necessary to read a large amount of data from the shared storage to the server memory, and there are many bottlenecks in this process, such as the large difference between the server memory and the disk IO rate, the number of data transmission pipelines and the bandwidth. Satisfy the large amount of data transmission, whether there is a single point of failure in the shared storage, and whether there is a single point of insufficient performance in the shared storage

Images