A large-scale data processing method and system

Abstract

The present invention provides a large-scale data processing system and method, wherein the system includes: a flow collection subsystem and a flow processing subsystem; the flow collection subsystem is used to collect data flow, and mirror the collected data flow , and divide the obtained mirrored traffic into P-way sub-traffic and send it to the traffic storage cluster in the traffic processing subsystem, where P is an integer greater than 1; the traffic storage cluster is composed of M storage servers, and each storage server has Hang N disks, M is a positive integer, N is an integer greater than 1, and M×N≥P; each storage server receives the sub-traffic distributed, and uses load balancing technology to write the sub-traffic distributed to N disks attached to it. In this way, the pressure of continuous writing on the disk is reduced, and the problem of large-scale data storage is better solved.

Description

【Technical field】 [0001] The invention relates to computer network technology, in particular to a large-scale data processing method and system. 【Background technique】 [0002] With the continuous expansion of network users, the amount of data on the Internet has grown explosively, and people have a new understanding of network transmission speed, data security and reliability. User data is widely distributed in many places. For users, data storage and backup that are not well managed pose hidden dangers to business operations. The speed and quality of data transmission affect user experience. In addition, with the development of cloud services Gradually rising and popularizing, the processing needs of large-scale data storage, statistics or analysis have become urgent problems to be solved. However, the existing data processing systems and methods cannot meet the processing requirements of large-scale data due to the impact of performance. For example, if the existing data...

AI Technical Summary

Problems solved by technology

User data is widely distributed in many places. For users, data storage and backup that are not well managed pose hidden dangers to business operations. The speed and quality of data transmission affect user experience. In addition, with the development of cloud services With the gradual rise and promotion, the processing needs of large-scale data storage, statistics or analysis have become urgent problems to be solved

However, the existing data processing systems and methods cannot meet the processing requirements of large-scale data due to the impact of performance. For example, if the existing data processing systems and methods are directly applied to the storage of large-scale data, it will bring unbearable Data read and write pressure

Images