A system and method for distributing data in a big data platform

Abstract

The system and method for distributing data in the big data platform of the present invention adopts asynchronous I / O as the technical basis to build a big data distribution unit for high-speed distribution of big data, adopts the separation of server and client threads to improve the throughput of data distribution, The complete structure of the data is guaranteed by the multi-dimensional structure storage unit, and the accuracy and correctness of the data distribution are guaranteed by the big data management center and the data bus module, so that all parts can cooperate at high speed without waiting for each other for resources, and can fully utilize resource. At the same time, the whole system shows good scalability.

Description

Technical field [0001] The present invention relates to the field of data security, and more particularly to a method and system for distributing data in a large data platform. Background technique [0002] In the prior art, the large data platform based on Hadoop architecture has high scalability, high reliability, and high-capacity. At present, a large number of data queries and data flows are widely used in memory databases (Memory DB) and non-relational database technology (NOSQL), and cache technology (cache) has achieved good progress. However, in the actual large data service processing, such as wireless application protocol Internet log, large user mail system, blog log analysis, user information tracking and analysis, etc., current large data platforms on data I / O processing methods There is a problem of defects, especially for unstructured, semi-structural, large data volume, and I / O treatment speeds have a serious problem, which is mainly reflected in: [0003] 1. ...

AI Technical Summary

Problems solved by technology

[0003] 1. In the case of a large amount of data, especially in the case of continuous writing of a large amount of data, the I / O performance is relatively slow, and the I / O speed acceleration ratio is not linearly related to the number of server nodes;

[0004] 2. In the processing of unstructured and semi-structured data such as LOG, BLOG, video, and social relationship information, it is not optimized according to the type and characteristics of big data storage, and the processing speed is slow;

[0005] 3. The multi-service synchronous writing technology is used, which leads to a long synchronization time when the network and storage device conditions are unknown, which makes it take a long time to process data consistency

Images