A system and method for distributing data in a big data platform

A big data platform and data distribution technology, applied in the field of data security, can solve the problems of non-linear relationship between the I/O speed acceleration ratio and the number of server nodes, long time spent on data consistency, and slow I/O performance. The effect of improving throughput, ensuring structure, and ensuring accuracy and correctness

Active Publication Date: 2021-07-09
亿阳安全技术有限公司
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] 1. In the case of a large amount of data, especially in the case of continuous writing of a large amount of data, the I / O performance is relatively slow, and the I / O speed acceleration ratio is not linearly related to the number of server nodes;
[0004] 2. In the processing of unstructured and semi-structured data such as LOG, BLOG, video, and social relationship information, it is not optimized according to the type and characteristics of big data storage, and the processing speed is slow;
[0005] 3. The multi-service synchronous writing technology is used, which leads to a long synchronization time when the network and storage device conditions are unknown, which makes it take a long time to process data consistency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A system and method for distributing data in a big data platform
  • A system and method for distributing data in a big data platform
  • A system and method for distributing data in a big data platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0050] An exemplary embodiment of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown, it is understood that the present disclosure can be implemented in various forms without limitation. Instead, these embodiments are provided to be more thoroughly understood to disclose the present disclosure, and can communicate the scope of the present disclosure to those skilled in the art.

[0051] According to an embodiment of the present invention, a system that distributes data in a large data platform, such as attached figure 1 As shown, the system specifically includes: large data distribution unit M101, data bus module M102, management center M103 and large data adaptation module M104; one,

[0052]The large data distribution unit is configured to receive massive data sent by a plurality of clients and store it in its own cache; obtain data distribution rules fr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The system and method for distributing data in the big data platform of the present invention adopts asynchronous I / O as the technical basis to build a big data distribution unit for high-speed distribution of big data, adopts the separation of server and client threads to improve the throughput of data distribution, The complete structure of the data is guaranteed by the multi-dimensional structure storage unit, and the accuracy and correctness of the data distribution are guaranteed by the big data management center and the data bus module, so that all parts can cooperate at high speed without waiting for each other for resources, and can fully utilize resource. At the same time, the whole system shows good scalability.

Description

Technical field [0001] The present invention relates to the field of data security, and more particularly to a method and system for distributing data in a large data platform. Background technique [0002] In the prior art, the large data platform based on Hadoop architecture has high scalability, high reliability, and high-capacity. At present, a large number of data queries and data flows are widely used in memory databases (Memory DB) and non-relational database technology (NOSQL), and cache technology (cache) has achieved good progress. However, in the actual large data service processing, such as wireless application protocol Internet log, large user mail system, blog log analysis, user information tracking and analysis, etc., current large data platforms on data I / O processing methods There is a problem of defects, especially for unstructured, semi-structural, large data volume, and I / O treatment speeds have a serious problem, which is mainly reflected in: [0003] 1. ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08
CPCH04L67/02H04L67/06H04L67/562H04L67/55H04L67/568H04L67/60
Inventor 周伟俞力赵贵阳周春楠
Owner 亿阳安全技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap