Unlock instant, AI-driven research and patent intelligence for your innovation.

Big data transit transmission synchronization and storage method

A big data and data technology, applied in the field of big data transfer, transmission, synchronization and storage

Active Publication Date: 2015-11-11
G CLOUD TECH
View PDF5 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] With the rapid development of information technology and the explosive growth of data information, the traditional data storage and processing mode has been difficult to meet the needs of high-speed and sensitive response in today's era, and the application of distributed and virtualization technology has brought solutions At the same time as the solution, other requirements and problems were also raised, and the transmission of big data between distributed systems has also become a key issue

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data transit transmission synchronization and storage method
  • Big data transit transmission synchronization and storage method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] See figure 1 , 2 As shown, the method described in the present invention includes methods for transferring and synchronizing large data between different network domains, concurrent transmission of massive data, data integrity verification and retransmission strategies, distributed file storage, and RESTful interface definition of HDFS use. Between distributed clusters in different network domains, a node with a public IP in a network is used as a name node to store the mapping information and block information of other data nodes in the local area network; HDFS is selectively built in the network domain where the name node is located. Transit server storage system; between client clusters, through communication with the transit server, according to the specific network environment, the transit server's storage system is used as a springboard to transmit the required massive big data. The HDFS interface used for data transmission is published in RESTful style as a var...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention belongs to the data transmission and synchronization technical field and relates to a big data transit transmission synchronization and storage method applied to different network domains. The method includes the following steps that: among distributed clusters of different network domains, a node with a public IP in a network is adopted as a name node, and node mapping information and block information of the other data nodes located in a local area network are stored; an HDFS is selectively built in the network domain where the name node is located and is adopted as a storage system of a transit server; among client clusters, a large quantity of required big data can be transmitted based on communication with the transit server, according to specific network environment and with the storage system of the transit server adopted as a springboard; HDFS interfaces used in data transmission are distributed in a RESTful style and are realized through the varieties of a webHDFS; and data integrity verification mechanisms are built in data nodes of the transit server, so that the integrity and security of the data can be ensured. The big data transit transmission synchronization and storage method of the invention can be applied to the transmission and synchronization of cross-domain distributed mass data.

Description

technical field [0001] The invention relates to the technical field of data transmission and synchronization, in particular to a large data transmission synchronization and storage method between different network domains. Background technique [0002] With the rapid development of information technology and the explosive growth of data information, the traditional data storage and processing mode has been difficult to meet the needs of high-speed and sensitive response in today's era, and the application of distributed and virtualization technology has brought solutions At the same time as the solution, other requirements and problems were raised, and the transmission of big data between distributed systems has also become a key issue. How to solve fast, safe, and stable cross-network transmission has become a hot spot of concern, and webHDFS, which appears in the Hadoop distributed file system, gives a possibility to solve this problem. Contents of the invention [0003...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/10H04L67/1095H04L67/1097H04L67/568
Inventor 郭宇艾建文季统凯
Owner G CLOUD TECH