Data storage system, metadatabase synchronization method and data cross-domain calculation method

A data storage system and database technology, applied in the field of information processing, can solve the problems of high-cost investment, instability, and inability of data to flow out, and achieve the effect of improving performance and reducing network overhead.

Active Publication Date: 2019-05-07
TRANSWARP INFORMATION TECH SHANGHAI
View PDF10 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Each data center is equivalent to a domain. The network within the domain is fast, but the network between domains is much slower and unstable than the network within the domain. Therefore, if a large amount of network overhead is generated during joint computing, There will be a relatively large performance problem
[0003] At present, there are strongly consistent and scalable global distributed databases on the market. The above-mentioned global distributed databases have two main defects. One is that it requires high-cost investment, and the other is that it does not meet the data compliance requirements. Regulatory requirements, that is, the requirement that data in a data center cannot be outflowed to other data centers
[0004] The reason for the first defect is that in

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data storage system, metadatabase synchronization method and data cross-domain calculation method
  • Data storage system, metadatabase synchronization method and data cross-domain calculation method
  • Data storage system, metadatabase synchronization method and data cross-domain calculation method

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0035] Example one

[0036] figure 1 This is a structural diagram of the data storage system provided by Embodiment 1 of the present invention, such as figure 1 As shown, the data storage system includes: at least two data centers (as an example and not a limitation, in figure 1 Three data centers are shown in, namely: data center A, 110, data center B, 120, and data center C, 130).

[0037] Among them, a communication connection is established between different data centers (for example, a public network or a private network is used for connection). Typically, each data center uses a distributed database system as a whole.

[0038] Each data center (in figure 1 Taking data center A and 110 as an example) including: access layer 1101, compilation layer 1102, computing layer 1103, storage layer 1104, and underlying container cloud platform 1105;

[0039] The compilation layer 1102 includes a metadata database 11021 and at least one compilation node 11022 (in figure 1 Taking three compi...

Example Embodiment

[0065] Example two

[0066] figure 2 This is a flowchart of a method for synchronizing a metadata database provided in the second embodiment of the present invention. This embodiment is applicable to the case of performing data synchronization on the metadata stored in each data center in the data storage system according to the embodiment of the present invention. The method can be executed by the metadata database synchronization device provided by the embodiment of the present invention. The device can be implemented in software and / or hardware, and can generally be integrated in a data storage system by one or more compilation layers in the data storage system. Execution, for example, is executed in cooperation with each server integrated with the compilation layer of each data center in the data storage system.

[0067] Such as figure 2 As shown, the method of the embodiment of the present invention includes:

[0068] S210. Determine a master database and a slave database amo...

Example Embodiment

[0077] Example three

[0078] Figure 3a This is a flowchart of a method for synchronizing a metadata database provided in the third embodiment of the present invention. This embodiment is optimized on the basis of the above-mentioned embodiment. In this embodiment, the main database is determined among all metadata databases in the data storage system. And from the database, and the operation of establishing the cascading topology map between the meta-databases is embodied, correspondingly, the method of the embodiment of the present invention specifically includes:

[0079] S310. In all metadata databases included in the data storage system, count the communication delay between the two metadata databases.

[0080] In this embodiment, all metadata databases included in the data storage system are connected in pairs. Correspondingly, the test information can be sent and received between the two-element databases to calculate the communication delay between the two-element databases...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data storage system, a metadatabase synchronization method and a data cross-domain calculation method. The data storage system comprises at least two data centers, wherein each data center comprises an access layer, a compiling layer, a computing layer, a storage layer and a bottom container cloud platform; The access layer is used for providing a unified data access interface; The compiling node is used for inquiring the metadatabase according to the received SQL statement, generating a matched execution plan and distributing the job task to the computing node for execution according to the execution plan; The computing nodes are used for acquiring data from data nodes of a data center where the computing nodes are located for computing according to the operationtasks and sending computing results to the computing nodes serving as summarizing nodes; The data node is used for storing data; And the bottom container cloud platform is used for carrying out containerization management on all services of the same data center. According to the technical scheme provided by the embodiment of the invention, the input cost is saved while the cross-domain computingservice is provided with high quality, and the data compliance requirement is met.

Description

technical field [0001] Embodiments of the present invention relate to information processing technologies, and in particular to a data storage system, metadata database synchronization, and data cross-domain computing methods. Background technique [0002] With the increasing amount of data and the need for business expansion, more and more enterprises have begun to deploy their own data centers. Because of the needs of some special industries, sometimes it is necessary to combine the data of multiple enterprises or organizations for joint computing, that is to say, it is necessary to solve the problem of data joint computing across data centers. Each data center is equivalent to a domain. The network within the domain is fast, but the network between domains is much slower and unstable than the network within the domain. Therefore, if a large amount of network overhead is generated during joint computing, There will be a relatively large performance problem. [0003] At p...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/27G06F16/25
Inventor 李光跃边雨刘汪根
Owner TRANSWARP INFORMATION TECH SHANGHAI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap