A distributed data storage method and system

Abstract

The invention provides a distributed data storage method and system. Storage units of data are data blocks with given length, the data is organized to be of a tree structure, and each writing operation request on the data corresponds to one node of the tree structure. The method comprises the steps that a writing operation request is received, wherein the writing operation request comprises to-be-written data length; the node, corresponding to the writing operation request, in the tree structure is determined; when the to-be-written data length does not exceed a remaining space of the data block where the node corresponding to the writing operation request is located, the data block is locked, and writing operation is executed; and otherwise, one or more new data blocks are allocated according to the to-be-written data length, a subtree with the node being a root is transferred to the allocated new data blocks, a link is set in the data block where the node is originally located to indicate the position of the subtree, the new data blocks are locked, and the writing operation is executed.

Description

technical field [0001] The present invention relates to the technical field of computers and computer software, in particular to a distributed data storage method and system. Background technique [0002] Today, with the booming e-commerce business, more and more operators use e-commerce as their important sales channel, or even the only sales channel. More consumers obtain all kinds of daily necessities they need through e-commerce to meet their daily needs and improve their quality of life. Demand promotes production, and strong demand prompts the emergence of more and better products and their sales activities. Therefore, how to better manage and process these product data has become a problem that e-commerce companies have to pay attention to. [0003] Judging from the data released by e-commerce at present, the existing active product data is already in the billions. Adding the historical data, this will be a very large amount of data. Traditional data storage methods ...

AI Technical Summary

Problems solved by technology

[0007] 1. The main disadvantage of traditional database RDBMS is that the small size (million level) data volume is already the limit of its processing, and the larger data volume will make its processing performance drop sharply

However, this sub-database and sub-table scheme can only slightly alleviate the problem. There are problems such as the underlying storage structure being opaque to users. What is more serious is that connection queries across multiple data nodes and distributed transactions have become a real problem or even a problem. Impossible (like MySql);

[0008] 2. NoSql is inherently designed for processing big data, but NoSql has its specific applicable scenarios. For example, for hbase, it only supports random read with Row Key as the keyword. Obviously, this cannot satisfy the diversified read of e-commerce business Requirements; for MongoDB, it stores data in the form of documents. When the data increases and more space needs to be applied, the performance will degrade rapidly;

[0009] 3. More importantly, traditional relational databases and NoSql do not consider the life cycle of data

Traditional data solutions (including relational databases and emerging NoSql solutions) do not take into account the life cycle of data, useless garbage data, historical data and hot data (according to the vast software system development practice, from the perspective of data use, a small amount The active data will be frequently and repeatedly accessed in a short period of time, including reading, updating, etc. (we call this part of data hot data) and store them together, which leads to a great waste of system space and affects the system based on The use of various acceleration techniques for data pre-reading

When data surges, the phenomenon of degradation in access efficiency is very surprising

Images