High-availability and elastic expansion system and method for distributed database
By using heterogeneous resource pools and dynamic storage strategies, the problem of ineffective utilization of enterprise computing equipment in distributed database systems is solved, achieving elastic data expansion and high availability, avoiding overload of core storage nodes, and ensuring that data is not easily lost in the event of node failure.
Patent Information
- Application Number
- CN202511727668.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-02-24
AI Technical Summary
Existing distributed database systems typically use high-performance servers and dedicated server storage devices, which cannot effectively utilize the large number of personal and business computing devices present in enterprises. This causes the dedicated server storage devices to become unusable when storing massive amounts of data.
A heterogeneous resource pool, including storage nodes and compute nodes, is adopted. The resource management module monitors the node status in real time, and the data sharding and scheduling module dynamically selects storage nodes, giving priority to edge storage nodes and storing excess data on core storage nodes. Through the periodic management strategy of the data sharding and scheduling module, elastic data expansion is achieved.
It enables elastic scaling of data, avoids overload of core storage nodes, improves the flexibility and availability of data storage, and ensures that data is not easily lost in the event of node failure.
Smart Images
Figure CN121560237A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed databases, specifically to a highly available and elastically scalable system and method for distributed databases. Background Technology
[0002] With the development of technology, the demand for integrating distributed databases and storage technologies is increasing. Integrating distributed databases and storage technologies can significantly improve the performance of distributed databases. Currently, in solutions that integrate distributed databases and storage technologies, data-related content in the distributed database, such as redo logs, is directly mapped and stored on storage nodes.
[0003] However, existing distributed systems typically use high-performance servers and dedicated server storage devices to store data, which cannot effectively utilize the large number of personal and business computing devices in enterprises. When there is a massive amount of data to be stored, the dedicated server storage devices will fill up directly, causing them to freeze and become unusable. Therefore, they do not meet the current needs. To address this, we propose a highly available and elastically scalable system and method for distributed databases. Summary of the Invention
[0004] The purpose of this invention is to provide a highly available and elastically scalable system and method for distributed databases, in order to solve the problem mentioned in the background art that distributed systems typically use high-performance servers and dedicated server storage devices to store data, which cannot effectively utilize the large number of personal and commercial computing devices in enterprises. Once there is a large amount of data to be stored, the dedicated server storage devices will directly fill up and become unusable.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a high availability and elastic scaling system for distributed databases, including a heterogeneous resource pool for storing various data from various databases. The heterogeneous resource pool includes storage nodes and computing nodes. The computing nodes are used to analyze and process the input data, and the storage nodes are used to store the analyzed data.
[0006] The resource management module is used to register, discover and monitor the status information of all nodes in the heterogeneous resource pool in real time. The status information includes storage space, computing load and network health.
[0007] Data sharding is used to create replicas of data within storage nodes in different heterogeneous resource pools.
[0008] The scheduling module uses the status information provided by the resource management module to dynamically select and allocate storage nodes for each data shard in the heterogeneous resource pool;
[0009] The data management module is used to maintain the mapping relationship between data shards and storage nodes;
[0010] The storage nodes include core storage nodes and edge storage nodes. Edge storage nodes prioritize storing data, while core storage nodes store excess data.
[0011] Preferably, the core storage node includes a central data processing library, server-dedicated storage, and data storage devices; the edge storage node includes commercial data computers and personal data terminals; and the computing node includes commercial servers, personal data servers, and secondary servers.
[0012] Preferably, the data within the data sharding and scheduling module is allocated using a message digest algorithm. The scheduling module prioritizes storing data in edge storage nodes and stores excess data in core storage nodes.
[0013] Preferably, each data shard creates a replica on at least two different storage nodes. When the resource management module detects a failure in any storage node, it automatically reads, writes, and copies the data on the storage node where the replica is located, and then migrates the copied data to a healthy storage node.
[0014] Preferably, the data sharding and scheduling module performs a periodic management strategy based on the data access frequency and performance requirements when allocating data.
[0015] Preferably, the periodic management strategy includes: prioritizing the allocation of frequently accessed data to the commercial server and server-dedicated storage; allocating infrequently accessed data to the commercial data computer and secondary server; and archiving rarely accessed data to the data storage device, personal data server, or personal data terminal.
[0016] Preferably, the resource management module maintains communication with all storage nodes and computing nodes. When the resource management module detects the addition of a new storage node or computing node, it automatically includes it in the heterogeneous resource pool. When a storage node or computing node is detected to be faulty and offline, it marks it as unavailable and initiates a data repair process.
[0017] The data repair process includes: the resource management module obtains the resource status of all healthy nodes and creates new replicas of the resources of the healthy nodes; selects one or more optimal storage and compute nodes for the new replicas; and the main data processing library sends a replication instruction to a healthy storage and compute node to transfer the new replica data to the healthy storage and compute node.
[0018] The personal data terminal and personal data server can also be set as edge computing nodes to execute edge computing tasks issued by the general data processing library.
[0019] A method for a highly available and elastically scalable system for a distributed database, the method comprising the following steps:
[0020] S1: The main data processing library discovers and registers all available nodes in the heterogeneous resource pool and receives data write requests;
[0021] S2: When the resource management module detects that more than 10TB of data has entered the system within a day, the scheduling module divides the data to be stored into multiple data fragments.
[0022] S3: The data sharding and scheduling module performs calculations to preferentially select an edge storage node from the heterogeneous resource pool for each data shard for storage;
[0023] S4: When the resource management module detects that the storage space of the edge storage node has decreased to 10%, the scheduling module will store the excess data in the core storage node;
[0024] S5: When the resource management module detects that less than 10TB and more than 1TB of data enters the system within a day, the resource management module compresses the data in the core storage node and deletes the garbled garbage data in the core storage node.
[0025] S6: When the resource management module detects that less than 1TB of data has entered the system within a day, the resource management module compresses the data in the edge storage node and deletes the garbled garbage data in the edge storage node.
[0026] Compared with the prior art, the beneficial effects of the present invention are:
[0027] 1. The data sharding and scheduling module in this invention is used to divide the data object to be stored into multiple data shards through calculation, and dynamically select and allocate target storage nodes for each data shard and its replica in the heterogeneous resource pool based on the status information provided by the resource management module, so as to realize elastic expansion of storage capacity.
[0028] 2. When using this invention, when the resource management module detects that more than 10TB of data has entered the system within a day, the scheduling module divides the data to be stored into multiple data shards, and selects an edge storage node from the heterogeneous resource pool for each data shard to store.
[0029] When the resource management module detects that less than 10TB and more than 1TB of data enters the system within a day, the resource management module compresses the data in the core storage node and deletes the garbled garbage data in the core storage node.
[0030] When the resource management module detects that less than 1TB of data has entered the system within a day, it compresses the data in the edge storage nodes and deletes garbled and junk data in the edge storage nodes.
[0031] In summary, the above settings enable data resources to be stored in a distributed manner within the system, and by prioritizing the use of edge storage nodes for storage, the utilization rate of core storage nodes is reduced, thus achieving elastic scaling of data storage. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the actual layout of the overall system of the present invention;
[0033] Figure 2 This is a schematic diagram of the overall system composition of the present invention;
[0034] Figure 3 This is a diagram showing the composition of the heterogeneous resource pool in the overall system of this invention;
[0035] Figure 4 This is a diagram of the cycle management strategy in the overall system of this invention;
[0036] Figure 5 This is a flowchart of the data repair process in the overall system of this invention;
[0037] Figure 6 This is a method diagram of the system of the present invention.
[0038] In the diagram: 1. Commercial data computer; 2. Commercial server; 3. Personal data terminal; 4. Personal data server; 5. Secondary server; 6. Server-specific storage; 7. Data storage device; 8. Main data processing library. Detailed Implementation
[0039] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0040] Please see Figures 1 to 6This invention provides an embodiment of a high-availability and elastic scaling system for distributed databases, comprising: a heterogeneous resource pool for storing various data from various databases, the heterogeneous resource pool including storage nodes and computing nodes, computing nodes for analyzing and processing input data, and storage nodes for storing the analyzed data; a resource management module for real-time registration, discovery, and monitoring of the status information of all nodes in the heterogeneous resource pool, the status information including storage space, computing load, and network health; data sharding for creating replicas of data within storage nodes in different heterogeneous resource pools; a scheduling module for dynamically selecting and allocating storage nodes for each data shard in the heterogeneous resource pool using the status information provided by the resource management module; and a data management module for maintaining the mapping relationship between data shards and storage nodes; the storage nodes include core storage nodes and edge storage nodes, with edge storage nodes prioritizing the storage of data and core storage nodes storing excess data.
[0041] The core storage nodes include a main data processing library 8, a server-dedicated storage device 6, and a data storage device 7; the edge storage nodes include a commercial data computer 1 and a personal data terminal 3; and the computing nodes include a commercial server 2, a personal data server 4, and a secondary server 5.
[0042] The data sharding and scheduling module uses a message digest algorithm to allocate data. The scheduling module prioritizes storing data in edge storage nodes and stores excess data in core storage nodes. When adding or removing nodes in this setup and operation, only a small number of affected data shards need to be migrated to achieve smooth and elastic expansion.
[0043] Each data shard creates a replica on at least two different storage nodes. When the resource management module detects a failure in any storage node, it automatically reads, writes, and copies the data on the storage node where the replica is located. The copied data is then migrated to a healthy storage node. This setup ensures that the data has two copies, thus preventing data loss.
[0044] When allocating data, the data sharding and scheduling module implements a periodic management strategy based on the data access frequency and performance requirements. The periodic management strategy includes: prioritizing the allocation of frequently accessed data to the commercial server 2 and server-dedicated storage 6; allocating infrequently accessed data to the commercial data computer 1 and secondary server 5; and archiving rarely accessed data to the data storage device 7, personal data server, or personal data terminal 3.
[0045] Furthermore, the resource management module maintains communication with all storage nodes and computing nodes. When the resource management module detects the addition of new storage nodes and computing nodes, it automatically includes them in the heterogeneous resource pool. When a storage node or computing node is detected to be faulty and offline, it marks it as unavailable and initiates a data repair process. The data repair process includes: the resource management module obtaining the resource status of all healthy nodes and creating new replicas of the resources of the healthy nodes; selecting one or more optimal storage and computing nodes for the new replicas; the main data processing library 8 sending a replication instruction to a healthy storage and computing node to transfer the new replica data to the healthy storage and computing node; the personal data terminal 3 and the personal data server 4 can also be set as edge computing nodes to execute edge computing tasks issued by the main data processing library 8.
[0046] The overall system can be used by some e-commerce platforms. Under normal circumstances, the system mainly runs 10 commercial servers 2, which can store 1PB of popular product images. During some promotional events, the image access volume and storage demand are expected to increase by 50%. The administrator connects 20 commercial data computers 1 in the office to the system in advance.
[0047] The main data processing database 8 automatically identifies newly added resources and gradually migrates some data to commercial server 2 during off-peak periods, while another portion of images is directly processed by edge computing nodes, reducing the storage pressure on data storage device 7. After the promotion ends, commercial server 2 is gradually taken offline, and data is automatically replicated to the main data processing database 8. This setup allows excess data generated during the event to be temporarily stored via edge computing nodes, preventing excessive data from occupying storage space on data storage device 7 and commercial server 2. This elastic expansion of data storage avoids data overload causing lag or crashes on data storage device 7 and commercial server 2.
[0048] During this period, if a commercial server 2 storing important product images crashes due to hard drive failure, the system will detect the fault within 45 seconds and immediately access two other healthy storage and computing nodes, such as a commercial data computer 1 and a personal data server, and start data repair in the background. The reconstruction of all missing data is completed within 2 hours, without affecting the user's normal browsing of products.
[0049] A method for a highly available and elastically scalable system for distributed databases, the method comprising the following steps:
[0050] S1: The main data processing library 8 discovers and registers all available nodes in the heterogeneous resource pool and receives data write requests;
[0051] S2: When the resource management module detects that more than 10TB of data has entered the system within a day, the scheduling module divides the data to be stored into multiple data fragments.
[0052] S3: The data sharding and scheduling module performs calculations to preferentially select an edge storage node from the heterogeneous resource pool for each data shard for storage;
[0053] S4: When the resource management module detects that the storage space of the edge storage node has decreased to 10%, the scheduling module will store the excess data in the core storage node;
[0054] S5: When the resource management module detects that less than 10TB and more than 1TB of data enters the system within a day, the resource management module compresses the data in the core storage node and deletes the garbled garbage data in the core storage node.
[0055] S6: When the resource management module detects that less than 1TB of data has entered the system within a day, the resource management module compresses the data in the edge storage node and deletes the garbled garbage data in the edge storage node.
[0056] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A highly available and elastically scalable system for distributed databases, characterized in that: include: The heterogeneous resource pool is used to store various data from various databases. The heterogeneous resource pool includes storage nodes and computing nodes. The computing nodes are used to analyze and process the input data, and the storage nodes are used to store the analyzed data. The resource management module is used to register, discover and monitor the status information of all nodes in the heterogeneous resource pool in real time. The status information includes storage space, computing load and network health. Data sharding is used to create replicas of data within storage nodes in different heterogeneous resource pools. The scheduling module uses the status information provided by the resource management module to dynamically select and allocate storage nodes for each data shard in the heterogeneous resource pool; The data management module is used to maintain the mapping relationship between data shards and storage nodes; The storage nodes include core storage nodes and edge storage nodes. Edge storage nodes prioritize storing data, while core storage nodes store excess data.
2. The high availability and elastic scaling system for distributed databases according to claim 1, characterized in that: The core storage node includes a main data processing library (8), a server-specific storage (6), and a data storage device (7). Edge storage nodes include commercial data computers (1) and personal data terminals (3); The computing nodes include commercial servers (2), personal data servers (4), and secondary servers (5).
3. A high-availability and elastic scaling system for distributed databases according to claim 2, characterized in that: The data within the data sharding and scheduling module is allocated using a message digest algorithm. The scheduling module prioritizes storing data in edge storage nodes and stores excess data in core storage nodes.
4. A high-availability and elastic scaling system for distributed databases according to claim 3, characterized in that: Each data shard creates a replica on at least two different storage nodes. When the resource management module detects a failure in any storage node, it automatically reads, writes, and copies the data on the storage node where the replica is located. The copied replica data is then migrated to a healthy storage node.
5. A high-availability and elastic scaling system for distributed databases according to claim 4, characterized in that: When allocating data, the data sharding and scheduling module employs a periodic management strategy based on the data access frequency and performance requirements.
6. A high-availability and elastic scaling system for distributed databases according to claim 5, characterized in that: The periodic management strategy includes: prioritizing the allocation of frequently accessed data to the commercial server (2) and server-dedicated storage (6); Infrequently accessed data is distributed to the commercial data computer (1) and the secondary server (5). Data that is rarely accessed is archived to the data storage device (7), personal data server or personal data terminal (3).
7. A high-availability and elastic scaling system for distributed databases according to claim 6, characterized in that: The resource management module maintains communication with all storage nodes and computing nodes; When the resource management module detects the addition of new storage nodes and compute nodes, it automatically includes them in the heterogeneous resource pool; When a storage node or compute node is detected to be faulty and offline, it is marked as unavailable and a data recovery process is initiated.
8. A high-availability and elastic scaling system for distributed databases according to claim 7, characterized in that: The data repair process includes: the resource management module obtains the resource status of all healthy nodes and creates new copies of the resources of the healthy nodes; Select one or more optimal storage and compute nodes for the new replica; The main data processing library (8) sends a replication instruction to a healthy storage and compute node to transfer the new copy data to the healthy storage and compute node.
9. A high-availability and elastic scaling system for distributed databases according to claim 8, characterized in that: The personal data terminal (3) and personal data server (4) can also be set as edge computing nodes to execute edge computing tasks issued by the general data processing library (8).
10. A method for a highly available and elastically scalable system for a distributed database according to any one of claims 1-9, characterized in that: The method includes the following steps: S1: The main data processing library (8) discovers and registers all available nodes in the heterogeneous resource pool and receives data write requests; S2: When the resource management module detects that more than 10TB of data has entered the system within a day, the scheduling module divides the data to be stored into multiple data fragments. S3: The data sharding and scheduling module performs calculations to preferentially select an edge storage node from the heterogeneous resource pool for each data shard for storage; S4: When the resource management module detects that the storage space of the edge storage node has decreased to 10%, the scheduling module will store the excess data in the core storage node; S5: When the resource management module detects that less than 10TB and more than 1TB of data enters the system within a day, the resource management module compresses the data in the core storage node and deletes the garbled garbage data in the core storage node. S6: When the resource management module detects that less than 1TB of data has entered the system within a day, the resource management module compresses the data in the edge storage node and deletes the garbled garbage data in the edge storage node.