A fast partition splitting solution in distributed data storage systems

Abstract

The present disclosure provides a distributed data storage system including a fast partition splitting method. The distributed data storage system includes a parent partition, multiple intermediate partitions and multiple child partitions. The parent partition is divided into multiple intermediate partitions, including a first intermediate partition. When data replication occurs from the parent partition to the first intermediate partition, the first intermediate partition is divided into multiple child partitions. Each intermediate partition includes a reference file pointing to a data file in the parent partition. Each child partition includes a reference file pointing to a reference file of the first intermediate partition, and another reference file pointing to a data file in the first intermediate partition. The distributed data storage system of the present disclosure also includes a split reference counter to track when a partition can be deleted from the system. In addition, the present disclosure provides a method for fast partition splitting, and a non-temporary computer-readable medium storing an instruction set, which can be executed by one or more processors to perform the method for fast partition splitting. The present disclosure can effectively reduce the access adjustments of traditional systems caused by partition splitting.

Description

Background Art

[0001] In distributed data storage systems, storing all data in a single table is often infeasible. Instead, such systems typically divide a table into distinct parts, called partitions. As the amount of data in a table grows, partitions need to be split to keep their size manageable. Because partitions are often inaccessible while the split is in progress, partition splits can cause significant latency in distributed data storage systems. Therefore, traditional distributed data storage systems need improvement. Summary of the Invention

[0002] An embodiment of the present disclosure provides a distributed data storage system, including a fast partition splitting method. The distributed data storage system includes a parent partition, multiple intermediate partitions, and multiple child partitions. The parent partition is divided into multiple intermediate partitions, including a first intermediate partition. When data replication from the parent partition to the intermediate partition occurs, the first intermediate partition is divided into multiple child partitions. Each intermediate partition includes a reference file pointing to a data file in the parent partition. Each child partition includes a reference file pointing to a reference file of the first intermediate partition. The distributed data storage system of the present disclosure also includes a split reference counter to track when a partition can be deleted from the system.

[0003] The present disclosure provides a method for fast partition splitting, which includes splitting a parent partition into multiple intermediate partitions, splitting a first intermediate partition of the multiple intermediate partitions into multiple child partitions, and simultaneously copying the first intermediate partition from the data file of the parent partition to the data file of the first intermediate partition.

[0004] Splitting the parent partition into intermediate partitions includes initiating data copying from a data file of the parent partition to a data file of a first intermediate partition of the intermediate partition, and creating a pointer between the data file of the parent partition and a reference file of the first intermediate partition. Splitting the first intermediate partition into a plurality of child partitions includes initiating data copying from at least one of the data file of the parent partition or the data file of the intermediate partition to data files of child partitions of the plurality of child partitions, creating a pointer between the first child reference file and the data file of the first intermediate partition, and creating a second pointer between the second child reference file and the reference file of the first intermediate partition.

[0005] An embodiment of the present disclosure further provides a non-transitory computer-readable medium having an instruction set stored thereon, wherein the instruction set can be executed by one or more processors to perform a method for quickly splitting a partition, the method comprising:

[0006] Splitting a parent partition into multiple intermediate partitions, and dividing a first intermediate partition among the multiple intermediate partitions into multiple child partitions, while the first intermediate partition is copied from the data file of the parent partition to the data file of the first intermediate partition; splitting the parent partition into intermediate partitions includes starting data copying from the data file of the parent partition to the data file of the first intermediate partition of the intermediate partition, and creating a pointer between the data file of the parent partition and the reference file of the first intermediate partition; splitting the first intermediate partition into multiple child partitions includes starting data copying from at least one of the data file of the parent partition or the data file of the intermediate partition to the data file of a child partition of the multiple child partitions, creating a first pointer between the first child reference file and the data file of the first intermediate partition, and creating a second pointer between the second child reference file and the reference file of the first intermediate partition. BRIEF DESCRIPTION OF THE DRAWINGS

[0007]

[0011] Embodiments and aspects of the present disclosure are described in the following detailed description and accompanying drawings.The various features shown in the drawings are not drawn to scale.

[0008] Figure 1A is a diagram illustrating an exemplary distributed data storage system including servers with partitions according to some embodiments of the present disclosure.

[0009] Figure 1B is a schematic diagram illustrating an exemplary server of a distributed data storage system according to some embodiments of the present disclosure.

[0010] Figure 2 is a diagram illustrating an exemplary distributed data storage system using fast partition splitting according to some embodiments of the present disclosure.

[0011] Figure 3 is a diagram illustrating an exemplary distributed data storage system using fast partition splitting and split reference counters according to some embodiments of the present disclosure.

[0012] Figure 4 is a diagram illustrating an exemplary distributed data storage system using fast partition splitting for multiple data files in a parent partition according to some embodiments of the present disclosure.

[0013] Figure 5 is a diagram illustrating an exemplary distributed data storage system using fast partition splitting involving a three-way split according to some embodiments of the present disclosure.

[0014] Figure 6 is a diagram illustrating an exemplary distributed data storage system using fast partition splitting involving multiple levels of intermediate partitions according to some embodiments of the present disclosure.

[0015] Figure 7 is a flowchart of an exemplary method for fast partition splitting in a distributed data storage system according to some embodiments of the present disclosure.

[0016] Figure 8 is a flowchart of an exemplary method for fast partition splitting in combination with split reference counters in a distributed data storage system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] The following description refers to the accompanying drawings, in which the same numerals in different drawings represent the same or similar elements unless otherwise specified. The embodiments set forth in the following description of exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with the relevant aspects of the present disclosure as described in the appended claims.

[0018] In distributed data storage systems, data tables are typically divided into multiple partitions. As partition size increases, large partitions need to be split into smaller ones to keep the size manageable. Large partitions slow down the redistribution process, leading to slow recovery, load balancing difficulties, and the potential risk of insufficient disk space on the underlying servers.

[0019] Traditional distributed data storage systems have various drawbacks when it comes to partition splitting. In traditional distributed data storage systems, when a parent partition is split into two child partitions, access to the parent partition must be shut down before data in the parent partition can be copied to the child partitions. During the data copy process, access to both child partitions is denied until the data copy is complete. Due to the long processing time for data copying, the data in the parent partition remains unavailable, significantly reducing the access efficiency of the distributed data storage system.

[0020] To alleviate this problem, some traditional systems attempt to optimize partition splitting by allowing read and write requests to child partitions while data replication is in progress. For each data file in the parent partition, a reference file is created in the child partition. The reference file contains a pointer to the parent partition's data file and the child partition's partition key range. As a result, read operations on the child partition are converted to read operations on the parent partition's data file through the reference file, and the partition key range in the reference file allows read operations to operate within the child partition's key range. Distributed data storage systems no longer need to wait for data replication to complete before performing read operations. Child partitions can perform write operations directly.

[0021] While this solution improves access efficiency, it has some significant drawbacks. While a child partition is replicating data from its parent partition, it cannot be further split until the child partition completes data replication. By the time a distributed data storage system decides to split a partition, it is likely already experiencing heavy traffic, necessitating further splits immediately after the parent partition is split. Because data replication typically takes a long time to complete, the distributed data storage system incurs significant overhead in managing access to further split child partitions.

[0022] Embodiments of the present disclosure provide a system that incorporates fast partition splitting to alleviate the problems of conventional systems. Fast partition splitting allows for instant splitting of subpartitions. Figure 1A FIG is a schematic diagram illustrating an exemplary distributed data storage system including servers with partitions according to some embodiments of the present disclosure. Figure 1A As shown, exemplary distributed data storage system 100 includes multiple servers 1-N, each server having ten different partitions. For example, server 1 includes partitions A1-A10, server 2 includes partitions B1-B10, and server N includes partitions N1-N10. Distributed data storage system 100 allows for each partition to be reassigned to a different server. It will be appreciated that one or more of these servers may include a fast partition split.

[0023] Figure 1B FIG. 1 is a schematic diagram illustrating an exemplary server 110 of a distributed data storage system according to some embodiments of the present disclosure. Figure 1B As shown, server 110 includes a bus 112 or other communication mechanism for communicating information, and one or more processors 116 communicatively coupled with bus 112 for processing information. Processor 116 may be, for example, one or more microprocessors.

[0024] Server 110 also includes storage 114, which may include random access memory (RAM), read-only memory (ROM), and a data storage system composed of partitions. Storage 114 may be communicatively coupled to processor 116 via bus 112. Storage 114 may include main memory, which may be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 116. Such instructions, after being stored in a non-transitory storage medium accessible to processor 116, render server 110 a special-purpose machine customized to perform the operations specified in the instructions.

[0025] Server 110 can transmit data to another server 130 or communicate with another server 130 via network 122. Network 122 can be a local network, an internet service provider, the internet, or any combination thereof. Communication interface 118 of server 110 is connected to network 122. In addition, server 110 can be coupled to peripheral devices 140 via bus 112. Peripheral devices 140 include a display (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a touch screen, etc.) and input devices (e.g., a keyboard, a mouse, a soft keyboard, etc.).

[0026] Server 110 may be implemented using custom hardwired logic, one or more ASICs or FPGAs, firmware, or program logic that, in combination with the server, makes server 110 a special purpose machine.

[0027] As used herein, the term "non-transitory media" refers to any non-transitory medium that stores data or instructions that cause a machine to operate in a specific manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-transitory media include, for example, optical or magnetic disks, dynamic memory, floppy disks, diskettes, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, NVRAM, flash memory, registers, cache, any other memory chip or cartridge memory, and network versions of the same.

[0028] Various forms of media may be involved in transmitting one or more sequences of one or more instructions to processor 116 for execution. For example, the instructions may initially be executed on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to server 110 may receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 112. Bus 112 transfers the data to main memory within storage device 114, from which processor 116 retrieves and executes the instructions.

[0029] Figure 2 is a schematic diagram illustrating an exemplary distributed data storage system using fast partition splitting according to some embodiments of the present disclosure. In this example, the parent partition O includes the data file O1.data. When the parent partition O is split, as shown in FIG. Figure 2 As shown, create middle partition A and middle partition B.

[0030] In addition, during the split, the parent partition O is closed to service read and write requests, and the distributed data storage system initiates data replication from partition O to intermediate partitions A and B. In some embodiments, the data replication is performed using compression (e.g., primary compression). In compression, all data files of the target partition (e.g., intermediate partition A and intermediate partition B), including the reference files of the partition, are merged into a large data file. The large data file is placed in a temporary directory during the compression process. When the merge is complete and the large data file is verified, the large data file is passed to the partition in a single atomic operation, and the obsolete data files and reference files are deleted. Compression simplifies error handling by preventing the partition from accessing partially completed data files. In addition, compression improves the read performance of the partition by reducing the number of data files in the partition.

[0031] Intermediate partitions A and B can create data files A1.data and B1.data, respectively, to store newly written data in intermediate partitions A and B. Intermediate partitions A and B can also create reference files O1.A.ref and O1.B.ref, respectively. Reference files O1.A.ref and O1.B.ref both point to the parent partition's data file O1.data.

[0032] This arrangement of data files and reference files on the intermediate partitions allows for the ability to service read and write requests for the corresponding data. For example, read requests served on intermediate partitions A and B are directed to O1.data through pointers in reference files O1.A.ref and O1.B.ref, respectively, while write requests served on intermediate partitions A and B are executed directly on data files A1.data and B1.data.

[0033] Before completing the data copy from partition O to intermediate partition A, intermediate partition A is further split into sub-partition C and sub-partition D. Sub-partition C includes a data file C1.data and two reference files A1.C.ref and O1.ACref, while sub-partition D includes a data file D1.data and two reference files A1.D.ref and O1.ADref. Reference files A1.C.ref and A1.D.ref point to data file A1.data of intermediate partition A. Reference files O1.ACref and O1.ADref point to reference file O1.A.ref of intermediate partition A, which points to data file O1.data of parent partition O. When intermediate partition A is split, intermediate partition A is closed to service read and write requests, and the distributed data storage system initiates data copy from partition O and intermediate partition A to sub-partition C and sub-partition D. In some embodiments, data copying from partition O and intermediate partition A to sub-partition C and sub-partition D is performed by compression.

[0034] Read requests for sub-partitions C and D are directed to the data file A1.data of the intermediate partition A through reference files A1.C.ref and A1.D.ref, and to the data file O1.data of the parent partition O through reference files O1.ACref and O1.ADref.

[0035] For example, to perform a read request on subpartition C, the distributed data storage system accesses reference file O1.ACref in subpartition C and discovers that the reference file has a key range [2000, 2499] and a pointer to another reference file, O1.A.ref. The distributed data storage system then opens reference file O1.A.ref in intermediate partition A and discovers that the reference file has a key range [2000, 2999] and a pointer to data file O1.data. The distributed data storage system intersects all key ranges to obtain the final range [2000, 2499] and converts the read request for reference file O1.ACref into a read request for data file O1.data filtered by the final range [2000, 2499]. Write requests on subpartitions C and D are serviced directly on data files C1.data and D1.data.

[0036] One advantage of the distributed data storage system according to some embodiments of the present disclosure is timely splitting. In traditional systems, intermediate partition A cannot be split until data is copied from partition O to intermediate partition A. Due to the large size of the parent partition, data copying takes a long time, which can cause significant delays in servicing read and write requests. The distributed data storage system of the present disclosure allows intermediate partition A to be split immediately before data copying is complete, and there is no delay in servicing read and write requests.

[0037] Another advantage of the distributed data storage system according to some embodiments of the present disclosure is reduced data duplication. In conventional systems, data replication for subpartitions C and D does not begin until data replication from partition O to intermediate partition A is complete. In the present disclosure, data replication for subpartitions C and D can be initiated immediately after the split. In some embodiments, instead of waiting for data replication from partition O to intermediate partition A to complete, data replication for subpartitions C and D can be initiated immediately after the split using compression, and data replication from partition O to intermediate partition A is abandoned. Thus, the distributed data storage system avoids duplicate data replication and saves valuable computing resources for other important tasks.

[0038] In some embodiments of the present disclosure, a distributed data storage system implements a split reference counter to track when partitions can be deleted from the system. Each partition creates a reference split counter when it is split. Each time a reference file is created from a child partition that points to a data file in its parent partition, the parent partition's split reference counter increments. Each time a child partition reference file that points to a data file in its parent partition is deleted from the system, the parent partition's split reference counter decrements. When the parent partition's split reference counter reaches 0, the distributed data storage system deletes the parent partition.

[0039] picture. Figure 3 3 is a schematic diagram illustrating an exemplary distributed data storage system using fast partition splitting and split reference counters according to some embodiments of the present disclosure. According to FIG. 3 , parent partition O is split into intermediate partition A and intermediate partition B. Since two reference files O1.A.ref and O1.B.ref are created to point to data file O1.data of parent partition O, the split reference counter of parent partition O is incremented to 2. Before completing the data copy from parent partition O to intermediate partition A, intermediate partition A is further split into child partitions C and D. Since two reference files A1.C.ref and A1.D.ref are created to point to data file A1.data of intermediate partition A, the split reference counter of intermediate partition A is incremented to 2.

[0040] When the data copying of child partition C is completed, the distributed data storage system deletes the reference files A1.C.ref and O1.ACref. Then the split reference counter of partition A is decremented from 2 to 1. When the data copying of child partition D is completed, the distributed data storage system deletes the reference files A1.D.ref and O1.ADref. Then the split reference counter of partition A is decremented from 1 to 0, and the intermediate partition A is deleted. At the same time as the intermediate partition A is deleted, the reference file O1.A.ref is also deleted, which causes the split reference counter of the parent partition O to be decremented from 2 to 1. If the intermediate partition B also deletes the reference file O1.B.ref, then the split reference counter of the parent partition O is decremented from 1 to 0, thereby deleting the parent partition O.

[0041] In some embodiments of the present disclosure, a parent partition contains multiple data files. When a parent partition is split into two other partitions, the data files of each parent partition can be allocated to a single partition or split into different partitions. Figure 4FIG1 is a schematic diagram illustrating an exemplary distributed data storage system that uses rapid partition splitting for multiple data files in a parent partition according to some embodiments of the present disclosure. The parent partition O.4 in FIG1 contains two data files, O1.data and O2.data. When the parent partition O is split into an intermediate partition A and an intermediate partition B, the data file O1.data is copied to the intermediate partition A, and the data file O2.data is copied to the intermediate partition B.

[0042] A reference file O1.A.ref from the middle partition A is created to point to the data file O1.data, and a reference file O2.B.ref from the middle partition B is created to point to the data file O2.data. Since two reference files O1.A.ref and O2.B.ref are created to point to the data files in the parent partition O, the split reference counter of the parent partition O is incremented to 2.

[0043] Before intermediate partition A completes data copying, intermediate partition A is further split into sub-partition C and sub-partition D. When sub-partitions C and D complete data copying, the split reference counter of intermediate partition A is decremented to 0. As a result, intermediate partition A is deleted, causing the split reference counter of parent partition O to decrement from 2 to 1. When intermediate partition B completes data copying, the reference file O2.B.ref is deleted, causing the split reference counter of parent partition O to decrement from 1 to 0. The distributed data storage system then deletes parent partition O.

[0044] In some embodiments of the present disclosure, the number of new partitions created due to the split is greater than two. Figure 51 is a schematic diagram illustrating an exemplary distributed data storage system using fast partition splitting involving three-way splitting according to some embodiments of the present disclosure. The parent partition O of FIG1. ​​5 contains two data files O1.data and O2.data. The parent partition O is divided into three partitions: an intermediate partition A, an intermediate partition B, and an intermediate partition E. A reference file O1.A.ref is created from the intermediate partition A to point to the data file O1.data and a reference file O2.B is created. A ref for the intermediate partition B is created to point to the data file O2.data, and a reference file O2.E.ref for the intermediate partition E is created to point to the data file O2.data. Since three reference files O1.A.ref, O2.B.ref, and O2.E.ref are created to point to the data files in the parent partition O, the split reference counter of the parent partition O is incremented to 3. Before the intermediate partition A completes data replication, the intermediate partition A is further split into sub-partitions C and D. When sub-partitions C and D complete data replication, the split reference counter of the intermediate partition A is reduced to 0. As a result, intermediate partition A is deleted, causing the split reference counter of parent partition O to decrease from 3 to 2. When intermediate partitions B and intermediate partition E complete data replication, reference files O2.B.ref and O2.E.ref are deleted, causing the split reference counter of parent partition O to decrease from 2 to 0. The distributed data storage system then deletes parent partition O.

[0045] In some embodiments of the present disclosure, multiple levels of intermediate partitions can exist. The term intermediate partition is used to describe partitions that are split before data replication is complete. If a subpartition is split before data replication is complete, the subpartition becomes an intermediate partition. As shown in the figure. Figure 6 1 is a schematic diagram illustrating an exemplary distributed data storage system using fast partition splitting involving multi-level intermediate partitions according to some embodiments of the present disclosure. The parent partition O.6 of FIG1 contains a data file O1.data. The parent partition O is divided into two partitions: an intermediate partition A and an intermediate partition B. A reference file O1.A.ref is created for the intermediate partition A to point to the data file O1.data, and a reference file O1.B.ref is created for the intermediate partition B to point to the data file O1.data. Since two reference files O1.A.ref and O1.B.ref are created to point to the data file in the parent partition O, the split reference counter of the parent partition O is incremented to 2. Before the intermediate partition A completes the data copy, the intermediate partition A is further divided into a sub-partition C and a sub-partition D.

[0046] Before subpartition C completes data copying, it is further split into subpartition F and subpartition G. Subpartition C becomes an intermediate partition because it is split before data copying is complete. Subpartition F has one data file, F1.data, and three reference files, C1.F.ref, A1.CFref, and O1.ACFref. Subpartition G has one data file, G1.data, and three reference files, C1.G.ref, A1.CGref, and O1.ACGref. Reference files C1.F.ref and C1.G.ref point to data file C1.data. Reference files A1.CFref and A1.CGref point to A1.C.ref, which in turn points to data file A1.data. Reference files O1.ACFref and O1.ACGref point to O1.ACref, which in turn points to O1.A.ref, which in turn points to data file O1.data. When child partitions F and G complete data replication, the split reference counter of intermediate partition C is decremented to 0. As a result, intermediate partition C is deleted, causing the split reference counter of parent partition A to be decremented from 2 to 1. D completes data replication, and the split reference counter of intermediate partition A is decremented to 0. As a result, intermediate partition A is deleted, causing the split reference counter of parent partition O to be decremented from 2 to 1. When intermediate partition B completes data replication, the reference file O2.B.ref is deleted, causing the split reference counter of parent partition O to be decremented from 1 to 0. The distributed data storage system then deletes parent partition O.

[0047] The embodiment of the present invention also provides a method for fast partition splitting. Figure 7 FIG1 is a flowchart illustrating an exemplary method 1000 for fast partition splitting in a distributed data storage system according to some embodiments of the present disclosure. It is understood that the method 1000 can be implemented by a distributed data storage system (e.g., Figure 1A distributed data storage system 100) or by one or more servers (e.g., Figure 1B exemplarily executed by the server 110).

[0048] In step 1010, the parent partition containing the data files is divided into at least two intermediate partitions, each of which has a corresponding reference file. For example, as shown in FIG. 2, the parent partition O is split into intermediate partitions A and B.

[0049] In step 1020, data is copied from the parent partition to the intermediate partitions, including the first intermediate partition. For example, as shown in FIG. 2, data is copied from parent partition O to intermediate partitions A and B.

[0050] In step 1030, a pointer is created between the data file of the parent partition and the reference file in the first intermediate partition. For example, as shown in FIG. 2, a reference file O1.A.ref is created, pointing to the data file O1.data of the parent partition O.

[0051] In step 1040, the first intermediate partition is split into multiple subpartitions, while data replication for the first intermediate partition is occurring. Each subpartition has a corresponding reference file. For example, as shown in FIG. 2, intermediate partition A is split into subpartitions C and D. Subpartition C has a reference file O1.ACref, and subpartition D has a reference file O1.ADref.

[0052] In step 1050, data replication is initiated from the parent partition and the first intermediate partition to the child partitions. For example, as shown in Figure 2, child partition C copies data from data files A1.data and O1.data to C1.data, and child partition D copies data from data files A1.data and O1.data to D1.data.

[0053] In step 1060, pointers are created between the data files in the parent partition and each reference file in the child partition using the reference files in the first intermediate partition. For example, as shown in FIG. 2, reference files O1.ACref and O1.ADref are created to point to reference file O1.A.ref, which in turn points to data file O1.data.

[0054] In some embodiments, method 1000 further includes additional steps involving splitting the reference counter. Figure 8 A flowchart of an exemplary method for fast partition splitting in a distributed data storage system in combination with split reference counters according to some embodiments of the present disclosure is illustrated. Figure 7 On the basis of Figure 8 The method further includes step 1035, step 1065, step 1075, step 1085 and step 1095. It is understood that the method 1000 can be implemented by a distributed data storage system (e.g., Figure 1A ) or by one or more servers (e.g., Figure 1B exemplarily server 110).

[0055] Step 1035 is performed after step 1030. In step 1035, the split reference counter of the parent partition is increased by a count equal to the number of new reference files created to point to the data file of the parent partition. Figure 3As shown, since two new reference files O1.A.ref and O1.B.ref are created, the split reference counter of the parent partition O increases from 0 to 2.

[0056] Step 1065 is executed after step 1060. In step 1065, the reference counter of the first intermediate partition is increased by the number of new reference files, and the new reference files are created to point to the data files of the first intermediate partition. Figure 3 As shown, since two new reference files A1.C.ref and A1.D.ref are created to point to the data file A1.data of the middle partition A, the split reference counter of the middle partition A increases from 0 to 2.

[0057] In step 1075, whenever a child partition completes data replication and deletes a child reference file pointing to the first intermediate partition data file, the split reference counter of the first intermediate partition is reduced by 1. Figure 3 As shown, after the sub-partition C completes the data copy, the reference file A1.C.ref is deleted, and the split reference counter of the middle partition A is decremented from 2 to 1.

[0058] In step 1085, whenever the intermediate partition completes data replication and deletes the reference file of the intermediate partition pointing to the parent partition data file, the split reference counter of the parent partition is reduced by 1. Figure 3 As shown, when the split reference counter of the middle partition A is 0, the middle partition A is deleted, causing the split reference counter of the parent partition O to decrease from 2 to 1.

[0059] In step 1095, when the parent partition split reference counter or the middle partition split reference counter is 0, the parent partition or the middle partition is deleted. Figure 3 As shown, when the intermediate partition B completes data copying, the reference file O1.B.ref is deleted, causing the split reference counter of the parent partition O to decrease from 1 to 0. Then the parent partition O is deleted.

[0060] The various example embodiments described herein are described in the general context of method steps or processes, which can be implemented in one aspect by a computer program product embodied in a computer-readable medium, including computer-executable instructions, such as program code, that are executed by a computer in a network environment. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact disks (CDs), digital versatile disks (DVDs), etc. Program modules may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for performing the steps of the methods disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in these steps or processes.

[0061] In the foregoing description, embodiments have been described with reference to many specific details, which may vary depending on the implementation. Certain modifications and variations may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art, given the description and practice of the invention disclosed herein. It is intended that the description and embodiments be considered merely exemplary, with the true scope and spirit of the invention being indicated by the appended claims. The order of steps shown in the figures is also intended to be for illustrative purposes only and is not intended to be limiting to any particular order of steps. Therefore, it will be understood by those skilled in the art that these steps may be performed in different orders while implementing the same method.

[0062] While exemplary embodiments have been disclosed in the drawings and the specification, many variations and modifications are possible. Therefore, although specific terms are used, they are used in a generic and descriptive sense only and not for purposes of limitation, the scope of the embodiments being defined by the appended claims.

Claims

1. A distributed data storage system comprising: The parent partition that contains the parent data file; an intermediate partition split from the parent partition, the intermediate partition comprising an intermediate data file and an intermediate reference file pointing to the parent data file, wherein the intermediate partition is configured to initiate data replication while the intermediate partition remains readable and writable; and A first sub-partition is split from the intermediate partition during data replication of the intermediate partition, the first sub-partition including a first sub-data file and a first sub-reference file and a second sub-reference file, wherein the first sub-reference file is configured to point to the intermediate data file, the second sub-reference file is configured to point to the intermediate reference file, and the first sub-partition is configured to start data replication while the first sub-partition remains readable and writable.

2. The distributed data storage system according to claim 1, further comprising: A second sub-partition, wherein the second sub-partition is split from the intermediate partition during data replication of the intermediate partition, the second sub-partition includes a second sub-data file and a third sub-reference file and a fourth sub-reference file, wherein the third sub-reference file is configured to point to the intermediate data file, the fourth sub-reference file is configured to point to the intermediate reference file, and the second sub-partition is configured to start data replication while the second sub-partition remains readable and writable.

3. The distributed data storage system according to claim 1 or 2, further comprising: a third subpartition, which is split from the first subpartition during data replication of the first subpartition; the third subpartition includes a third sub-data file and a fifth sub-reference file, a sixth sub-reference file, and a seventh sub-reference file, wherein the fifth sub-reference file is configured to point to the first sub-data file, the sixth sub-reference file is configured to point to the first sub-reference file, and the seventh sub-reference file is configured to point to the second sub-reference file; and the third subpartition is configured to start data replication while the third subpartition remains readable and writable.

4. The distributed data storage system according to claim 1 or 2, wherein: The data replication is performed using compression.

5. The distributed data storage system according to claim 1 or 2, characterized in that: The middle partition further includes a split reference counter configured to: Incrementing a count of the number of new reference files of the child partitions pointing to the data files of the intermediate partition; and The value is decremented by 1 in response to completion of data copying for a child partition.

6. The distributed data storage system according to claim 5, wherein: The middle partition is further configured to be deleted in response to a split reference counter of the middle partition reaching zero.

7. The distributed data storage system according to any one of claims 1 to 2 and 6, wherein: The intermediate reference file of the intermediate partition is configured to be deleted in response to completion of data replication of the intermediate partition.

8. The distributed data storage system according to any one of claims 1 to 2 and 6, wherein: The child reference file of the child partition is configured to be deleted in response to completion of data replication of the child partition.

9. The distributed data storage system according to any one of claims 1 to 2 and 6, characterized in that: The parent partition also includes a split reference counter for: Incrementing a count by the number of new reference files pointing to the parent data file; and Decremented by 1 in response to completion of data replication for an intermediate partition.

10. The distributed data storage system according to claim 9, wherein: The parent partition is configured to be deleted in response to a split reference counter of the parent partition reaching zero.

11. A method for splitting partitions in a distributed data storage system, comprising: Splitting the parent partition into intermediate partitions, the splitting comprising: Initiating data replication from the parent partition to a first intermediate partition of the intermediate partitions, and creating a pointer between the data file of the parent partition and the reference file of the first intermediate partition; and When data replication occurs from the parent partition to the first intermediate partition, the first intermediate partition is split into a plurality of sub-partitions, where the splitting of the first intermediate partition includes: initiating data replication from at least one of the parent partition or the intermediate partition to a first child partition of the plurality of child partitions, creating a first pointer between a first sub-reference file in the first sub-partition and a data file in the first intermediate partition, and A second pointer is created between the second sub-reference file in the first sub-partition and the reference file in the first middle partition.

12. The method according to claim 11, wherein Splitting the first intermediate partition into multiple sub-partitions further includes: initiating data replication from at least one of the parent partition or the intermediate partition to a second child partition of the plurality of child partitions, creating a third pointer between a third sub-reference file in the second sub-partition and the data file in the first intermediate partition, and A fourth pointer is created between a fourth sub-reference file in the second sub-partition and the reference file in the first middle partition.

13. The method according to claim 11 or 12, further comprising: The first sub-partition is split while data is being copied to the first sub-partition, wherein the splitting of the first sub-partition includes: initiating data replication from at least one of the parent partition, the first intermediate partition, or the first child partition to a third child partition, creating a fifth pointer between a fifth sub-reference file in the third sub-partition and the data file of the first sub-partition, creating a sixth pointer between the sixth sub-reference file in the third sub-partition and the first sub-reference file in the first sub-partition, and A seventh pointer is created between the seventh sub-reference file in the third sub-partition and the second sub-reference file in the first sub-partition.

14. The method according to claim 11 or 12, wherein: The data replication is performed using compression.

15. The method according to claim 11 or 12, wherein: Splitting the first intermediate partition into a plurality of sub-partitions further comprises creating a split reference counter for the first intermediate partition, wherein the split reference counter increases by a count corresponding to the number of new reference files of the sub-partitions pointing to the data files of the first intermediate partition, and decreases by one in response to a sub-partition completing data copying.

16. The method according to claim 15, wherein splitting the first intermediate partition into a plurality of sub-partitions further comprises: In response to the split reference counter of the first intermediate partition reaching 0, the first intermediate partition is deleted.

17. The method according to any one of claims 11 to 12 or 16, wherein: Splitting the parent partition into intermediate partitions further includes: In response to the completion of data replication in the first intermediate partition, the reference file of the first intermediate partition is deleted.

18. The method according to any one of claims 11 to 12 or 16, wherein: Splitting the first intermediate partition into multiple sub-partitions further includes: In response to the first sub-partition completing the data copy, the first sub-reference file and the second sub-reference file are deleted.

19. The method according to any one of claims 11-12 and 16, wherein splitting the parent partition into intermediate partitions further comprises creating a split reference counter for the parent partition, wherein the split reference counter increases a count that is the number of new reference files pointing to the data files of the parent partition and decreases by 1 in response to the completion of data copying by an intermediate partition.

20. The method according to claim 19, wherein Splitting the parent partition into intermediate partitions further includes: in response to a split reference counter of the parent partition reaching 0, deleting the parent partition.

21. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by one or more processors of a device to perform a method for splitting a partition in a distributed data storage system, the method comprising: Splitting the parent partition into intermediate partitions, the splitting comprising: Initiating data replication from the parent partition to a first intermediate partition of the intermediate partitions, and creating a pointer between the data file of the parent partition and the reference file of the first intermediate partition; and When data replication occurs from the parent partition to the first intermediate partition, the first intermediate partition is split into a plurality of sub-partitions, where the splitting of the first intermediate partition includes: initiating data replication from at least one of the parent partition or the intermediate partition to a first child partition of the plurality of child partitions, creating a first pointer between a first sub-reference file in the first sub-partition and a data file in the first intermediate partition, and A second pointer is created between the second sub-reference file in the first sub-partition and the reference file in the first middle partition.

22. The medium of claim 21, wherein dividing the first intermediate partition into a plurality of sub-partitions further comprises: initiating data replication from at least one of the parent partition or the intermediate partition to a second child partition of the plurality of child partitions, creating a third pointer between a third sub-reference file in the second sub-partition and the data file in the first intermediate partition, and A fourth pointer is created between a fourth sub-reference file in the second sub-partition and the reference file in the first middle partition.

23. The medium of claim 21 or 22, wherein the set of instructions is executable by one or more processors of the device to further perform: The first sub-partition is split while data is being copied to the first sub-partition, wherein the splitting of the first sub-partition includes: initiating data replication from at least one of the parent partition, the first intermediate partition, or the first child partition to a third child partition, creating a fifth pointer between a fifth sub-reference file in the third sub-partition and the data file of the first sub-partition, creating a sixth pointer between the sixth sub-reference file in the third sub-partition and the first sub-reference file in the first sub-partition, and A seventh pointer is created between the seventh sub-reference file in the third sub-partition and the second sub-reference file in the first sub-partition.

24. The medium according to claim 21 or 22, wherein The data replication is performed using compression.

25. The medium of claim 21 or 22, wherein splitting the first intermediate partition into a plurality of sub-partitions further comprises creating a split reference counter for the first intermediate partition, wherein The split reference counter increases by a count corresponding to the number of new reference files of the child partitions pointing to the data file of the first intermediate partition, and decreases by 1 in response to completion of data copying of a child partition.

26. The medium of claim 25, wherein splitting the first intermediate partition into a plurality of sub-partitions further comprises: In response to the split reference counter of the first intermediate partition reaching 0, the first intermediate partition is deleted.

27. The medium according to any one of claims 21-22, 26, wherein splitting the parent partition into intermediate partitions further comprises: In response to the completion of data replication in the first intermediate partition, the reference file of the first intermediate partition is deleted.

28. The medium of any one of claims 21-22, 26, wherein splitting the first intermediate partition into a plurality of sub-partitions further comprises: In response to the completion of data replication in the first sub-partition, the first sub-reference file and the second sub-reference file are deleted.

29. The medium of any one of claims 21-22, 26, wherein splitting the parent partition into intermediate partitions further comprises creating a split reference counter for the parent partition, wherein the split reference counter increases by a count that is the number of new reference files pointing to the data files of the parent partition, and decreases by 1 in response to an intermediate partition completing data copying.

30. The medium of claim 29, wherein splitting the parent partition into intermediate partitions further comprises: In response to the split reference counter of the parent partition reaching 0, the parent partition is deleted.

Images