Sharding method, device, server and medium of distributed table system

Abstract

The embodiment of the present invention provides a shard splitting method, device, server and medium for a distributed table system, and relates to the field of distributed storage technology. The scheme of the embodiment of the present application includes: determining a shard to be split from a plurality of shards, and then establishing a hard link file for the shard to be split. The hard link file is an index of a database file of the shard to be split, and then the shard to be split is split based on the hard link file. The embodiment of the present invention takes less time to split the shard based on the hard link file, which can reduce the time consumed by the shard splitting and improve the availability of the write service.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a sharding method, device, server and medium for a distributed table system. Background Art

[0002] When the amount of data in a table is large, the table can be sliced ​​horizontally to obtain multiple slices, each of which covers a certain range of the table, that is, the database file of each slice stores a certain range of data in the table. The ranges covered by the slices corresponding to a table have no intersection, and the union of the ranges covered by the slices is equal to the range covered by the table. When data needs to be read and written to a table, the client sends a request to the proxy server. After receiving the request, the proxy server first requests the metadata server to obtain the information of the data server to which the slice containing the requested data belongs, and then the proxy server sends this request to the data server.

[0003] In order to improve the read and write efficiency of the shard and the space utilization of the data server, when the data volume of the database file corresponding to the shard reaches the threshold, the shard can be split into two sub-shards. The current splitting method is: suspend the writing of the shard to be split, copy a database file of the shard to be split, and obtain a copy file. The copy file is a newly generated file, and the content included in the copy file is the same as the database file. Create two sub-shards, recorded as sub-shard 1 and sub-shard 2. The database file of sub-shard 1 is the database file of the shard to be split, and the database file of sub-shard 2 is the copy file. Delete part of the data in the database file of sub-shard 1, and delete another part of the data in the database file of sub-shard 2. The data retained by the two sub-shards have no intersection, and the union of the data retained by the two sub-shards is equal to the data in the database file of the shard to be split.

[0004] When splitting a shard, you need to suspend writing to the shard first, and the amount of data in the shard's database file is large, so it takes a long time to copy the database file. Writing to the shard will not resume until the split is complete. It can be seen that the splitting of the shard takes a long time, and suspending the writing of the shard during the split affects the availability of the write service. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a shard splitting method, device, server and medium for a distributed table system to improve the availability of write services. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a shard splitting method for a distributed table system, which is applied to a storage server of a distributed table system, wherein the distributed table system includes at least one table, each table includes multiple shards, and the multiple shards are stored in different storage servers. The method includes:

[0007] Determine a shard to be split from the multiple shards;

[0008] Creating a hard link file of the shard to be split, wherein the hard link file is an index of the database file of the shard to be split;

[0009] Based on the hard link file, the shard to be split is split.

[0010] Optionally, the splitting the shard to be split based on the hard link file includes:

[0011] Create a first sub-shard and a second sub-shard;

[0012] Using the database file of the shard to be split as the database file of the first sub-shard, and setting the data range included in the database file of the first sub-shard to: the first specified range in the database file of the shard to be split;

[0013] The hard link file is used as the database file of the second sub-shard, and the data range included in the database file of the second sub-shard is set to: the second specified range in the database file of the shard to be split, the intersection of the first specified range and the second specified range is an empty set, and the union of the first specified range and the second specified range is the data range included in the database file of the shard to be split.

[0014] Optionally, determining a shard to be split from the multiple shards includes:

[0015] Check the amount of data in each primary shard;

[0016] The primary shard whose data volume meets the preset conditions is determined as the shard to be split.

[0017] Optionally, after determining the shard to be split from the multiple shards, the method further includes:

[0018] Generate a split request, where the split request is used to request to split a slave shard corresponding to the shard to be split;

[0019] Writing the split request into a log, and sending the log to a slave server to which the slave shard corresponding to the shard to be split belongs;

[0020] When it is detected that a preset number of slave servers have successfully stored the log, the step of establishing the hard link file of the shard to be split is performed.

[0021] Optionally, after splitting the shard to be split based on the hard link file, the method further includes:

[0022] Determine the first sub-shard as a primary shard, and determine the second sub-shard as a secondary shard;

[0023] Based on a preset election rule, a primary shard is elected from the second sub-shards and other second sub-shards of the shard to be split, where the other second sub-shards are: the second sub-shards obtained after the slave shard corresponding to the shard to be split is split.

[0024] Optionally, determining a shard to be split from the multiple shards includes:

[0025] Receive a log sent by a master server to which a primary shard belongs, wherein the log includes a split request, and the split request is used to request to split a slave shard corresponding to the primary shard;

[0026] The slave shard corresponding to the master shard is determined as the shard to be split.

[0027] Optionally, after splitting the shard to be split based on the hard link file, the method further includes:

[0028] Determine that the first sub-shard and the second sub-shard are slave shards;

[0029] Based on a preset election rule, a primary shard is elected from the second sub-shards of the shard to be split and other second sub-shards, wherein the other second sub-shards include: second sub-shards obtained by splitting the primary shard corresponding to the shard to be split.

[0030] In a second aspect, an embodiment of the present invention provides a shard splitting device for a distributed table system, which is applied to a storage server of a distributed table system, wherein the distributed table system includes at least one table, each table includes multiple shards, and the multiple shards are stored in different storage servers, and the device includes:

[0031] A determination module, for determining a shard to be split from the plurality of shards;

[0032] An establishing module, for establishing a hard link file of the shard to be split determined by the determining module, wherein the hard link file is an index of the database file of the shard to be split;

[0033] A splitting module is used to split the shard to be split based on the hard link file established by the establishment module.

[0034] Optionally, the splitting module is specifically used for:

[0035] Create a first sub-shard and a second sub-shard;

[0036] Using the database file of the shard to be split as the database file of the first sub-shard, and setting the data range included in the database file of the first sub-shard to: the first specified range in the database file of the shard to be split;

[0037] The hard link file is used as the database file of the second sub-shard, and the data range included in the database file of the second sub-shard is set to: the second specified range in the database file of the shard to be split, the intersection of the first specified range and the second specified range is an empty set, and the union of the first specified range and the second specified range is the range corresponding to the shard to be split.

[0038] Optionally, the determining module is specifically used to:

[0039] Check the amount of data in each primary shard;

[0040] The primary shard whose data volume meets the preset conditions is determined as the shard to be split.

[0041] Optionally, the device further comprises: a generating module, a writing module and a detecting module;

[0042] The generating module is configured to generate a split request after determining a shard to be split from the multiple shards, wherein the split request is used to request to split a slave shard corresponding to the shard to be split;

[0043] The writing module is used to write the split request generated by the generating module into a log, and send the log to the slave server to which the slave shard corresponding to the shard to be split belongs;

[0044] The detection module is used to execute the step of establishing the hard link file of the shard to be split when it is detected that a preset number of slave servers have successfully stored the log.

[0045] Optionally, the device further comprises: an election module;

[0046] The determination module is further configured to determine, after the to-be-split shard is split based on the hard link file, that the first sub-shard is a primary shard and that the second sub-shard is a secondary shard;

[0047] The election module is used to elect a master shard from the second sub-shards and other second sub-shards of the shard to be split based on a preset election rule, where the other second sub-shards are: the second sub-shards obtained after the slave shard corresponding to the shard to be split is split.

[0048] Optionally, the determining module is specifically used to:

[0049] Receive a log sent by a master server to which a primary shard belongs, wherein the log includes a split request, and the split request is used to request to split a slave shard corresponding to the primary shard;

[0050] The slave shard corresponding to the master shard is determined as the shard to be split.

[0051] Optionally, the device further comprises: an election module;

[0052] The determination module is further configured to determine that the first sub-shard and the second sub-shard are slave shards after the shard to be split is split based on the hard link file;

[0053] The election module is used to elect a primary shard from the second sub-shards and other second sub-shards of the shard to be split based on a preset election rule, wherein the other second sub-shards include: second sub-shards obtained by splitting the primary shard corresponding to the shard to be split.

[0054] In a third aspect, an embodiment of the present invention provides a server, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0055] Memory, used to store computer programs;

[0056] The processor is used to implement the steps of any of the above-mentioned sharding methods of the distributed table system when executing the program stored in the memory.

[0057] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the shard splitting method of any of the above-mentioned distributed table systems are implemented.

[0058] In a fifth aspect, an embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned shard splitting methods for a distributed table system.

[0059] The technical solution of the embodiment of the present invention can at least bring the following beneficial effects: after determining the shard to be split, a hard link file of the shard to be split is established. Then, based on the hard link file, the shard to be split is split. Since the hard link file is an index of the database file of the shard to be split, and the time to create the file index is less than the time to copy the file, the time consumed for shard splitting based on the hard link file is shorter, which can reduce the time consumed for shard splitting and improve the availability of the write service.

[0060] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A flow chart of a shard splitting method for a distributed table system provided by an embodiment of the present invention;

[0063] Figure 2 A flow chart of another shard splitting method for a distributed table system provided by an embodiment of the present invention;

[0064] Figure 3 A flow chart of a method for determining a sub-shard role provided by an embodiment of the present invention;

[0065] Figure 4 A flow chart of another method for determining a sub-shard role provided by an embodiment of the present invention;

[0066] Figure 5 A schematic diagram of the structure of a shard splitting device for a distributed table system provided by an embodiment of the present invention;

[0067] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] In order to improve the availability of write services, an embodiment of the present application provides a shard splitting method for a distributed table system, which can be applied to a storage server of a distributed table system, wherein the distributed table system includes at least one table, each table includes multiple shards, and the multiple shards are stored in different storage servers. Figure 1 As shown, the method includes the following steps.

[0070] Step 101: determine a shard to be split from multiple shards.

[0071] Step 102, create a hard link file for the shard to be split.

[0072] The hard link file is an index of the database file of the shard to be split. The database file is used to store table data within the range corresponding to the shard to be split.

[0073] Step 103: Split the shard to be split based on the hard link file.

[0074] The technical solution of the embodiment of the present invention can at least bring the following beneficial effects: after determining the shard to be split, a hard link file of the shard to be split is established. Then, based on the hard link file, the shard to be split is split. Since the hard link file is an index of the database file of the shard to be split, and the time to create the file index is less than the time to copy the file, the time consumed for shard splitting based on the hard link file is shorter, which can reduce the time consumed for shard splitting and improve the availability of the write service.

[0075] It is understandable that in a distributed table system, a table will be horizontally sliced ​​to form multiple shards, and the database file of each shard stores a part of the table data. In order to make the data have a certain redundancy, a shard usually has multiple copies, including a copy with the role of leader and one or more copies with the role of follower. The copy with the role of leader is called the primary shard, and the copy with the role of follower is called the slave shard.

[0076] When the write service is running, the master server to which the primary shard belongs receives the write request sent by the client, then generates a log based on the write request and synchronizes the log to the slave server to which the slave shard belongs. After the slave server successfully synchronizes the log, it sends a message to the master server that the log is successfully synchronized. After the master server detects that the servers to which most replicas belong have successfully synchronized the log, it writes the data requested by the write request into its database file and sends a message to the client that the write is successful. The master server and the slave server are both storage servers in the distributed table system.

[0077] The slave server to which the slave shard belongs may process the write request after detecting that the master server to which the master shard belongs processes the write request. For example, the slave server may write the data requested to be written by the write request into its database file after the next log synchronization is successful, or may write the data into its database file at other times. The embodiment of the present invention does not specifically limit the time when the slave server writes the data into its database file.

[0078] For example, a shard has 5 copies. When the master server to which the primary shard belongs receives a message of successful log synchronization from the slave servers to which two slave shards belong, since the log also exists in the master server, it can be determined that the servers to which the three copies belong have successfully synchronized logs. At this time, the data requested by the write request can be written into its database file.

[0079] In the embodiment of the present invention, the shard to be split may be a primary shard. Based on this, the method of determining the shard to be split in step 101 includes: detecting the data volume of each primary shard, and then determining the primary shard whose data volume meets the preset condition as the shard to be split.

[0080] For example, a primary shard whose data volume in a database file exceeds 800 megabytes (MB) is determined as a shard to be split.

[0081] In the embodiment of the present invention, after the above step 101, the master server to which the master shard belongs may also notify the slave server to which the slave shard belongs to perform shard splitting. The following steps are included.

[0082] Step 1: Generate a split request, wherein the split request is used to request the splitting of the slave shard corresponding to the shard to be split.

[0083] For example, a shard of a table includes three replicas, namely, primary shard A, secondary shard B, and secondary shard C. When the shard to be split is primary shard A, the secondary shards corresponding to the shard to be split are secondary shard B and secondary shard C.

[0084] Step 2: Write the split request into a log and send the log to the slave server to which the slave shard corresponding to the shard to be split belongs.

[0085] Step three, when it is detected that a preset number of slave servers have successfully stored logs, execute the step of creating a hard link file of the shard to be split in the above step 102.

[0086] In combination with step 2 and step 3, in one implementation, the master server to which the shard to be split belongs sends a log to the slave server to which the slave shard belongs, and when the slave server stores the log successfully, it sends a message confirming the successful storage to the master server. When the master server receives a preset number of messages confirming the successful storage sent by the slave servers, the step of establishing a hard link file for the shard to be split in step 102 is executed.

[0087] Optionally, the preset number can be set to half of the total number of replica shards. For example, if the total number of replicas is 5, the preset number can be set to 3.

[0088] The technical solution of the embodiment of the present application can also bring the following beneficial effects: the master server can split the main shard when most slave servers receive logs successfully, without having to wait until all slave servers receive logs successfully before splitting the shards, thus saving the time spent on splitting and improving the efficiency of splitting.

[0089] In the embodiment of the present invention, the shard to be split may be a slave shard. Based on this, the method of determining the shard to be split in the above step 101 includes: receiving a log sent by a master server to which the master shard belongs, wherein the log includes a split request, and the split request is used to request to split the slave shard corresponding to the master shard. Then, the slave shard corresponding to the master shard is determined as the shard to be split.

[0090] In an embodiment of the present invention, after detecting that the master server processes the split request, the slave server may determine the shard to be split according to the split request, and then split the shard to be split.

[0091] Optionally, after the next successful log synchronization, the slave server can determine that the master server has processed the request in the last synchronized log, and the slave server can process the split request based on its last synchronized log.

[0092] The technical solution of the embodiment of the present application can also bring the following beneficial effects: the slave server can perform the shard splitting operation after detecting that the master server has performed the shard splitting operation, thereby reducing the situation where most slave servers have not synchronized logs but some slave servers have performed the shard splitting operation, thereby improving the consistency of data storage between replicas.

[0093] In the embodiments of the present invention, see Figure 2 The method of splitting the shard to be split in the above step 103 includes the following steps.

[0094] Step 201: Create a first sub-shard and a second sub-shard.

[0095] Step 202: Use the database file of the shard to be split as the database file of the first sub-shard, and set the data range included in the database file of the first sub-shard to: the first specified range in the database file of the shard to be split.

[0096] Step 203: Use the hard link file as the database file of the second sub-shard, and set the data range included in the database file of the second sub-shard to: the second specified range in the database file of the shard to be split.

[0097] The intersection of the first specified range and the second specified range is an empty set, and the union of the first specified range and the second specified range is the data range included in the database file of the shard to be split.

[0098] It can be understood that creating a hard link for file A is equivalent to linking a new file name to file A. The file with this new file name is called a hard link file. The file linked to the new file name is file A. It can be seen that the hard link file of the shard to be split and the database file of the shard to be split point to the same file. When modifying one of the hard link file and the database file, the other file will also be modified.

[0099] Exemplarily, in combination with the above steps 202 and 203, when the range corresponding to the shard to be split is [start_key, end_key], split_key can be determined, and the first specified range is determined to be [start_key, split_key), and the second specified range is determined to be [split_key, end_key].

[0100] Optionally, the difference between the amount of data covered by the first specified range and the amount of data covered by the second specified range may be set to not exceed a preset difference. in, Indicates taking an integer, for example,

[0101] The technical solution of the embodiment of the present invention can also bring the following beneficial effects: the embodiment of the present invention can split a shard into two sub-shards, and the amount of data corresponding to the sub-shards is smaller than the amount of data corresponding to the split shard, thereby improving the reading and writing speed of the shards and improving the reading and writing efficiency.

[0102] Optionally, after the above step 103, the embodiment of the present application can also set the role of each sub-shard in the following two ways.

[0103] Method 1: When the shard to be split is the primary shard, Figure 3 As shown, the role of each sub-shard can be determined through the following steps.

[0104] Step 301: determine the first sub-shard as the primary shard, and determine the second sub-shard as the secondary shard.

[0105] Optionally, the sub-shard that inherits the log of the shard to be split can be called the first sub-shard; and the sub-shard that does not inherit the log of the shard to be split can be called the second sub-shard.

[0106] Optionally, the first sub-shard and the second sub-shard of the slave shard corresponding to the master shard are both slave shards.

[0107] Optionally, the first sub-shard of the shard to be split and the first sub-shard of the slave shard may be determined as a group, and the second sub-shard of the shard to be split and the second sub-shard of the slave shard may be determined as a group. The database files of the sub-shards in the same group include the same data range.

[0108] Step 302: elect a primary shard from the second sub-shards of the shard to be split and other second sub-shards based on a preset election rule.

[0109] Among them, the other second sub-shards are: the second sub-shards obtained after the shard to be split corresponds to the shard to be split.

[0110] It can be understood that since the second sub-shard of the slave shard corresponding to the shard to be split is a slave shard, and the second sub-shard of the shard to be split is also a slave shard, there is no master shard in each second sub-shard, and the master shard needs to be determined through election.

[0111] In one implementation, the master server to which the shard to be split belongs can initiate a master election and send a master election message to the slave servers to which the slave shards belong. If a specified number of slave servers send messages agreeing to the master election, the master server determines that its own master election is successful and sets the second sub-shard of the shard to be split as the master shard.

[0112] Alternatively, the slave server to which the shard belongs can initiate the election of a master. If it receives a specified number of messages indicating that it agrees to the election, the slave server determines that its election is successful and sends a heartbeat message to the servers to which other replicas belong. After receiving the heartbeat message sent by the slave server, the server to which the shard to be split belongs determines that the second sub-shard in the slave server is the master shard.

[0113] Method 2: When the shard to be split is a slave shard, Figure 4 As shown, the role of each sub-shard can be determined through the following steps.

[0114] Step 401: Determine that the first sub-shard and the second sub-shard are slave shards.

[0115] Optionally, the sub-shard that inherits the log of the shard to be split can be called the first sub-shard; and the sub-shard that does not inherit the log of the shard to be split can be called the second sub-shard.

[0116] Optionally, the first sub-shard of the primary shard corresponding to the shard to be split is the primary shard, and the second sub-shard of the primary shard corresponding to the shard to be split is the secondary shard.

[0117] Step 402: elect a primary shard from the second sub-shards of the shard to be split and other second sub-shards based on a preset election rule.

[0118] The other second sub-shards include: second sub-shards obtained by splitting the main shard corresponding to the shard to be split. The other second sub-shards may also include: second sub-shards obtained by splitting other slave shards corresponding to the shard to be split.

[0119] It can be understood that since the second sub-shard of the primary shard corresponding to the shard to be split is a slave shard, the second sub-shards of other slave shards corresponding to the shard to be split are slave shards, and the second sub-shard of the shard to be split is also a slave shard, there is no primary shard in each second sub-shard, and the primary shard needs to be determined through election.

[0120] In one implementation, the slave server to which the shard to be split belongs can initiate a master election and send a master election message to the master server and other slave servers. If a specified number of messages indicating agreement to the master election are received, the slave server determines that its own master election is successful and determines the second sub-shard of the shard to be split as the master shard.

[0121] Alternatively, when the slave server to which the shard to be split belongs receives a heartbeat message sent by other servers, it is determined that the second sub-shard in the server that sends the heartbeat message is the master shard.

[0122] Optionally, after setting the role of the sub-shard of the shard to be split, the shard to be split can be deleted, and the read and write services of the first sub-shard and the second sub-shard can be started.

[0123] Optionally, after setting the roles of the sub-shards of the shard to be split, you can also start the delayed compression tasks of the first sub-shard and the second sub-shard. It is understandable that when the shard is split, it may also receive a write task, but the write operation cannot be performed at this time. When the shard is successfully split, the sub-shards are delayed compressed, that is, the data required to be written by the write tasks received during the period of shard splitting are written to the first sub-shard and the second sub-shard respectively. However, the data required to be written by the write tasks received during the period of shard splitting does not necessarily belong to the first specified range or the second specified range. Therefore, when the first sub-shard is delayed compressed, the data that does not belong to the first specified range needs to be deleted from its database file; when the second sub-shard is delayed compressed, the data that does not belong to the second specified range needs to be deleted from its database file.

[0124] Combination Figure 3and Figure 4 The following uses an example to compare the shard splitting method of the distributed table system provided in the embodiment of the present application with the shard splitting method in the related art.

[0125] Assume that a shard in a distributed table system has three copies, A, B, and C, where A is the primary shard, B is the secondary shard, and C is the secondary shard. The primary server to which A belongs detects that the data volume of A's database file meets the preset conditions and determines that A is the shard to be split. The sub-shards of A are AL and AR, the sub-shards of B are BL and BR, and C is not split. AL, BL, and C are a group, and AR and BR are a group.

[0126] In the related art, AL and AR are both set as the primary shard, and BL and BR are both set as the secondary shard. At this time, the specified data needs to be written into AR, AR writes the data into the log and synchronizes the log to BR, and AR and BR write the specified data. Assume that the server to which AL belongs crashes, and C is determined as the primary shard through master election. When C is elected as the master, it learns that the logs including the split request have been synchronized in AL and BL, so C starts to perform the shard splitting operation, and C is split into sub-shards CL and CR. Since C is the primary shard at this time, CL and CR are both set as the primary shard. At this time, AL, BL and CL become a group, and AR, BR and CR become a group. Since the above-mentioned specified data is not written into CR, but CR is the primary shard of the group of AR, BR and CR, when CR synchronizes data to AR and BR thereafter, the specified data is lost in the system.

[0127] In the embodiment of the present application, AL is set as the master shard, AR is set as the slave shard, and BL and BR are both set as slave shards. At this time, the specified data needs to be written into AR, AR writes the data into the log and synchronizes the log to BR, and AR and BR write the specified data. Assume that the server to which AL belongs is down, and C is determined to be the master shard through master election. When C is elected as the master, it is learned that the logs including the split request have been synchronized in AL and BL, so C starts to perform the shard splitting operation, and C is split into sub-shards CL and CR. Since C is the master shard at this time, CL is set as the master shard, and CR is set as the slave shard. At this time, AL, BL and CL become a group, and AR, BR and CR become a group. There is no master shard in the group of AR, BR and CR, so the master election begins. Since the specified data is written into the logs of AR and BR, the logs of AR and BR are newer than the logs of CR, and the master shard will be generated in AR and BR. The specified data exists in the database files of AR and BR, and the specified data is still in the system.

[0128] It can be seen from the above examples that the application of the technical solution of the embodiments of the present application can reduce data loss and better ensure the integrity of the data included in the distributed table system.

[0129] Corresponding to the above method embodiment, the embodiment of the present invention provides a shard splitting device for a distributed table system, which is applied to a storage server of the distributed table system. The distributed table system includes at least one table, each table includes multiple shards, and the multiple shards are stored in different storage servers. Figure 5 As shown, the device includes: a determination module 501, an establishment module 502 and a splitting module 503;

[0130] Determining module 501, determining a shard to be split from multiple shards;

[0131] Establishing module 502, because establishing the hard link file of the shard to be split determined by determining module 501, the hard link file is the index of the database file of the shard to be split;

[0132] The splitting module 503 is used to split the shard to be split based on the hard link file established by the establishing module 502.

[0133] Optionally, the splitting module 503 is specifically used for:

[0134] Create a first sub-shard and a second sub-shard;

[0135] The database file of the shard to be split is used as the database file of the first sub-shard, and the data range included in the database file of the first sub-shard is set to: the first specified range in the database file of the shard to be split;

[0136] The hard link file is used as the database file of the second sub-shard, and the data range included in the database file of the second sub-shard is set to: the second specified range in the database file of the shard to be split, the intersection of the first specified range and the second specified range is an empty set, and the union of the first specified range and the specified second range is the range corresponding to the shard to be split.

[0137] Optionally, the determination module 501 is specifically configured to:

[0138] Check the amount of data in each primary shard;

[0139] The primary shard whose data volume meets the preset conditions is determined as the shard to be split.

[0140] Optionally, the device further includes: a generating module 504, a writing module 505 and a detecting module 506;

[0141] A generating module 504 is used to generate a split request after determining a shard to be split from multiple shards, where the split request is used to request to split a slave shard corresponding to the shard to be split;

[0142] The writing module 505 is used to write the split request generated by the generating module 504 into a log, and send the log to the slave server to which the slave shard corresponding to the shard to be split belongs;

[0143] The detection module 506 is used to execute the step of establishing a hard link file of the shard to be split when it is detected that a preset number of slave servers have successfully stored logs.

[0144] Optionally, the device further includes: an election module 507;

[0145] The determination module 501 is further used to determine the first sub-shard as the primary shard and the second sub-shard as the secondary shard after the shard to be split is split based on the hard link file;

[0146] The election module 507 is used to elect a master shard from the second sub-shards of the shard to be split and other second sub-shards based on a preset election rule. The other second sub-shards are: the second sub-shards obtained after the slave shard corresponding to the shard to be split is split.

[0147] Optionally, the determination module 501 is specifically configured to:

[0148] Receive the log sent by the master server to which the primary shard belongs. The log includes a split request, which is used to request the split of the slave shard corresponding to the primary shard.

[0149] The slave shard corresponding to the primary shard is determined as the shard to be split.

[0150] Optionally, the device further includes: an election module 507;

[0151] The determination module 501 is further configured to determine the first sub-shard and the second sub-shard as slave shards after the shard to be split is split based on the hard link file;

[0152] The election module 507 is used to elect a primary shard from the second sub-shards and other second sub-shards of the shard to be split based on a preset election rule. The other second sub-shards include: the second sub-shards obtained after the primary shard corresponding to the shard to be split is split.

[0153] The embodiment of the present invention also provides a server, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0154] Memory 603, used for storing computer programs;

[0155] The processor 601 is used to implement the method steps in the above method embodiment when executing the program stored in the memory 603.

[0156] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0157] The communication interface is used for communication between the above electronic device and other devices.

[0158] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0159] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0160] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the shard splitting method of any of the above-mentioned distributed table systems are implemented.

[0161] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the shard splitting method of any distributed table system in the above embodiments.

[0162] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0163] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0164] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A sharding method for a distributed table system, characterized in that: A storage server applied to a distributed table system, wherein the distributed table system includes at least one table, each table includes multiple slices, and the multiple slices are stored in different storage servers, and the method includes: Determine a shard to be split from the multiple shards; Creating a hard link file of the shard to be split, wherein the hard link file is an index of the database file of the shard to be split; Based on the hard link file, split the shard to be split; In the case where the shard to be split is a primary shard, after the shard to be split is split based on the hard link file, the method further includes: Determine the first sub-shard as the primary shard, and determine the second sub-shard as the secondary shard, the first sub-shard being the sub-shard that inherits the log of the shard to be split after the shard to be split is split, and the second sub-shard being the sub-shard that does not inherit the log of the shard to be split after the shard to be split is split; Based on a preset election rule, a primary shard is elected from the second sub-shards and other second sub-shards of the shard to be split, where the other second sub-shards are: the second sub-shards obtained after the slave shard corresponding to the shard to be split is split.

2. The method according to claim 1, characterized in that The step of splitting the shard to be split based on the hard link file includes: Create a first sub-shard and a second sub-shard; Using the database file of the shard to be split as the database file of the first sub-shard, and setting the data range included in the database file of the first sub-shard to: the first specified range in the database file of the shard to be split; The hard link file is used as the database file of the second sub-shard, and the data range included in the database file of the second sub-shard is set to: the second specified range in the database file of the shard to be split, the intersection of the first specified range and the second specified range is an empty set, and the union of the first specified range and the second specified range is the data range included in the database file of the shard to be split.

3. The method according to claim 1, characterized in that The determining a shard to be split from the multiple shards includes: Check the amount of data in each primary shard; The primary shard whose data volume meets the preset conditions is determined as the shard to be split.

4. The method according to claim 3, characterized in that After determining the shard to be split from the multiple shards, the method further includes: Generate a split request, where the split request is used to request to split a slave shard corresponding to the shard to be split; Writing the split request into a log, and sending the log to a slave server to which the slave shard corresponding to the shard to be split belongs; When it is detected that a preset number of slave servers have successfully stored the log, the step of establishing the hard link file of the shard to be split is performed.

5. The method according to claim 2, characterized in that: The determining a shard to be split from the multiple shards includes: Receive a log sent by a master server to which a primary shard belongs, wherein the log includes a split request, and the split request is used to request to split a slave shard corresponding to the primary shard; The slave shard corresponding to the master shard is determined as the shard to be split.

6. The method according to claim 5, characterized in that After splitting the shard to be split based on the hard link file, the method further includes: Determine that the first sub-shard and the second sub-shard are slave shards; Based on a preset election rule, a primary shard is elected from the second sub-shards of the shard to be split and other second sub-shards, wherein the other second sub-shards include: second sub-shards obtained by splitting the primary shard corresponding to the shard to be split.

7. A sharding device for a distributed table system, characterized in that: A storage server applied to a distributed table system, wherein the distributed table system includes at least one table, each table includes multiple slices, and the multiple slices are stored in different storage servers, and the device includes: A determination module, for determining a shard to be split from the plurality of shards; An establishing module, for establishing a hard link file of the shard to be split determined by the determining module, wherein the hard link file is an index of the database file of the shard to be split; A splitting module, used for splitting the to-be-split shard based on the hard link file established by the establishing module; In the case where the shard to be split is a primary shard, the device further includes: an election module; The determination module is further configured to determine, after the shard to be split is split based on the hard link file, the first sub-shard as the primary shard and the second sub-shard as the secondary shard, the first sub-shard being the sub-shard that inherits the log of the shard to be split after the shard to be split is split, and the second sub-shard being the sub-shard that does not inherit the log of the shard to be split after the shard to be split is split; The election module is used to elect a master shard from the second sub-shards and other second sub-shards of the shard to be split based on a preset election rule, where the other second sub-shards are: the second sub-shards obtained after the slave shard corresponding to the shard to be split is split.

8. The device according to claim 7, characterized in that The splitting module is specifically used for: Create a first sub-shard and a second sub-shard; Using the database file of the shard to be split as the database file of the first sub-shard, and setting the data range included in the database file of the first sub-shard to: the first specified range in the database file of the shard to be split; The hard link file is used as the database file of the second sub-shard, and the data range included in the database file of the second sub-shard is set to: the second specified range in the database file of the shard to be split, the intersection of the first specified range and the second specified range is an empty set, and the union of the first specified range and the second specified range is the range corresponding to the shard to be split.

9. The device according to claim 7, characterized in that The determination module is specifically used for: Check the amount of data in each primary shard; The primary shard whose data volume meets the preset conditions is determined as the shard to be split.

10. The device according to claim 9, characterized in that The device also includes: a generating module, a writing module and a detecting module; The generating module is configured to generate a split request after determining a shard to be split from the multiple shards, wherein the split request is used to request to split a slave shard corresponding to the shard to be split; The writing module is used to write the split request generated by the generating module into a log, and send the log to the slave server to which the slave shard corresponding to the shard to be split belongs; The detection module is used to execute the step of establishing the hard link file of the shard to be split when it is detected that a preset number of slave servers have successfully stored the log.

11. The device according to claim 8, characterized in that The determining module is specifically used for: Receive a log sent by a master server to which a primary shard belongs, wherein the log includes a split request, and the split request is used to request to split a slave shard corresponding to the primary shard; The slave shard corresponding to the master shard is determined as the shard to be split.

12. The device according to claim 11, characterized in that The device also includes: an election module; The determination module is further configured to determine that the first sub-shard and the second sub-shard are slave shards after the shard to be split is split based on the hard link file; The election module is used to elect a primary shard from the second sub-shards and other second sub-shards of the shard to be split based on a preset election rule, wherein the other second sub-shards include: second sub-shards obtained by splitting the primary shard corresponding to the shard to be split.

13. A server, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images