Database splitting method and device

Abstract

The invention relates to a database splitting method and device. The method comprises the following steps: acquiring database information of an original database; wherein the original database is a key value database, and the database information comprises file attribute information of each disk file in the original database; the file attribute information of each magnetic disk file at least comprises a maximum key and a middle key in a key value pair stored in the magnetic disk file, layer information of a storage layer to which the magnetic disk file belongs in the original database, and the size of the magnetic disk file; according to the database information, selecting split keys used for splitting the original database from the key value pairs stored in the disk files; and splitting the original database according to the splitting key, the database information and the system characteristics of the file system corresponding to the original database. By adopting the method, the split key can be quickly determined, and the database can be quickly and effectively split.

Description

Technical Field

[0001] The present application relates to the technical field of data storage and processing, and in particular to a database splitting method and device. Background Art

[0002] In distributed databases, key-value pairs are usually used for storage. When users need to find data, they can use the primary key as an index to find the corresponding value. Taking Rocksdb as an example, data can be stored in the structure of LSM-tree (Log-Structured Merge-tree). However, as the amount of stored data continues to increase, the risk of database availability will increase, resulting in reduced system performance and write amplification. In order to avoid affecting the performance of the database, you can usually use database splitting to split a database instance into two database instances.

[0003] Currently, the splitting strategy based on index is used to split the database. However, in actual applications, the access to data is uneven, which makes the splitting strategy based on index ineffective and unable to solve the problem, resulting in unbalanced load of access requests, which in turn leads to unreasonable splitting points and inaccurate splitting of the two databases. Summary of the invention

[0004] Based on this, it is necessary to provide a database splitting method and device that can quickly determine the split key and quickly and effectively split the database in response to the above technical problems.

[0005] In a first aspect, the present application provides a database splitting method, comprising:

[0006] Obtaining database information of an original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0007] According to the database information, selecting a split key for splitting the original database from the key-value pairs stored in each of the disk files;

[0008] The original database is split according to the split key, the database information, and system characteristics of a file system corresponding to the original database.

[0009] In one embodiment, the obtaining of database information of the original database includes:

[0010] For any disk file in the original database, the last key stored in the disk file is used as the maximum key;

[0011] Selecting an intermediate key from the keys stored in the disk file;

[0012] Determine, according to a merge log generated during a merge operation on the disk file, layer information of a storage layer to which the disk file belongs in the original database;

[0013] A query is performed through the file system corresponding to the original database to obtain the disk file size of the disk file.

[0014] In one embodiment, selecting a split key for splitting the original database from the key-value pairs stored in each of the disk files according to the database information includes:

[0015] The sum of the disk file sizes of the disk files is used as the database capacity of the original database;

[0016] Taking half of the database capacity as reference capacity;

[0017] According to the maximum key and layer information corresponding to each of the disk files, all disk files belonging to the same storage layer are sorted in descending order to obtain sorting information of different storage layers in the original database;

[0018] A split key for splitting the original database is selected from the key-value pairs stored in each of the disk files according to the reference capacity and sorting information of different storage layers in the original database.

[0019] In one of the embodiments, the sorting information of each storage layer includes the arrangement sequence number corresponding to each disk file in the storage layer;

[0020] The selecting, from the key-value pairs stored in each of the disk files, a split key for splitting the original database according to the reference capacity and the sorting information of different storage layers in the original database, comprises:

[0021] For the current round, the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer is used as the candidate disk file;

[0022] Deleting the arrangement sequence number corresponding to the candidate disk file from the sorting information of the storage layer to which the candidate disk file belongs;

[0023] The sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round; wherein, if the current round is the first round, the previous file size is a set value;

[0024] If the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as a split key for splitting the original database.

[0025] In one embodiment, the method further comprises:

[0026] If the current file size is smaller than the reference capacity, the current round is taken as the new previous round, the current file size is taken as the new previous file size, and the next round is taken as the new current round. The operation of returning to execute the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement number in each storage layer as the candidate disk file is performed.

[0027] In one embodiment, splitting the original database according to the split key, the database information, and a system characteristic of a file system corresponding to the original database includes:

[0028] Creating an initial database according to the system characteristics of the file system corresponding to the original database;

[0029] According to the split key and the database information, data processing is performed on the initial database and the original database to obtain a first database and a second database obtained by splitting the original database.

[0030] In one embodiment, creating the initial database according to the system characteristics of the file system corresponding to the original database includes:

[0031] If it is determined according to the system characteristics of the file system that the file system supports a hard link function, hard links of the disk files in the original database are generated, and a new database directory of the initial database is created according to the hard links;

[0032] If it is determined according to the system characteristics of the file system that the file system does not support the hard link function, a new database directory of the initial database is created according to the split key and the file attribute information of each disk file in the original database.

[0033] In one embodiment, the file attribute information of each disk file further includes the smallest key in the key-value pair stored in the disk file;

[0034] The step of creating a new database directory of the initial database according to the split key and the file attribute information of each disk file in the original database comprises:

[0035] Create an initial database directory for the initial database;

[0036] Using the disk file whose minimum key is greater than the split key as a migration file, and migrating the file directory of the migration file in the original database directory corresponding to the original database to the initial database directory; and

[0037] The disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key is used as a copy file, and the file directory of the copy file in the original database directory is copied to the initial database directory to create a new database directory for the initial database.

[0038] In one embodiment, the performing data processing on the initial database and the original database according to the split key and the database information to obtain the first database and the second database obtained by splitting the original database includes:

[0039] Based on a database writing mechanism, the key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and

[0040] The key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

[0041] In a second aspect, the present application further provides a database splitting device, including:

[0042] An information acquisition module is used to acquire database information of an original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0043] A split key determination module, configured to select a split key for splitting the original database from the key-value pairs stored in each of the disk files according to the database information;

[0044] The database splitting module is used to split the original database according to the splitting key, the database information and the system characteristics of the file system corresponding to the original database.

[0045] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0046] Obtaining database information of an original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0047] According to the database information, selecting a split key for splitting the original database from the key-value pairs stored in each of the disk files;

[0048] The original database is split according to the split key, the database information, and system characteristics of a file system corresponding to the original database.

[0049] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0050] Obtaining database information of an original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0051] According to the database information, selecting a split key for splitting the original database from the key-value pairs stored in each of the disk files;

[0052] The original database is split according to the split key, the database information, and system characteristics of a file system corresponding to the original database.

[0053] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0054] Obtaining database information of an original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0055] According to the database information, selecting a split key for splitting the original database from the key-value pairs stored in each of the disk files;

[0056] The original database is split according to the split key, the database information, and system characteristics of a file system corresponding to the original database.

[0057] The above-mentioned database splitting method and device can quickly select a split key for splitting the original database from the key-value pairs stored in each disk file based on the file attribute information of each disk file in the original database (i.e., the key-value database), wherein the file attribute information of each disk file at least includes the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file; further, in combination with the system characteristics of the file system corresponding to the original database, and based on the split key and database information, the original database is split, thereby improving the flexibility of the classification process and further ensuring the feasibility and efficiency of the process of splitting the original database. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A schematic diagram of a database storage structure in one embodiment;

[0060] Figure 2 A schematic diagram of a process of splitting a database in one embodiment;

[0061] Figure 3 A schematic diagram of a process of obtaining database information of an original database in one embodiment;

[0062] Figure 4 A schematic diagram of a process for determining a split bond in one embodiment;

[0063] Figure 5 A schematic diagram of a process for determining a split bond in another embodiment;

[0064] Figure 6 A schematic diagram of a process of splitting an original database in one embodiment;

[0065] Figure 7 A schematic diagram of a process for creating a new database directory for an initial database in one embodiment;

[0066] Figure 8 is a flowchart of a database splitting method in another embodiment;

[0067] Fig. 9 is a structural block diagram of a database splitting device in one embodiment;

[0068] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] The database splitting method provided in the embodiment of the present application can be applied to an application environment for splitting a database, especially to an application environment for splitting a key-value database. In the embodiment of the present application, the key-value database can be a database using an LSM-tree structure, such as a RocksDB database. The database splitting method provided in the embodiment of the present application can be executed by a computer device, which can be a server or a terminal with powerful computing capabilities.

[0071] Optionally, for a key-value database using an LSM-tree structure (such as RocksDB), the storage structure is as follows: Figure 1 As shown in the figure, LSM-tree consists of layers. After data is written to MemTable (Memory Table), it is persisted as SSTable (Sorted String Table) file of Level0 layer, that is, disk file. As the number or capacity of SSTable of Level0 reaches the threshold, the SSTable in Level0 is merged (Compaction) to Level1. When the total amount of data in Level1 reaches a certain threshold, it is merged to Level2, and so on to higher layers.

[0072] Among them, MemTable is used to temporarily store key-value pairs that are about to be written to disk. Level refers to the hierarchical structure of data storage in LSM-tree, where data starts from the memory table (MemTable) and gradually moves down to deeper disk storage levels. LSM-tree maintains and optimizes the data structure by storing data on different levels and performing data compaction regularly. Except for Level0, the SSTable files in other levels do not overlap, and the key range contained in each file is unique.

[0073] In one embodiment, Figure 2 As shown, a database splitting method is provided, which is described by taking the method applied to a server as an example, and specifically includes the following steps:

[0074] S201, obtaining database information of the original database.

[0075] Among them, the original database is a key-value database. In an embodiment of the present application, the original database can be a RocksDB database; the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file.

[0076] Among them, the disk file is a file used to store key-value pairs stored in the SSTable (Sorted String Table) persistent data structure. The data in the disk file is stored in the sorted order of the keys.

[0077] Optionally, you can obtain the file attribute information of each disk file from the Manifest file of the original database. The Manifest file is used to record the status and structure information of the database. It contains the list of SSTables, the layer information of SSTable files, the compression strategy, and other key database configuration information. The Manifest file records each database change operation, such as adding or deleting files, to ensure that the database can accurately rebuild its state during failure recovery.

[0078] S202: According to the database information, a split key for splitting the original database is selected from the key-value pairs stored in each disk file.

[0079] The split key is the split point when splitting the database according to the size of the key in lexicographic order. For example, all key-value pairs in the database whose lexicographic order is less than or equal to the split key can be stored in one database instance, and all key-value pairs greater than the split key can be stored in another database instance.

[0080] Optionally, a split key that can divide the original database into equal parts can be selected from the key-value pairs stored in each disk file based on the database information to ensure that the amount of data stored in the two databases after the split is approximately equal.

[0081] Optionally, the key-value pairs stored in each disk file can be analyzed based on the maximum key and middle key in the key-value pairs stored in the disk file in the file attribute information of each disk file in the database information, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file, so as to select a split key for splitting the original database from the key-value pairs stored in each disk file.

[0082] S203, splitting the original database according to the split key, database information, and system characteristics of the file system corresponding to the original database.

[0083] The system characteristics of the file system corresponding to the original database represent whether the file system corresponding to the original database supports hard links or supports cross-file system transmission.

[0084] Optionally, the system characteristics of the file system corresponding to the original database are different, and accordingly, the method of splitting the original database according to the split key and the database information is also different.

[0085] Optionally, if it is determined based on the system characteristics of the file system corresponding to the original database that the file system supports hard links, the key-value pair data stored in the original database can be split according to the split key using hard links to obtain two new databases that split the original database.

[0086] Optionally, if it is determined based on the system characteristics of the file system corresponding to the original database that the file system does not support hard links, it is necessary to manually migrate and copy the key-value pairs stored in the original database based on the split key and database information to obtain two new databases that split the original database.

[0087] In the above database splitting method, based on the file attribute information of each disk file in the original database (i.e., the key-value database), wherein the file attribute information of each disk file at least includes the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file, a split key for splitting the original database can be quickly selected from the key-value pairs stored in each disk file; further, in combination with the system characteristics of the file system corresponding to the original database, and based on the split key and database information, the original database is split, thereby improving the flexibility of the classification process and further ensuring the feasibility and efficiency of the process of splitting the original database.

[0088] Optionally, in one embodiment, Figure 3 As shown, a method for obtaining database information of an original database is provided, which specifically includes the following steps:

[0089] S301, for any disk file in the original database, the last key stored in the disk file is used as the maximum key.

[0090] The maximum key of any disk file is the key with the largest lexicographic order among the key-value pairs stored in the disk file.

[0091] Optionally, for any disk file in the original database, since the data organization of the key-value pairs stored in the disk file is in ascending order, the last key stored in the disk file can be used as the maximum key.

[0092] S302: Select an intermediate key from the keys stored in the disk file.

[0093] Optionally, two pointers (fast pointer and slow pointer) can be introduced to traverse from the beginning of the disk file. Every time the fast pointer moves back two keys, the slow pointer moves back one key. Finally, when the fast pointer moves to the last key of the disk file, the key pointed to by the slow pointer is the middle key.

[0094] S303: Determine the layer information of the storage layer to which the disk file belongs in the original database according to the merge log generated in the process of performing the merge operation on the disk file.

[0095] Optionally, when a disk file is written to a new disk file of the next layer due to a merge operation, a merge log is generated, and the layer information of the storage layer to which the disk file belongs in the original database can be obtained from the merge log.

[0096] S304, querying through the file system corresponding to the original database to obtain the disk file size of the disk file.

[0097] Optionally, the file query function provided in the file system corresponding to the original database may be used to query the disk file size of the disk file.

[0098] In this embodiment, by introducing the maximum key, intermediate key, layer information and disk file size of each disk file, a data basis is provided for determining the split key; at the same time, a method for determining the maximum key, intermediate key, layer information and disk file size of each disk file is provided.

[0099] Optionally, in one embodiment, Figure 4 As shown, a method for determining a split key for splitting an original database is provided, which specifically includes the following steps:

[0100] S401, taking the sum of the disk file sizes of the disk files as the database capacity of the original database.

[0101] The database capacity is the total capacity of all disk files stored in the original database.

[0102] Optionally, after obtaining the disk file size of each disk file, the disk file size of each disk file may be added together, and the sum obtained by the addition is used as the database capacity of the original database.

[0103] S402: Use half of the database capacity as a reference capacity.

[0104] The reference capacity is the capacity used as a reference when splitting the original database.

[0105] Optionally, in order to ensure that the file sizes of disk files stored in the two databases obtained after splitting the original database are similar, half of the database capacity can be used as a reference capacity.

[0106] S403, according to the maximum key and layer information corresponding to each disk file, all disk files belonging to the same storage layer are sorted in descending order to obtain sorting information of different storage layers in the original database.

[0107] The sorting information of each storage layer may include but is not limited to the arrangement sequence numbers corresponding to each disk file in the storage layer.

[0108] Optionally, for all disk files, each disk file is classified according to the layer information, that is, the disk files of the same storage layer are classified into one category, and all disk files corresponding to each storage layer are sorted in descending order according to the size of the largest key, so as to obtain the sorting information of different storage layers in the original database.

[0109] S404 , selecting a split key for splitting the original database from the key-value pairs stored in each disk file according to the reference capacity and the sorting information of different storage layers in the original database.

[0110] Optionally, based on the sorting information of different storage layers in the original database, a portion of disk files can be selected from different storage layers to ensure that the total size of the disk files of the selected portion does not exceed and is close to the reference capacity; further, based on the sorting information of the selected portion of disk files, a split key for splitting the original database is selected from the key-value pairs stored in the selected portion of disk files.

[0111] In this embodiment, by introducing the reference capacity and the sorting information of different storage layers in the original database, the accuracy of the determined split key is guaranteed, that is, it is guaranteed that the amount of data stored in the two databases obtained by splitting the original database according to the split key is approximately equal.

[0112] Optionally, the sorting information of each storage layer includes the sorting sequence number corresponding to each disk file in the storage layer. In this case, in one embodiment, Figure 5 As shown, a method for determining a split key for splitting an original database is provided, which specifically includes the following steps:

[0113] S501 , for the current round, taking the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer as a candidate disk file.

[0114] Optionally, in the current round, for each storage layer, the disk file with the smallest arrangement number in each storage layer is extracted, and the maximum keys corresponding to the disk files extracted from each storage layer are compared; further, the disk file with the largest maximum key value is selected as a candidate disk file.

[0115] S502: Delete the arrangement sequence number corresponding to the candidate disk file from the sorting information of the storage layer to which the candidate disk file belongs.

[0116] Optionally, after determining the candidate disk file, the layer to which the candidate disk file belongs is determined according to the layer information of the candidate disk file, and further, the arrangement sequence number corresponding to the candidate disk file is deleted from the sorting information of the storage layer to which the candidate disk file belongs.

[0117] S503: The sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round.

[0118] If the current round is the first round, the previous file size is a set value, and in the embodiment of the present application, the set value is 0. The previous round is the previous round of the current round.

[0119] Optionally, the disk file size of the candidate disk file is determined, and the disk file size is accumulated with the previous file size corresponding to the previous round, and the accumulated sum is used as the current file size corresponding to the current round.

[0120] S504: If the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as a split key for splitting the original database.

[0121] Optionally, if the current file size is equal to or greater than the reference capacity, it means that the sum of the file sizes of the candidate disk files extracted in each round is greater than the reference capacity. At this time, the intermediate key of the candidate disk file can be used as the split key for splitting the original database.

[0122] Optionally, if the current file size is smaller than the reference capacity, the current round is used as the new previous round, the current file size is used as the new previous file size, and the next round is used as the new current round, and the operation of returning to execute the operation of using the disk file corresponding to the smallest arrangement number in each storage layer and the largest maximum key as the candidate disk file. That is, when the current file size is smaller than the reference capacity, the next round is started until the current file size is equal to or greater than the reference capacity.

[0123] In this embodiment, by introducing the arrangement sequence number corresponding to each disk file in each storage layer, and performing multiple rounds of traversal on the disk files in each storage layer based on the arrangement sequence number, the accuracy of the determined split key is guaranteed.

[0124] Optionally, in one embodiment, Figure 6 A method for splitting an original database is provided, which specifically includes the following steps:

[0125] S601, creating an initial database according to the system characteristics of the file system corresponding to the original database.

[0126] The initial database is an empty database that is created.

[0127] Optionally, the system characteristics of the file system corresponding to the original database are different, and the creation strategy of the corresponding initial database is also different.

[0128] Optionally, if it is determined based on the system characteristics of the file system that the file system supports the hard link function, a hard link is generated for each disk file in the original database, and a new database directory for the initial database is created based on each hard link. Specifically, the new database directory for the initial database copies the Manifest file of the original database to the new database directory, generates hard links for all disk files in the original database directory of the original database, and stores the hard links in the new database directory to create the initial database. It can be understood that in a file system that supports hard links, the original database is split by hard linking, which improves the convenience and accuracy of the splitting process.

[0129] Optionally, if it is determined based on the system characteristics of the file system that the file system does not support the hard link function, a new database directory for the initial database is created based on the split key and the file attribute information of each disk file in the original database. Specifically, if it is determined based on the system characteristics of the file system that the file system does not support the hard link function, then the action of splitting the original database to generate a new database needs to rely on file copying. In this scenario, the process of splitting to generate a new Rocksdb database is optimized to minimize file copying. In the worst case, the optimized split action will only copy all files of Level 0 + one file at each layer + one Manifest file. This greatly reduces the number of file copies.

[0130] S602: Perform data processing on the initial database and the original database according to the split key and the database information to obtain a first database and a second database obtained by splitting the original database.

[0131] Optionally, based on a database writing mechanism, key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

[0132] Specifically, based on the database writing mechanism, a CompactionFilter is set for the original database. The rule of the CompactionFilter is to retain all key-value pairs whose lexicographic order is less than or equal to the split key, and perform Compaction to obtain the first database. Furthermore, a CompactionFilter can be set for the initial database. The rule of the CompactionFilter is to retain all key-value pairs whose lexicographic order is greater than the split key, and perform Compaction to obtain the first database.

[0133] At this point, the first database stores all key-value pair data that is less than or equal to the classification key, and the second database stores all key-value pair data that is greater than the split key, and the splitting of the original database is completed.

[0134] In this embodiment, an initial database is created according to the system characteristics of the file system corresponding to the original database, thereby ensuring the applicability of the created initial database. At the same time, data processing is performed on the initial database and the original database according to the splitting key and database information, thereby ensuring the accuracy of the first database and the second database obtained by the splitting.

[0135] Optionally, the file attribute information of each disk file also includes the smallest key in the key-value pair stored in the disk file; in this case, in one embodiment, Figure 7 As shown, a method for creating a new database directory of an initial database is provided, which specifically includes the following steps:

[0136] S701, creating an initial database directory of an initial database.

[0137] Optionally, create an initial database directory for the initial database, and copy the manifest of the original database to the initial database directory.

[0138] S702: Use the disk file whose minimum key is greater than the split key as a migration file, and migrate the file directory of the migration file in the original database directory corresponding to the original database to the initial database directory.

[0139] Optionally, a disk file with a minimum key greater than the split key is selected as a migration file, and the file directory of the migration file in the original database directory corresponding to the original database is migrated to the initial database directory. Furthermore, a deletion record of the migration file is appended to the Manifest file of the original database.

[0140] S703, taking a disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key as a copy file, and copying the file directory of the copy file in the original database directory to the initial database directory to create a new database directory for the initial database.

[0141] Optionally, a disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key is selected as a copy file, and the file directory of the copy file in the original database directory is copied to the initial database directory. Furthermore, deletion records of all files in all disk files except the migration file and the copy file are appended to the Manifest in the initial database.

[0142] In this embodiment, in a file system that does not support hard links, the original database is split by copying and duplicating, thereby optimizing the original database splitting process and reducing the amount of data copying.

[0143] Figure 8FIG. 2 is a flow chart of a database splitting method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a database splitting method. Figure 8 The specific implementation process is as follows:

[0144] S801, taking the last key stored in each disk file in the original database as the maximum key of each disk file.

[0145] S802, selecting the intermediate key of each disk file from the keys stored in each disk file.

[0146] S803, determining the layer information of the storage layer to which each disk file belongs in the original database according to the merge log generated in the process of executing the merge operation on each disk file.

[0147] S804, querying through the file system corresponding to the original database to obtain the disk file size of each disk file.

[0148] S805: The sum of the disk file sizes of the disk files is used as the database capacity of the original database.

[0149] S806: Use half of the database capacity as a reference capacity.

[0150] S807, according to the maximum key and layer information corresponding to each disk file, all disk files belonging to the same storage layer are sorted in descending order to obtain sorting information of different storage layers in the original database.

[0151] S808 , selecting a split key for splitting the original database from the key-value pairs stored in each disk file according to the reference capacity and the sorting information of different storage layers in the original database.

[0152] The sorting information of each storage layer includes the arrangement sequence number corresponding to each disk file in the storage layer.

[0153] Optionally, for the current round, the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement number in each storage layer is used as a candidate disk file; the arrangement number corresponding to the candidate disk file is deleted from the sorting information of the storage layer to which the candidate disk file belongs; the sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round; wherein, if the current round is the first round, the previous file size is a set value; if the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as the split key for splitting the original database.

[0154] Optionally, if the current file size is smaller than the reference capacity, the current round is used as the new previous round, the current file size is used as the new previous file size, and the next round is used as the new current round. The operation of returning to execute the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer as the candidate disk file is performed.

[0155] S809: Create an initial database according to the system characteristics of the file system corresponding to the original database.

[0156] Optionally, if it is determined based on the system characteristics of the file system that the file system supports the hard link function, a hard link is generated for each disk file in the original database, and a new database directory for the initial database is created based on each hard link; if it is determined based on the system characteristics of the file system that the file system does not support the hard link function, a new database directory for the initial database is created based on the split key and the file attribute information of each disk file in the original database.

[0157] Optionally, create an initial database directory for the initial database; use disk files whose minimum keys are greater than the split key as migration files, and migrate the file directory of the migration files in the original database directory corresponding to the original database to the initial database directory; and use disk files whose minimum keys are less than or equal to the split key and whose maximum keys are greater than the split key as copy files, and copy the file directory of the copy files in the original database directory to the initial database directory to create a new database directory for the initial database.

[0158] S810, performing data processing on the initial database and the original database according to the split key and the database information to obtain a first database and a second database obtained by splitting the original database.

[0159] Optionally, based on a database writing mechanism, key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

[0160] The specific process of the above S801-S810 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0161] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0162] Based on the same inventive concept, the embodiment of the present application also provides a database splitting device for implementing the database splitting method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more database splitting device embodiments provided below can refer to the limitations of the database splitting method above, and will not be repeated here.

[0163] In an exemplary embodiment, Fig. 9 As shown, a database splitting device 900 is provided, comprising: an information acquisition module 910, a split key determination module 920 and a database splitting module 930, wherein:

[0164] The information acquisition module 910 is used to obtain the database information of the original database; wherein the original database is a key-value database, and the database information includes the file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file.

[0165] The split key determination module 920 is used to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the database information.

[0166] The database splitting module 930 is used to split the original database according to the splitting key, database information and system characteristics of the file system corresponding to the original database.

[0167] The above database splitting device,

[0168] In one embodiment, the information acquisition module 910 is specifically used to:

[0169] For any disk file in the original database, the last key stored in the disk file is used as the maximum key; the middle key is selected from the keys stored in the disk file; the layer information of the storage layer to which the disk file belongs in the original database is determined based on the merge log generated during the merge operation on the disk file; and the file system corresponding to the original database is queried to obtain the disk file size of the disk file.

[0170] In one embodiment, the split key determination module 920 includes:

[0171] The first determining unit is used to use the sum of the disk file sizes of the disk files as the database capacity of the original database.

[0172] The second determining unit is configured to use half of the database capacity as a reference capacity.

[0173] The information determination unit is used to sort all disk files belonging to the same storage layer in descending order according to the maximum key and layer information corresponding to each disk file, so as to obtain the sorting information of different storage layers in the original database.

[0174] The split key determination unit is used to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the reference capacity and the sorting information of different storage layers in the original database.

[0175] In one embodiment, the sorting information of each storage layer includes the arrangement sequence number corresponding to each disk file in the storage layer; the split key determination unit is specifically used to:

[0176] For the current round, the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement number in each storage layer is used as the candidate disk file; the arrangement number corresponding to the candidate disk file is deleted from the sorting information of the storage layer to which the candidate disk file belongs; the sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round; wherein, if the current round is the first round, the previous file size is a set value; if the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as the split key for splitting the original database.

[0177] In one embodiment, the split key determination unit is further configured to:

[0178] If the current file size is smaller than the reference capacity, the current round is used as the new previous round, the current file size is used as the new previous file size, and the next round is used as the new current round. The operation of returning to execute the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer as the candidate disk file is performed.

[0179] In one embodiment, the database splitting module 930 includes:

[0180] The first creation unit is used to create an initial database according to the system characteristics of the file system corresponding to the original database.

[0181] The second creation unit is used to perform data processing on the initial database and the original database according to the split key and the database information to obtain a first database and a second database obtained by splitting the original database.

[0182] In one embodiment, the first creation unit specifically includes:

[0183] The first slave unit is used to generate hard links for each disk file in the original database if it is determined according to the system characteristics of the file system that the file system supports the hard link function, and to create a new database directory for the initial database according to each hard link.

[0184] The second slave unit is used to create a new database directory of the initial database according to the split key and the file attribute information of each disk file in the original database if it is determined according to the system characteristics of the file system that the file system does not support the hard link function.

[0185] In one embodiment, the file attribute information of each disk file also includes the smallest key in the key-value pair stored in the disk file; the second slave unit is specifically used for:

[0186] An initial database directory of the initial database is created; a disk file whose minimum key is greater than the split key is used as a migration file, and the file directory of the migration file in the original database directory corresponding to the original database is migrated to the initial database directory; and a disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key is used as a copy file, and the file directory of the copy file in the original database directory is copied to the initial database directory to create a new database directory for the initial database.

[0187] In one embodiment, the second creation unit is specifically used for:

[0188] Based on the database writing mechanism, key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

[0189] Each module in the above-mentioned database splitting device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0190] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a database splitting method is implemented.

[0191] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0192] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0193] Obtaining database information of the original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0194] According to the database information, a split key for splitting the original database is selected from the key-value pairs stored in each disk file;

[0195] The original database is split according to the split key, database information, and system characteristics of the file system corresponding to the original database.

[0196] In one embodiment, when the processor executes the computer program to obtain database information of the original database, the processor further implements the following steps:

[0197] For any disk file in the original database, the last key stored in the disk file is used as the maximum key; the middle key is selected from the keys stored in the disk file; the layer information of the storage layer to which the disk file belongs in the original database is determined based on the merge log generated during the merge operation on the disk file; and the file system corresponding to the original database is queried to obtain the disk file size of the disk file.

[0198] In one embodiment, when the processor executes the computer program to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the database information, the following steps are also implemented:

[0199] The sum of the disk file sizes of each disk file is used as the database capacity of the original database; half of the database capacity is used as the reference capacity; according to the maximum key and layer information corresponding to each disk file, all disk files belonging to the same storage layer are sorted in descending order to obtain the sorting information of different storage layers in the original database; according to the reference capacity and the sorting information of different storage layers in the original database, a split key for splitting the original database is selected from the key-value pairs stored in each disk file.

[0200] In one embodiment, the sorting information of each storage layer includes the arrangement sequence number corresponding to each disk file in the storage layer; when the processor executes the computer program to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the reference capacity and the sorting information of different storage layers in the original database, the following steps are also implemented:

[0201] For the current round, the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement number in each storage layer is used as the candidate disk file; the arrangement number corresponding to the candidate disk file is deleted from the sorting information of the storage layer to which the candidate disk file belongs; the sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round; wherein, if the current round is the first round, the previous file size is a set value; if the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as the split key for splitting the original database.

[0202] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0203] If the current file size is smaller than the reference capacity, the current round is used as the new previous round, the current file size is used as the new previous file size, and the next round is used as the new current round. The operation of returning to execute the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer as the candidate disk file is performed.

[0204] In one embodiment, when the processor executes the computer program to split the original database according to the split key, the database information, and the system characteristics of the file system corresponding to the original database, the following steps are also implemented:

[0205] An initial database is created according to the system characteristics of the file system corresponding to the original database; data processing is performed on the initial database and the original database according to the split key and database information to obtain a first database and a second database obtained by splitting the original database.

[0206] In one embodiment, when the processor executes the computer program to create the initial database according to the system characteristics of the file system corresponding to the original database, the processor further implements the following steps:

[0207] If it is determined based on the system characteristics of the file system that the file system supports the hard link function, a hard link is generated for each disk file in the original database, and a new database directory for the initial database is created based on each hard link; if it is determined based on the system characteristics of the file system that the file system does not support the hard link function, a new database directory for the initial database is created based on the split key and the file attribute information of each disk file in the original database.

[0208] In one embodiment, the file attribute information of each disk file also includes the smallest key in the key-value pair stored in the disk file; when the processor executes the computer program to create a new database directory of the initial database according to the split key and the file attribute information of each disk file in the original database, the processor also implements the following steps:

[0209] An initial database directory of the initial database is created; a disk file whose minimum key is greater than the split key is used as a migration file, and the file directory of the migration file in the original database directory corresponding to the original database is migrated to the initial database directory; and a disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key is used as a copy file, and the file directory of the copy file in the original database directory is copied to the initial database directory to create a new database directory for the initial database.

[0210] In one embodiment, when the processor executes the computer program to perform data processing on the initial database and the original database according to the split key and the database information to obtain the first database and the second database obtained by splitting the original database, the following steps are also implemented:

[0211] Based on the database writing mechanism, key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

[0212] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0213] Obtaining database information of the original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0214] According to the database information, a split key for splitting the original database is selected from the key-value pairs stored in each disk file;

[0215] The original database is split according to the split key, database information, and system characteristics of the file system corresponding to the original database.

[0216] In one embodiment, when the processor executes the computer program to obtain database information of the original database, the processor further implements the following steps:

[0217] For any disk file in the original database, the last key stored in the disk file is used as the maximum key; the middle key is selected from the keys stored in the disk file; the layer information of the storage layer to which the disk file belongs in the original database is determined based on the merge log generated during the merge operation on the disk file; and the file system corresponding to the original database is queried to obtain the disk file size of the disk file.

[0218] In one embodiment, when the processor executes the computer program to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the database information, the following steps are also implemented:

[0219] The sum of the disk file sizes of each disk file is used as the database capacity of the original database; half of the database capacity is used as the reference capacity; according to the maximum key and layer information corresponding to each disk file, all disk files belonging to the same storage layer are sorted in descending order to obtain the sorting information of different storage layers in the original database; according to the reference capacity and the sorting information of different storage layers in the original database, a split key for splitting the original database is selected from the key-value pairs stored in each disk file.

[0220] In one embodiment, the sorting information of each storage layer includes the arrangement sequence number corresponding to each disk file in the storage layer; when the processor executes the computer program to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the reference capacity and the sorting information of different storage layers in the original database, the following steps are also implemented:

[0221] For the current round, the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement number in each storage layer is used as the candidate disk file; the arrangement number corresponding to the candidate disk file is deleted from the sorting information of the storage layer to which the candidate disk file belongs; the sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round; wherein, if the current round is the first round, the previous file size is a set value; if the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as the split key for splitting the original database.

[0222] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0223] If the current file size is smaller than the reference capacity, the current round is used as the new previous round, the current file size is used as the new previous file size, and the next round is used as the new current round. The operation of returning to execute the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer as the candidate disk file is performed.

[0224] In one embodiment, when the processor executes the computer program to split the original database according to the split key, the database information, and the system characteristics of the file system corresponding to the original database, the following steps are also implemented:

[0225] An initial database is created according to the system characteristics of the file system corresponding to the original database; data processing is performed on the initial database and the original database according to the split key and database information to obtain a first database and a second database obtained by splitting the original database.

[0226] In one embodiment, when the processor executes the computer program to create the initial database according to the system characteristics of the file system corresponding to the original database, the processor further implements the following steps:

[0227] If it is determined based on the system characteristics of the file system that the file system supports the hard link function, a hard link is generated for each disk file in the original database, and a new database directory for the initial database is created based on each hard link; if it is determined based on the system characteristics of the file system that the file system does not support the hard link function, a new database directory for the initial database is created based on the split key and the file attribute information of each disk file in the original database.

[0228] In one embodiment, the file attribute information of each disk file also includes the smallest key in the key-value pair stored in the disk file; when the processor executes the computer program to create a new database directory of the initial database according to the split key and the file attribute information of each disk file in the original database, the processor also implements the following steps:

[0229] An initial database directory of the initial database is created; a disk file whose minimum key is greater than the split key is used as a migration file, and the file directory of the migration file in the original database directory corresponding to the original database is migrated to the initial database directory; and a disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key is used as a copy file, and the file directory of the copy file in the original database directory is copied to the initial database directory to create a new database directory for the initial database.

[0230] In one embodiment, when the processor executes the computer program to perform data processing on the initial database and the original database according to the split key and the database information to obtain the first database and the second database obtained by splitting the original database, the following steps are also implemented:

[0231] Based on the database writing mechanism, key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

[0232] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0233] Obtaining database information of the original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file;

[0234] According to the database information, a split key for splitting the original database is selected from the key-value pairs stored in each disk file;

[0235] The original database is split according to the split key, database information, and system characteristics of the file system corresponding to the original database.

[0236] In one embodiment, when the processor executes the computer program to obtain database information of the original database, the processor further implements the following steps:

[0237] For any disk file in the original database, the last key stored in the disk file is used as the maximum key; the middle key is selected from the keys stored in the disk file; the layer information of the storage layer to which the disk file belongs in the original database is determined based on the merge log generated during the merge operation on the disk file; and the file system corresponding to the original database is queried to obtain the disk file size of the disk file.

[0238] In one embodiment, when the processor executes the computer program to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the database information, the following steps are also implemented:

[0239] The sum of the disk file sizes of each disk file is used as the database capacity of the original database; half of the database capacity is used as the reference capacity; according to the maximum key and layer information corresponding to each disk file, all disk files belonging to the same storage layer are sorted in descending order to obtain the sorting information of different storage layers in the original database; according to the reference capacity and the sorting information of different storage layers in the original database, a split key for splitting the original database is selected from the key-value pairs stored in each disk file.

[0240] In one embodiment, the sorting information of each storage layer includes the arrangement sequence number corresponding to each disk file in the storage layer; when the processor executes the computer program to select a split key for splitting the original database from the key-value pairs stored in each disk file according to the reference capacity and the sorting information of different storage layers in the original database, the following steps are also implemented:

[0241] For the current round, the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement number in each storage layer is used as the candidate disk file; the arrangement number corresponding to the candidate disk file is deleted from the sorting information of the storage layer to which the candidate disk file belongs; the sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round; wherein, if the current round is the first round, the previous file size is a set value; if the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as the split key for splitting the original database.

[0242] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0243] If the current file size is smaller than the reference capacity, the current round is used as the new previous round, the current file size is used as the new previous file size, and the next round is used as the new current round. The operation of returning to execute the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer as the candidate disk file is performed.

[0244] In one embodiment, when the processor executes the computer program to split the original database according to the split key, the database information, and the system characteristics of the file system corresponding to the original database, the following steps are also implemented:

[0245] An initial database is created according to the system characteristics of the file system corresponding to the original database; data processing is performed on the initial database and the original database according to the split key and database information to obtain a first database and a second database obtained by splitting the original database.

[0246] In one embodiment, when the processor executes the computer program to create the initial database according to the system characteristics of the file system corresponding to the original database, the processor further implements the following steps:

[0247] If it is determined based on the system characteristics of the file system that the file system supports the hard link function, a hard link is generated for each disk file in the original database, and a new database directory for the initial database is created based on each hard link; if it is determined based on the system characteristics of the file system that the file system does not support the hard link function, a new database directory for the initial database is created based on the split key and the file attribute information of each disk file in the original database.

[0248] In one embodiment, the file attribute information of each disk file also includes the smallest key in the key-value pair stored in the disk file; when the processor executes the computer program to create a new database directory of the initial database according to the split key and the file attribute information of each disk file in the original database, the processor also implements the following steps:

[0249] An initial database directory of the initial database is created; a disk file whose minimum key is greater than the split key is used as a migration file, and the file directory of the migration file in the original database directory corresponding to the original database is migrated to the initial database directory; and a disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key is used as a copy file, and the file directory of the copy file in the original database directory is copied to the initial database directory to create a new database directory for the initial database.

[0250] In one embodiment, when the processor executes the computer program to perform data processing on the initial database and the original database according to the split key and the database information to obtain the first database and the second database obtained by splitting the original database, the following steps are also implemented:

[0251] Based on the database writing mechanism, key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

[0252] It should be noted that the data involved in this application (including but not limited to data used for analysis, storage, display, etc.) are all information and data fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0253] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0254] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0255] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A database splitting method, characterized in that: The method comprises: Obtaining database information of an original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file; According to the database information, selecting a split key for splitting the original database from the key-value pairs stored in each of the disk files; The original database is split according to the split key, the database information, and system characteristics of a file system corresponding to the original database.

2. The method according to claim 1, characterized in that The step of obtaining database information of the original database includes: For any disk file in the original database, the last key stored in the disk file is used as the maximum key; Selecting an intermediate key from the keys stored in the disk file; Determine, according to a merge log generated during a merge operation on the disk file, layer information of a storage layer to which the disk file belongs in the original database; A query is performed through the file system corresponding to the original database to obtain the disk file size of the disk file.

3. The method according to claim 2, characterized in that The step of selecting, according to the database information, a split key for splitting the original database from the key-value pairs stored in each of the disk files includes: The sum of the disk file sizes of the disk files is used as the database capacity of the original database; Taking half of the database capacity as reference capacity; According to the maximum key and layer information corresponding to each of the disk files, all disk files belonging to the same storage layer are sorted in descending order to obtain sorting information of different storage layers in the original database; A split key for splitting the original database is selected from the key-value pairs stored in each of the disk files according to the reference capacity and sorting information of different storage layers in the original database.

4. The method according to claim 3, characterized in that: The sorting information of each storage layer includes the arrangement sequence number corresponding to each disk file in the storage layer; The selecting, from the key-value pairs stored in each of the disk files, a split key for splitting the original database according to the reference capacity and the sorting information of different storage layers in the original database, comprises: For the current round, the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement sequence number in each storage layer is used as the candidate disk file; Deleting the arrangement sequence number corresponding to the candidate disk file from the sorting information of the storage layer to which the candidate disk file belongs; The sum of the disk file size of the candidate disk file and the previous file size corresponding to the previous round is used as the current file size corresponding to the current round; wherein, if the current round is the first round, the previous file size is a set value; If the current file size is equal to or greater than the reference capacity, the intermediate key of the candidate disk file is used as a split key for splitting the original database.

5. The method according to claim 4, characterized in that The method further comprises: If the current file size is smaller than the reference capacity, the current round is taken as the new previous round, the current file size is taken as the new previous file size, and the next round is taken as the new current round. The operation of returning to execute the disk file corresponding to the largest maximum key among the disk files corresponding to the smallest arrangement number in each storage layer as the candidate disk file is performed.

6. The method according to claim 1, characterized in that The splitting of the original database according to the split key, the database information, and the system characteristics of the file system corresponding to the original database includes: Creating an initial database according to the system characteristics of the file system corresponding to the original database; According to the split key and the database information, data processing is performed on the initial database and the original database to obtain a first database and a second database obtained by splitting the original database.

7. The method according to claim 6, characterized in that The step of creating an initial database according to the system characteristics of the file system corresponding to the original database includes: If it is determined according to the system characteristics of the file system that the file system supports a hard link function, hard links of the disk files in the original database are generated, and a new database directory of the initial database is created according to the hard links; If it is determined according to the system characteristics of the file system that the file system does not support the hard link function, a new database directory of the initial database is created according to the split key and the file attribute information of each disk file in the original database.

8. The method according to claim 7, characterized in that The file attribute information of each disk file also includes the smallest key in the key-value pair stored in the disk file; The step of creating a new database directory of the initial database according to the split key and the file attribute information of each disk file in the original database comprises: Create an initial database directory for the initial database; Using the disk file whose minimum key is greater than the split key as a migration file, and migrating the file directory of the migration file in the original database directory corresponding to the original database to the initial database directory; and The disk file whose minimum key is less than or equal to the split key and whose maximum key is greater than the split key is used as a copy file, and the file directory of the copy file in the original database directory is copied to the initial database directory to create a new database directory for the initial database.

9. The method according to claim 6, characterized in that The step of performing data processing on the initial database and the original database according to the split key and the database information to obtain a first database and a second database obtained by splitting the original database includes: Based on a database writing mechanism, the key-value pairs in the original database that are greater than the split key are written into the initial database to obtain a first database; and The key-value pairs in the original database that are less than or equal to the split key are written into the original database to obtain a second database.

10. A database splitting device, characterized in that: The device comprises: An information acquisition module is used to acquire database information of an original database; wherein the original database is a key-value database, and the database information includes file attribute information of each disk file in the original database; the file attribute information of each disk file includes at least the maximum key and the middle key in the key-value pair stored in the disk file, the layer information of the storage layer to which the disk file belongs in the original database, and the disk file size of the disk file; A split key determination module, configured to select a split key for splitting the original database from the key-value pairs stored in each of the disk files according to the database information; The database splitting module is used to split the original database according to the splitting key, the database information and the system characteristics of the file system corresponding to the original database.

Images