Log splitting method and device

Abstract

The embodiment of the invention provides a log splitting method and device, and the method comprises the steps: obtaining log metadata of a log corresponding to a fragment when the fragment is split into split fragments; the log metadata comprises fragment identifiers corresponding to the fragments and a data unit list; data unit metadata of data units in the data unit list, wherein the data unit metadata comprises a data unit identifier and a shared list; the shared list comprises fragment identifiers of the shared data units; splitting the log metadata to obtain split log metadata; modifying a fragment identifier of a fragment in the split log metadata into a fragment identifier of the split fragment pair to obtain target split log metadata; and after the fragment identifier of the fragment and the fragment identifier of the split fragment are written into a shared list of the log metadata and the target split log metadata, persistently storing the log metadata and the target split log metadata in a storage system. According to the embodiment of the invention, the log splitting cost is low, and the splitting process is short in time consumption and atomicity.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of storage technology, and in particular, to a log splitting method, a log splitting device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the field of distributed KV (Key-Value) storage, the currently popular distributed KV architecture is a storage (storage layer) and computing (computing layer) separation architecture. Usually the storage layer is an independent distributed storage system, and the computing layer uses the storage layer as a unified storage system. In the distributed KV system with a total order, each Shard manages a Range (data interval) of data, and the data falling into the Range needs to be written to the Journal for persistence. As the amount of data in a Shard continues to increase, the Shard needs to be split. When a Shard is split into two Shards, the Journal corresponding to the Shard also needs to be split into two Journals.

[0003] However, the current traditional log splitting method uses data replication, which rewrites part of the data in the original log into another log. Since it involves a large number of operations, such as replication operations, the cost of log splitting is huge, and various exceptions need to be handled during the splitting process. Summary of the invention

[0004] In view of the above problems, a log splitting method and device are proposed to overcome the above problems or at least partially solve the above problems. The specific technical solution is as follows:

[0005] In a first aspect of the implementation of the present invention, a log splitting method is first provided, which is applied to a storage system, wherein the storage system includes shards, each of which has a corresponding data interval and a log, and the data of the data interval corresponding to each shard is written into a corresponding log, wherein the log includes multiple data units, and the method includes:

[0006] When splitting a split shard from a shard, obtaining log metadata of a log corresponding to the shard; the log metadata includes a shard identifier and a data unit list corresponding to the shard; data unit metadata of a data unit in the data unit list, the data unit metadata includes a data unit identifier and a sharing list; the sharing list includes a shard identifier that shares the data unit;

[0007] Splitting split log metadata from the log metadata;

[0008] Modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, to obtain target split log metadata;

[0009] After writing the shard identifier of the shard and the shard identifier of the split shard into the shared list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system.

[0010] In one embodiment of the present invention, after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata, and after persistently storing the log metadata and the target split log metadata in the storage system, the method further includes:

[0011] A new data unit is written into the log metadata and / or the target split log metadata, and the new data unit is included in a data unit list in the log metadata and / or the target split log metadata.

[0012] In an embodiment of the present invention, the storage system includes a central node, the central node includes a local sharing list and a local deletion list corresponding to the data unit, and the method further includes:

[0013] When the shard or split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is not empty, the sharing list and the data unit identifier of the target data unit are sent to the central node;

[0014] Among them, the central node is used to update the local sharing list according to the sharing list of the target data unit, and to update the local deletion list according to the data unit identifier of the target data unit; when the shard identifiers in the local sharing list and the local deletion list corresponding to the target data unit are the same, a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deletion list.

[0015] In one embodiment of the present invention, the method further comprises:

[0016] When the shard or the split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is empty, the target data unit in the shard or the split shard is deleted.

[0017] In one embodiment of the present invention, the central node is used to delete the local shared list and the local deletion list corresponding to the shard corresponding to the shard identifier in the local deletion list when the shard corresponding to the shard identifier in the local deletion list successfully deletes the target data unit.

[0018] In an embodiment of the present invention, the log metadata also includes a data interval.

[0019] In an embodiment of the present invention, the data unit metadata also includes the data unit length and whether the data unit is shared; whether the data unit is shared is closed by default.

[0020] The embodiment of the present invention further discloses a log splitting device, which is applied to a storage system. The storage system includes shards, each of which has a corresponding data interval and a log. The data of the data interval corresponding to each shard is written into a corresponding log. The log includes multiple data units. The device includes:

[0021] A shard splitting module is used to obtain log metadata of the log corresponding to the shard when splitting the shard from the shard; the log metadata includes the shard identifier and the data unit list corresponding to the shard; the data unit metadata of the data unit in the data unit list, the data unit metadata includes the data unit identifier and the sharing list; the sharing list includes the shard identifier that shares the data unit;

[0022] A log splitting module, used for splitting the log metadata into split log metadata;

[0023] A shard identifier modification module, used to modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, to obtain target split log metadata;

[0024] A persistence module is used to persistently store the log metadata and the target split log metadata in the storage system after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata.

[0025] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes any of the log splitting methods described above.

[0026] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the log splitting methods described above.

[0027] Compared with the related art, the embodiments of the present invention have at least the following advantages:

[0028] In an embodiment of the present invention, the storage system is applied, and the storage system includes shards, each of which has a corresponding data interval and log. The data of the data interval corresponding to each shard is written into the corresponding log. The log includes multiple data units. When a split shard is split from a shard, the log metadata of the log corresponding to the shard is obtained; the log metadata includes the shard identifier and data unit list corresponding to the shard; the data unit metadata of the data unit in the data unit list, the data unit metadata includes the data unit identifier and a sharing list; the sharing list includes the shard identifier of the shared data unit; split the split log metadata from the log metadata; modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard pair, and obtain the target split log metadata; after writing the shard identifier of the shard and the shard identifier of the split shard in the sharing list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system. After the shard is split, the embodiment of the present invention splits the split log metadata of a log based on the log metadata of the log corresponding to the shard, thereby generating two new logs, and then writes the shard identifiers of the shards corresponding to the two logs into the shared list of data units of the log metadata of the two logs. At the same time, the shard identifier of the shard of the split log metadata is modified to the shard identifier of the split shard pair to obtain the target split log metadata. The log metadata and the target split log metadata can be persistently stored in the storage system to complete the log splitting. Since it is not necessary to copy part of the data of the sharded log to another log to realize the log splitting, there is no data replication during the log splitting process, and there is no need to involve a large number of operations. The cost of log splitting is low and the splitting process is short, and there is no need to handle various exceptions during the splitting process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0030] Figure 1 A flow chart of the steps of a log splitting method provided in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a sharded log according to an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of log metadata splitting provided in an embodiment of the present invention;

[0033] Figure 4 A schematic diagram of writing logs after slice splitting provided in an embodiment of the present invention

[0034] Figure 5 A schematic diagram of splitting a slice and then splitting and writing a log provided in an embodiment of the present invention;

[0035] Figure 6a-6c is a schematic diagram of deleting a log chunk provided in an embodiment of the present invention;

[0036] Figure 7 A structural block diagram of a log splitting device provided in an embodiment of the present invention;

[0037] Figure 8 The present invention is a block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] Reference Figure 1 , is a flow chart of the steps of a log splitting method provided in an embodiment of the present invention, such as Figure 1 As shown, the storage system includes shards, each of which has a corresponding data interval and a log. The data of the data interval corresponding to each shard is written into a corresponding log. The log includes multiple data units. The method may specifically include the following steps:

[0040] Step 101: when a split shard is split from a shard, obtain log metadata of a log corresponding to the shard; the log metadata includes a shard identifier and a data unit list corresponding to the shard; data unit metadata of a data unit in the data unit list, the data unit metadata includes a data unit identifier and a sharing list; the sharing list includes a shard identifier that shares the data unit;

[0041] In a specific implementation, the storage system may be a distributed KV system (distributed key-value pair system), such as a LAVA storage system or other distributed systems.

[0042] In an ordered distributed KV system, Shards may be included, which may be one or more. Specifically, Shards divide the entire data set into multiple smaller, independent parts, each of which is called a shard. Each shard is responsible for storing a portion of the data and can be managed and operated independently. For example, refer to Figure 2, is a schematic diagram of a sharded log of an embodiment of the present invention, and the shards have corresponding ranges (data intervals) and Journals (logs). Specifically, range is an ordered interval (data interval), which represents a set of ordered key value ranges in an ordered KV storage system. Journal is a log of a distributed KV system. When an IO (Input / Output) event is received, the reliability of the data is guaranteed by recording logs. Journal is based on ChunkList, so it naturally has storage reliability, where ChunkList is a data stream composed of a group of chunks (data units), which can break through the size limit of a single chunk and realize the unlimited append feature of storage. Chunk refers to a data unit that can be read and written in the LAVA storage system, which supports random read / sequential write. Chunk has a size limit, generally 128MB (megabytes).

[0043] In the specific implementation, Shard splitting is a common scenario. For example, when a Shard's data interval is [a,f], when [a,f] has too much data, [a,f] needs to be split into two data intervals [a,d] and [d,f]. After the shard is split, corresponding logs need to be generated for the split shards to ensure the reliability of the data of the split shards.

[0044] In an embodiment of the present invention, when a slice is split into another split slice, the log metadata of the journal corresponding to the slice is obtained. Specifically, the log metadata of a Journal includes: shard_id (shard identifier), range (data range), chunk_list (data unit list). Among them, shard_id represents the Shard of the storage system to which the Journal belongs, range represents the range of valid data in the Journal, chunk_list consists of a series of chunks, and the metadata of Chunk includes chunk_id (data unit identifier), chunk_len (length of chunk), Is_shared (whether the chunk is shared, Is_shared can be true (true) or empty), share_list (shared list, indicating which shards the chunk is shared by, share_list can include the shard_id of the shared chunk).

[0045] By default, chunks are not shared, so shared_list is empty by default. The size limit of each chunk is 128M. When the chunk space is full, the slice needs to apply for a new chunk and add the chunk_id of the new chunk to chunk_list. Among them, all the log metadata of the Journal is persisted in etcd (a distributed key-value storage system) to ensure the reliability of the log metadata.

[0046] Step 102: split the log metadata into split log metadata;

[0047] In the embodiment of the present invention, after the log metadata of the log is obtained, split log metadata may be split from the log metadata. In this case, the split log metadata may be the same as the log metadata.

[0048] Step 103: modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, to obtain target split log metadata;

[0049] In an embodiment of the present invention, for the split log metadata, the shard_id (shard identifier) ​​corresponding to the split log metadata can be modified to the shard_id (shard identifier) ​​of the split shard, indicating that the shard corresponding to the split log metadata is the split shard, wherein the split log metadata after the shard identifier of the shard is modified will be used as the target split log metadata.

[0050] Step 104: After writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system.

[0051] In an embodiment of the present invention, in the sharing list (share_list) of the chunk (data unit) of the log metadata and the target split log metadata, after writing the shard_id of the shard and the shard_id of the split shard respectively, it indicates that the slice and the split slice can share the chunk. Then, the log metadata and the target split log metadata can be persistently stored in the storage system. It can be understood that when the slice is split, the slices before and after the split can share the same chunk, which can reduce data redundancy and improve the utilization rate of storage resources of the storage system.

[0052] In an example of log splitting, refer to Figure 3, is a schematic diagram of log metadata splitting provided in an embodiment of the present invention. During the log splitting process, the Journal is firstly disabled from being written, and then the log metadata of the Journal is split into two parts (i.e., the log metadata and the split log metadata). Figure 3 It can be seen that the Journal with shard_id = 1 splits into a new Journal with shard_id = 2 (split log). In addition, the data range of the Journal with shard_id = 1 is reduced from [a, f] to [a, d]. The data range of the new Journal with shard_id = 2 is [d, f]. The data unit metadata is_shared of each chunk in chunk_list becomes true, and shared_list becomes "1, 2", indicating that this chunk is shared by the slices corresponding to shard_id = 1 and shard_id = 2.

[0053] Persisting the log metadata of the two journals shard_id=1 and shard_id=2 together in etcd (a distributed key-value pair storage system) means that the log splitting is completed. It should be noted that since the log metadata generation process is performed in memory, the persistent log metadata will not be destroyed, and the log metadata of the two journals generated are persisted together in etcd, so the process is atomic. Among them, atomicity means that an event satisfies the characteristics of either success or failure, and there will be no intermediate state.

[0054] In the above-mentioned log splitting method, it is applied to a storage system, the storage system includes shards, the shards respectively have corresponding data intervals and logs, the data of the data intervals corresponding to each shard is written into the corresponding log, the log includes multiple data units, when splitting a split shard from a shard, the log metadata of the log corresponding to the shard is obtained; the log metadata includes the shard identifier and data unit list corresponding to the shard; the data unit metadata of the data unit in the data unit list, the data unit metadata includes the data unit identifier and a sharing list; the sharing list includes the shard identifier of the shared data unit; split the split log metadata from the log metadata; modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard pair, and obtain the target split log metadata; after writing the shard identifier of the shard and the shard identifier of the split shard in the sharing list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system. After the shard is split, the embodiment of the present invention splits the split log metadata of a log based on the log metadata of the log corresponding to the shard, thereby generating two new logs, and then writes the shard identifiers of the shards corresponding to the two logs into the shared list of data units of the log metadata of the two logs. At the same time, the shard identifier of the shard of the split log metadata is modified to the shard identifier of the split shard pair to obtain the target split log metadata. The log metadata and the target split log metadata can be persistently stored in the storage system to complete the log splitting. Since it is not necessary to copy part of the data of the sharded log to another log to realize the log splitting, there is no data replication during the log splitting process, and there is no need to involve a large number of operations. The cost of log splitting is low and the splitting process is short, and there is no need to handle various exceptions during the splitting process.

[0055] In an embodiment of the present invention, after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata, and after persistently storing the log metadata and the target split log metadata in the storage system, the method may further include:

[0056] A new data unit is written into the log metadata and / or the target split log metadata, and the new data unit is included in a data unit list in the log metadata and / or the target split log metadata.

[0057] In the embodiment of the present invention, after the log metadata of the journal is persisted in the storage system, the two shards (i.e., the shard and the split shard) can continue to provide write services, and then the chunk_lists of the two Journals are no longer the same. Figure 4 , is a schematic diagram of writing logs after slice splitting provided in an embodiment of the present invention, from Figure 4 It can be seen that the new data of the Journal with shard_id=1 is written into the new data unit chunk_6, and the new data of the Journal with shard_id=2 is written into the new data unit chunk_7. In addition, chunk_6 and chunk_7 are not shared, so the shared_list of chunk_6 and chunk_7 is empty.

[0058] It should also be noted that the split Journal is split again. Because a shard may be split multiple times, the corresponding Journal of the shard will also be split multiple times. Figure 5 , is a schematic diagram of a slice split and then split and write a log provided in an embodiment of the present invention. The process of splitting the Journal after splitting is basically the same as before, except that the shared_list of the chunk inherits the previous shared_list. For example: chunk_id=3 has been shared by shard=1 and shard=2 before splitting, so the shared_list of chunk_id=3 becomes "1,2,3" after splitting, indicating that the slices of shard=1, shard=2 and shard=3 share the data unit corresponding to chunk_id=3.

[0059] In an embodiment of the present invention, the storage system includes a central node, the central node includes a local sharing list and a local deletion list corresponding to the data unit, and the method may further include:

[0060] When the shard or split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is not empty, the sharing list and the data unit identifier of the target data unit are sent to the central node;

[0061] Among them, the central node is used to update the local sharing list according to the sharing list of the target data unit, and to update the local deletion list according to the data unit identifier of the target data unit; when the shard identifiers in the local sharing list and the local deletion list corresponding to the target data unit are the same, a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deletion list.

[0062] In a specific implementation, sometimes it is necessary to recycle the logs of the storage system. In the embodiment of the present invention, the logs are recycled according to the chunk (data unit) granularity. Since some chunks are shared after splitting, if the chunk corresponding to the log is directly deleted, other logs sharing the chunk will be abnormal. Therefore, the shared chunk cannot be directly deleted. In view of the above problem, the embodiment of the present invention introduces an intermediate node (central node) to solve the sharedchunk (shared chunk) deletion problem.

[0063] The central node is responsible for recording the sharing information of chunks by recording two lists of chunks. One of the two lists is the local shared list (shared_list) of the central node, and the other is the local deleted list (deleted_list). In an embodiment of the present invention, when a certain shard initiates a deletion request for a target data unit, if the shared list (shared_list) corresponding to the target data unit is not empty, indicating that the target data unit is a chunk shared by multiple slices, the shared list (shared_list) and data unit identifier (chunk_id) of the target data unit can be sent to the central node, and the central node updates its local shared list (shared_list) according to the shared list (shared_list) and data unit identifier (chunk_id) of the target data unit, and updates the local deleted list (deleted_list) according to the data unit identifier (chunk_id) of the target data unit. When the local shared list (shared_list) corresponding to the target data unit is the same as the shard identifier in the local deleted list (deleted_list), a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deleted list, thereby completing the deletion of the target data unit of the shard.

[0064] It can be seen that in the embodiment of the present invention, when the local shared list (shared_list) and the local deleted list (deleted_list) of a certain chunk in the central node are the same, that is, the shard_id in the local shared list (shared_list) and the local deleted list (deleted_list) are the same, it means that the chunk can be truly deleted. For example, the local shared list (shared_list) and the local deleted list (deleted_list) are both "1,2", indicating that both slice 1 and slice 2 agree to delete the shared chunk, then both slice 1 and slice 2 can delete the chunk. Since the central node does not record the shared list of the chunk during the log splitting process, the central node will only start to record the local shared list (shared_list) and the local deleted list (deleted_list) corresponding to the chunk when the Journal initiates the deletion of a chunk (target data unit).

[0065] For example, refer to Figure 6a-6c , is a schematic diagram of deleting a log chunk provided in an embodiment of the present invention, and the deletion process of shard=2 deleting chunk=3 is specifically as follows:

[0066] Reference Figure 6a , the process of deleting chunk=3 from shard2 includes:

[0067] a. Shard=2 initiates a deletion operation to delete chunk=3.

[0068] b. First, shard=2 determines whether chunk=3 is a shared chunk. If it is not a shared chunk, it directly initiates deletion.

[0069] c. If it is a shared chunk, send the shared list and deleted list of chunk = 3 to the central node. Here the shared list (shared_list = "1,2") and the deleted list (deleted_list = "2")

[0070] d. After receiving the request, the central node updates the shared_list = "1,2" and deleted_list = "2" of chunk = 3, persists it, and returns success. Shard = 2 deletes chunk = 3 from chunk_list.

[0071] Reference Figure 6b , the process of deleting chunk=3 from shard1 includes:

[0072] e. Shard=3 initiates a delete operation on chunk=3.

[0073] f. First, shard=3 determines that chunk=3 is a shared chunk, and sends chunk=3's shared_list="1,2,3", deleted_list="3" to the central node.

[0074] g. The central node updates shared_list = "1,2,3", deleted_list = "2,3".

[0075] Reference Figure 6c , the process of deleting chunk=3 from shard1 includes:

[0076] h. Shard=1 initiates a delete operation on chunk=3.

[0077] i. First, shard=1 determines that chunk=3 is a shared chunk, and sends chunk=3's shared_list="1,2,3", deleted_list="1" to the central node.

[0078] j. The central node updates shared_list = "1,2,3", deleted_list = "1,2,3", and persists it to etcd. Because shared_list is equal to deleted_list, the store layer (chunk storage layer) initiates chunk deletion.

[0079] k. When the chunk is deleted successfully, the central node deletes the shared_list and deleted_list of chunk=3 from etcd to avoid wasting unnecessary storage resources.

[0080] In one embodiment of the present invention, the method may further include:

[0081] When the shard or the split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is empty, the target data unit in the shard or the split shard is deleted.

[0082] In an embodiment of the present invention, when a shard initiates a deletion request for a target data unit, if the shared list (shared_list) corresponding to the target data unit is empty, it means that the target data unit is not a chunk shared by the slice. At this time, the corresponding target data unit of the shard or split shard can be deleted.

[0083] The embodiment of the present invention provides a solution for quickly / atomically splitting a log into two logs based on a storage system. The core idea is that the data is not copied during the splitting process, but the log metadata of the journal is split to achieve the purpose of quickly splitting the journal. The two newly generated journals share data by sharing chunks (data units). In addition, during the log playback process, it is only necessary to filter out the chunks that are not needed by each journal.

[0084] In order to enable those skilled in the art to better understand, a specific example is used below to illustrate the process of implementing the log splitting function based on the distributed KV system.

[0085] 1. Write Journal:

[0086] a. Initially, the entire distributed KV system has only one Shard, which means that all data falls into this Shard. Each Shard has a corresponding Journal, and each data write request is first persisted by writing to the Journal.

[0087] Shard

[0088] Shard_id; 1

[0089] Journal

[0090] Range{begin="a",end="f"}

[0091] ChunkList{

[0092] {chunk_id=1, chunk_length=12M, is_shared=false, shared_list=""},

[0093] {chunk_id=2,chunk_length=64M, is_shared=false, shared_list=""}

[0094] }

[0095] }

[0096] }

[0097] b. Journal consists of multiple Chunks, and the Journal's log metadata records the chunk list

[0098] (Chunk metadata), each chunk in the chunk list records the chunk length, chunk_id, whether the chunk is shared, which shards share it, etc. The journal metadata is persisted through etcd to ensure the reliability of the journal metadata.

[0099] c. When data is continuously written, the Journal will apply for new chunks. The initial state of new chunks is non-shared, so the shared list is also empty. Add the new chunk to the ChunkList.

[0100] 2. Journal split:

[0101] a. When a shard has too much data, it needs to be split into two shards. Therefore, the Journal corresponding to the shard also needs to be split into two Journals.

[0102] b. The splitting of the Journal is completed by copying the log metadata of the Journal. A new copy of the log metadata of the Journal is copied, and then the Range (valid data interval) of the two Journals is modified. Each chunk in the log metadata is changed to a shared chunk, and the shared list of chunks is increased.

[0103] c. After generating two new Journal log metadata, write the log metadata of the two Journals into etcd at one time for persistence.

[0104] d. After the data is written to etcd successfully, the entire Journal splitting process is completed. Since the present invention writes to etcd once, the Journal splitting process is atomic.

[0105] 3. Chunk recovery (log recovery):

[0106] a. After a period of time, the data in some chunks in the Journal becomes invalid, so it is necessary to delete this chunk to free up storage space in the underlying storage.

[0107] b. After the Journal splits, some chunks are shared by two Journals. When a Journal deletes it, it needs to report it to the central node shared_list and deleted_list.

[0108] c. The central node merges the received shared_list and deleted_list with the locally recorded shared_list and deleted_list. If shared_list == deleted_list, a DeleteChunk request is sent to StoreLayer.

[0109] By applying the embodiment of the present invention, there is no need to migrate data when splitting a log. Instead, only the log metadata is modified through the shared chunk. By persisting the newly generated log metadata to etcd at one time, the log splitting is fast and atomic, and the entire splitting process can be completed in milliseconds. In addition, the embodiment of the present invention creatively introduces the central node / shared_list / deleted_list, which solves the problem that the deletion of a shared chunk affects other shared slices, and does not cause the accidental deletion of chunks or the problem of residual chunks.

[0110] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0111] Reference Figure 7 , is a structural block diagram of a log splitting device provided in an embodiment of the present invention, such as Figure 7 As shown, the storage system includes shards, each of which has a corresponding data interval and a log. The data of the data interval corresponding to each shard is written into a corresponding log. The log includes multiple data units. The device may specifically include the following modules:

[0112] The shard splitting module 701 is used to obtain log metadata of the log corresponding to the shard when splitting the shard from the shard; the log metadata includes the shard identifier and the data unit list corresponding to the shard; the data unit metadata of the data unit in the data unit list, the data unit metadata includes the data unit identifier and the sharing list; the sharing list includes the shard identifier that shares the data unit;

[0113] A log splitting module 702, configured to split the log metadata into split log metadata;

[0114] A shard identifier modification module 703 is used to modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, so as to obtain target split log metadata;

[0115] The persistence module 704 is used to persistently store the log metadata and the target split log metadata in the storage system after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata.

[0116] In one embodiment of the present invention, the device may further include:

[0117] A new data unit is written into the log metadata and / or the target split log metadata, and the new data unit is included in a data unit list in the log metadata and / or the target split log metadata.

[0118] In an embodiment of the present invention, the storage system includes a central node, the central node includes a local sharing list and a local deletion list corresponding to the data unit, and the device further includes:

[0119] When the shard or split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is not empty, the sharing list and the data unit identifier of the target data unit are sent to the central node;

[0120] Among them, the central node is used to update the local sharing list according to the sharing list of the target data unit, and to update the local deletion list according to the data unit identifier of the target data unit; when the shard identifiers in the local sharing list and the local deletion list corresponding to the target data unit are the same, a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deletion list.

[0121] In one embodiment of the present invention, the device may further include:

[0122] When the shard or the split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is empty, the target data unit in the shard or the split shard is deleted.

[0123] In one embodiment of the present invention, the central node is used to delete the local shared list and the local deletion list corresponding to the shard corresponding to the shard identifier in the local deletion list when the shard corresponding to the shard identifier in the local deletion list successfully deletes the target data unit.

[0124] In an embodiment of the present invention, the log metadata may further include a data interval.

[0125] In an embodiment of the present invention, the data unit metadata may further include the data unit length and whether the data unit is shared; whether the sharing of the data unit is closed by default, etc.

[0126] In an embodiment of the present invention, the storage system is applied, and the storage system includes shards, each of which has a corresponding data interval and log. The data of the data interval corresponding to each shard is written into the corresponding log. The log includes multiple data units. When a split shard is split from a shard, the log metadata of the log corresponding to the shard is obtained; the log metadata includes the shard identifier and data unit list corresponding to the shard; the data unit metadata of the data unit in the data unit list, the data unit metadata includes the data unit identifier and a sharing list; the sharing list includes the shard identifier of the shared data unit; split the split log metadata from the log metadata; modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard pair, and obtain the target split log metadata; after writing the shard identifier of the shard and the shard identifier of the split shard in the sharing list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system. After the shard is split, the embodiment of the present invention splits the split log metadata of a log based on the log metadata of the log corresponding to the shard, thereby generating two new logs, and then writes the shard identifiers of the shards corresponding to the two logs into the shared list of data units of the log metadata of the two logs. At the same time, the shard identifier of the shard of the split log metadata is modified to the shard identifier of the split shard pair to obtain the target split log metadata. The log metadata and the target split log metadata can be persistently stored in the storage system to complete the log splitting. Since it is not necessary to copy part of the data of the sharded log to another log to realize the log splitting, there is no data replication during the log splitting process, and there is no need to involve a large number of operations. The cost of log splitting is low and the splitting process is short, and there is no need to handle various exceptions during the splitting process.

[0127] As for the above-mentioned device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] It should be noted that the embodiments of the present invention may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0129] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0130] Memory 503, used for storing computer programs;

[0131] The processor 501 is configured to implement any log splitting method described in the above embodiments when executing the program stored in the memory 503:

[0132] When splitting a split shard from a shard, obtaining log metadata of a log corresponding to the shard; the log metadata includes a shard identifier and a data unit list corresponding to the shard; data unit metadata of a data unit in the data unit list, the data unit metadata includes a data unit identifier and a sharing list; the sharing list includes a shard identifier that shares the data unit;

[0133] Splitting split log metadata from the log metadata;

[0134] Modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, to obtain target split log metadata;

[0135] After writing the shard identifier of the shard and the shard identifier of the split shard into the shared list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system.

[0136] Optionally, the method further comprises:

[0137] A new data unit is written into the log metadata and / or the target split log metadata, and the new data unit is included in a data unit list in the log metadata and / or the target split log metadata.

[0138] Optionally, the storage system includes a central node, the central node includes a local sharing list and a local deletion list corresponding to the data unit, and the method further includes:

[0139] When the shard or split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is not empty, the sharing list and the data unit identifier of the target data unit are sent to the central node;

[0140] Among them, the central node is used to update the local sharing list according to the sharing list of the target data unit, and to update the local deletion list according to the data unit identifier of the target data unit; when the shard identifiers in the local sharing list and the local deletion list corresponding to the target data unit are the same, a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deletion list.

[0141] Optionally, the method further comprises:

[0142] When the shard or the split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is empty, the target data unit in the shard or the split shard is deleted.

[0143] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0144] The communication interface is used for communication between the above terminal and other devices.

[0145] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0146] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0147] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the log splitting method described in any one of the above embodiments.

[0148] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the log splitting method described in any one of the above embodiments.

[0149] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk SSD), etc.

[0150] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or still includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0151] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A log splitting method, characterized in that: Applied to a storage system, the storage system includes shards, the shards respectively have corresponding data intervals and logs, the data of the data intervals corresponding to each shard is written into the corresponding log, the log includes multiple data units, and the method includes: When splitting a split shard from a shard, obtaining log metadata of a log corresponding to the shard; the log metadata includes a shard identifier and a data unit list corresponding to the shard; data unit metadata of a data unit in the data unit list, the data unit metadata includes a data unit identifier and a sharing list; the sharing list includes a shard identifier that shares the data unit; Splitting split log metadata from the log metadata; Modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, to obtain target split log metadata; After writing the shard identifier of the shard and the shard identifier of the split shard into the shared list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system.

2. The method according to claim 1, characterized in that After writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata, and after persistently storing the log metadata and the target split log metadata in the storage system, the method further includes: A new data unit is written into the log metadata and / or the target split log metadata, and the new data unit is included in a data unit list in the log metadata and / or the target split log metadata.

3. The method according to claim 1, characterized in that The storage system includes a central node, the central node includes a local sharing list and a local deletion list corresponding to the data unit, and the method further includes: When the shard or split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is not empty, the sharing list and the data unit identifier of the target data unit are sent to the central node; Among them, the central node is used to update the local sharing list according to the sharing list of the target data unit, and to update the local deletion list according to the data unit identifier of the target data unit; when the shard identifiers in the local sharing list and the local deletion list corresponding to the target data unit are the same, a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deletion list.

4. The method according to claim 3, characterized in that The method further comprises: When the shard or the split shard initiates a deletion request for a target data unit, if the sharing list corresponding to the target data unit is empty, the target data unit in the shard or the split shard is deleted.

5. The method according to claim 3, characterized in that: The central node is used to delete the local shared list and the local deletion list corresponding to the shard corresponding to the shard identifier in the local deletion list when the shard corresponding to the shard identifier in the local deletion list successfully deletes the target data unit.

6. The method according to claim 1, characterized in that The log metadata also includes a data interval.

7. The method according to claim 1, characterized in that The data unit metadata also includes the data unit length and whether the data unit is shared; whether the data unit is shared is closed by default.

8. A log splitting device, characterized in that: Applied to a storage system, the storage system includes shards, the shards respectively have corresponding data intervals and logs, the data of the data intervals corresponding to each shard is written into the corresponding log, the log includes multiple data units, and the device includes: A shard splitting module is used to obtain log metadata of the log corresponding to the shard when splitting the shard from the shard; the log metadata includes the shard identifier and the data unit list corresponding to the shard; the data unit metadata of the data unit in the data unit list, the data unit metadata includes the data unit identifier and the sharing list; the sharing list includes the shard identifier that shares the data unit; A log splitting module, used for splitting the log metadata into split log metadata; A shard identifier modification module, used to modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, to obtain target split log metadata; A persistence module is used to persistently store the log metadata and the target split log metadata in the storage system after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images