Log splitting method and apparatus

Abstract

Embodiments of the present application provide a log splitting method and device, comprising: when splitting a split shard from a shard, obtaining log metadata of a log corresponding to the shard; the log metadata comprising shard identification and a data unit list corresponding to the shard; data unit metadata of a data unit in the data unit list, the data unit metadata comprising data unit identification and a shared list; the shared list comprising shard identification of a shared data unit; splitting the log metadata to obtain split log metadata; modifying the shard identification of the shard in the split log metadata to the shard identification of a split shard pair to obtain target split log metadata; after writing the shard identification of the shard and the shard identification of the split shard in the shared list of the log metadata and the target split log metadata, persistently storing the log metadata and the target split log metadata in a storage system. Embodiments of the present application have small cost of log splitting, short time consumption in the splitting process and atomicity.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of storage, in particular to a log splitting method, a log splitting device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In the field of distributed KV (Key-Value) storage, the popular distributed KV architecture is a storage (storage layer) and computing (computing layer) separation architecture. Generally, the storage layer is an independent distributed storage system, and the computing layer uses the storage layer as a unified storage system. In a distributed KV system with full-order distributed KV storage, each Shard manages the data of a Range, and the data falling into the Range needs to be written into a Journal for persistence. As the amount of data in a Shard increases, the Shard needs to be split. When a Shard is split into two shards, the Journal corresponding to the Shard also needs to be split into two Journals.

[0003] However, the current traditional log splitting method adopts a data replication method, which rewrites part of the data in the original log into another log. Since a large number of operations such as replication operations are involved, the cost of log splitting is huge, and various exceptions in the splitting process need to be handled. SUMMARY

[0004] In view of the above problems, a log splitting method and device are provided to overcome the above problems or at least partially solve the above problems. The specific technical solutions are as follows:

[0005] In a first aspect of the present application, a log splitting method is first provided, applied to a storage system, the storage system including shards, the shards respectively having corresponding data intervals and logs, the data of each data interval corresponding to the shard being written into the corresponding log, the log including a plurality of data units, and the method including:

[0006] When a split shard is split from a shard, log metadata of a log corresponding to the shard is obtained; the log metadata includes a shard identifier and a data unit list corresponding to the shard; the data unit metadata of a data unit in the data unit list includes a data unit identifier and a shared list; the shared list includes shard identifiers sharing the data unit;

[0007] Splitting log metadata is split from the log metadata;

[0008] modifying a shard identifier of the shard in the split log metadata to a shard identifier of the split shard, to obtain target split log metadata;

[0009] after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata, persistently storing the log metadata and the target split log metadata in the storage system.

[0010] In an embodiment of the present application, after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata, persistently storing the log metadata and the target split log metadata in the storage system, the method further comprises:

[0011] writing a new data unit in the log metadata and or the target split log metadata, and adding the new data unit in a data unit list in the log metadata and or the target split log metadata corresponding to the new data unit.

[0012] In an embodiment of the present application, the storage system comprises a center node, and the center node comprises a local shared list and a local deletion list corresponding to the data unit, and the method further comprises:

[0013] when the shard or the split shard initiates a deletion request for a target data unit, if a shared list corresponding to the target data unit is not empty, sending the shared list and a data unit identifier of the target data unit to the center node.

[0014] wherein the center node is configured to update the local shared list according to the shared list of the target data unit, and update the local deletion list according to the data unit identifier of the target data unit; and when the shard identifier in the local shared list and the local deletion list corresponding to the target data unit is the same, initiating a deletion request of the target data unit to the shard corresponding to the shard identifier in the local deletion list.

[0015] In an embodiment of the present application, the method further comprises:

[0016] when the shard or the split shard initiates a deletion request for a target data unit, if a shared list corresponding to the target data unit is empty, deleting the target data unit in the shard or the split shard.

[0017] In an embodiment of the present application, the central node is configured to delete the local shared list and the local deletion list corresponding to the shard identified by the shard identifier in the local deletion list when the shard identified by the shard identifier in the local deletion list successfully deletes the target data unit.

[0018] In an embodiment of the present application, the log metadata further comprises a data interval.

[0019] In an embodiment of the present application, the data unit metadata further comprises a data unit length and whether the data unit is shared; the whether the data unit is shared is closed by default.

[0020] The embodiment of the present application further discloses a log splitting device applied to a storage system, wherein the storage system comprises shards, the shards respectively have corresponding data intervals and logs, data of the data interval corresponding to each shard is written into the corresponding log, the log comprises a plurality of data units, and the device comprises:

[0021] a shard splitting module configured to acquire log metadata of a log corresponding to a shard when a split shard is split from the shard; the log metadata comprises a shard identifier corresponding to the shard and a data unit list; data unit metadata of a data unit in the data unit list comprises a data unit identifier and a shared list; the shared list comprises shard identifiers sharing the data unit;

[0022] a log splitting module configured to split a split log metadata from the log metadata;

[0023] a shard identifier modifying module configured to modify a shard identifier of the shard in the split log metadata into a shard identifier of the split shard to obtain a target split log metadata;

[0024] a persistence module configured to, after writing the shard identifier of the shard and the shard identifier of the split shard into the shared list of the log metadata and the target split log metadata, persistently store the log metadata and the target split log metadata in the storage system.

[0025] In another aspect of the embodiment of the present application, a computer readable storage medium is provided, wherein instructions are stored in the computer readable storage medium, and when the instructions are run on a computer, the computer is caused to perform the log splitting method according to any one of the above embodiments.

[0026] In another aspect of the embodiment of the present application, a computer program product comprising instructions is provided, and when the instructions are run on a computer, the computer is caused to perform the log splitting method according to any one of the above embodiments.

[0027] Compared with the related art, the embodiments of the present application have at least the following advantages:

[0028] In the embodiments of the present application, the method is applied to a storage system, the storage system includes shards, the shards respectively have corresponding data intervals and logs, data of the data intervals corresponding to the shards is written into the corresponding logs, the logs include a plurality of data units, when a split shard is split from a shard, log metadata of a log corresponding to the shard is acquired; the log metadata includes shard identifiers corresponding to the shard and a data unit list; data unit metadata of the data units in the data unit list, the data unit metadata includes data unit identifiers and a sharing list; the sharing list includes shard identifiers of shared data units; split log metadata is split from the log metadata; shard identifiers of the shards in the split log metadata are modified into shard identifiers of a split shard pair to obtain target split log metadata; after the shard identifiers of the shards and the shard identifiers of the split shards are written into the sharing lists of the log metadata and the target split log metadata, the log metadata and the target split log metadata are stored persistently in the storage system. In the embodiments of the present application, after the shard is split, a split log metadata of a log is split based on the log metadata of the log corresponding to the shard, thereby two new logs are generated, then the shard identifiers of the two logs corresponding to the shards are written into the sharing lists of the data units of the log metadata of the two logs, at the same time, the shard identifiers of the shards of the split log metadata are modified into the shard identifiers of the split shard pair to obtain the target split log metadata, so that the log metadata and the target split log metadata can be stored persistently in the storage system, the log splitting is completed, since it is not necessary to copy part of data of the log of the shard to another log to implement the log splitting, no data is copied in the log splitting process, and a large number of operations are not involved in the log splitting process, the cost of the log splitting is small, the time consumption of the splitting process is short, and various exceptions in the splitting process do not need to be processed. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0030] Figure 1 A step flowchart of the log splitting method provided in the embodiments of the present application;

[0031] Figure 2 A schematic diagram of a log of a shard in the embodiments of the present application;

[0032] Figure 3 A schematic diagram of log metadata splitting provided in the embodiments of the present application;

[0033] Figure 4 A schematic diagram of log writing after a slice splitting in the embodiments of the present application

[0034] Figure 5 This is a schematic diagram illustrating a method of writing logs after slicing and then splitting the slice, as provided in an embodiment of the present invention.

[0035] Figures 6a-6c This is a schematic diagram of a log deletion chunk provided in an embodiment of the present invention;

[0036] Figure 7 A structural block diagram of a log splitting device provided in an embodiment of the present invention;

[0037] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0039] Reference Figure 1 The above is a flowchart of the steps of a log splitting method provided in an embodiment of the present invention, as follows: Figure 1 As shown, the storage system includes shards, each shard having a corresponding data range and a log. Data from the data range corresponding to each shard is written into the corresponding log, and the log includes multiple data units. The method may specifically include the following steps:

[0040] Step 101: When splitting a shard from a shard, obtain the log metadata of the log corresponding to the shard; the log metadata includes the shard identifier and data unit list corresponding to the shard; the data unit metadata of the data units in the data unit list includes the data unit identifier and a sharing list; the sharing list includes the shard identifier that shares the data unit;

[0041] In a practical implementation, the storage system can be a distributed key-value system, such as the LAVA storage system or other distributed systems.

[0042] In an ordered distributed key-value system, shards can be included; there can be one, but usually multiple. Specifically, a shard refers to dividing the entire dataset into multiple smaller, independent parts, each called a shard. Each shard is responsible for storing a portion of the data and can be managed and manipulated independently. For example, refer to... Figure 2Fig. 1 is a schematic view of a fragmented log according to an embodiment of the present application, the fragments have corresponding ranges and journals. Specifically, a range is an ordered interval, representing a collection of ordered key-value ranges in an ordered KV storage system. A journal is a log of a distributed KV system, which guarantees the reliability of data by recording logs after receiving IO events. The journal is based on ChunkList, and thus has inherent storage reliability. ChunkList is a data stream composed of a group of chunks, which can break through the size limit of a single chunk to achieve the infinite append feature of storage. A chunk refers to a data unit that can be read and written in the LAVA storage system, which supports random read / sequential write. The chunk has a size limit, generally 128 MB.

[0043] In a specific implementation, shard splitting is a common scenario. For example, when the data interval of a shard is [a, f], when the [a, f] data is too much, [a, f] needs to be split into [a, d] and [d, f] two data intervals. After the splitting of the shard, the corresponding log needs to be generated for the split shard to ensure the reliability of the data of the split shard.

[0044] In an embodiment of the present application, when a certain slice splits another split slice, the log metadata of the log (Journal) corresponding to the slice is obtained. Specifically, the log metadata of a Journal includes shard_id, range, and chunk_list. Wherein, shard_id represents which shard of the storage system the Journal belongs to, range represents the range of valid data in the Journal, chunk_list is composed of a series of chunks, and the metadata of a Chunk includes chunk_id, chunk_len, Is_shared, and share_list. Wherein, chunk_id represents the identity of the chunk, chunk_len represents the length of the chunk, Is_shared represents whether the chunk is shared, and share_list represents which shard shares the chunk.

[0045] By default, chunks are not shared, so shared_list is empty by default. The size of each chunk is limited to 128M, and when the chunk space is full, the shard needs to apply for a new chunk and add the chunk_id of the new chunk to the chunk_list. The journal metadata of the Journal is all persisted to the etcd (a kind of distributed key-value storage system), to ensure the reliability of the journal metadata.

[0046] Step 102, splitting a split log metadata from the log metadata;

[0047] In the embodiment of the application, after obtaining the log metadata of the log, a split log metadata can be split from the log metadata, and at this time, the split log metadata can be the same as the log metadata.

[0048] Step 103, modifying the shard identifier of the shard in the split log metadata to the shard identifier of the split shard to obtain target split log metadata;

[0049] In the embodiment of the application, for the split log metadata, the shard_id (shard identifier) corresponding to the split log metadata can be modified to the shard_id (shard identifier) of the split shard, which represents that the shard corresponding to the split log metadata is the split shard. The split log metadata after modifying the shard identifier of the shard will be used as the target split log metadata.

[0050] Step 104, after writing the shard identifier of the shard and the shard identifier of the split shard in the shared list of the log metadata and the target split log metadata, persistently storing the log metadata and the target split log metadata in the storage system.

[0051] In the embodiment of the application, after writing the shard_id of the shard and the shard_id of the split shard in the shared list (share_list) of the chunk (data unit) of the log metadata and the target split log metadata, it is indicated that the shard and the split shard can share the chunk, and then the log metadata and the target split log metadata can be persistently stored in the storage system. It can be understood that the same chunk can be shared by the shard before splitting and the split shard, which can reduce data redundancy and improve the utilization rate of storage resources of the storage system.

[0052] In an example of log splitting, with reference to Figure 3Figure 1 is a schematic diagram of log metadata splitting provided in an embodiment of the present application. In the log splitting process, Journal is first disabled for writing, and then the Journal log metadata is split into two parts (i.e. log metadata and split log metadata), from which Figure 3 As can be seen, the Journal with shard_id = 1 splits a new Journal (split log) with shard_id = 2. Moreover, the Journal with shard_id = 1 is responsible for the data Range from [a, f] to [a, d]. The new Journal with shard_id = 2 is responsible for the data Range [d, f]. The data unit metadata is_shared of each chunk in the chunk_list becomes true, and the shared_list becomes "1, 2", indicating that the chunk is shared by the slices corresponding to shard_id = 1 and shard_id = 2.

[0053] Persisting the log metadata of the Journal with shard_id = 1 and shard_id = 2 together in etcd (a kind of distributed key-value pair storage system) means that the log splitting is completed. It should be noted that, since the process of generating the log metadata is performed in memory, the persistent log metadata will not be damaged, and the generated log metadata of the two journals is persisted in etcd together, so the process is atomic. Atomicity refers to the characteristic that an event is either successful or failed, and there is no intermediate state.

[0054] In the log splitting method, the method is applied to a storage system, the storage system includes shards, the shards respectively have corresponding data intervals and logs, data in the data interval corresponding to each shard is written into the corresponding log, the log includes a plurality of data units, when a split shard is split from a shard, log metadata of a log corresponding to the shard is acquired; the log metadata includes a shard identifier corresponding to the shard and a data unit list; data unit metadata of a data unit in the data unit list, the data unit metadata includes a data unit identifier and a shared list; the shared list includes shard identifiers of shared data units; split log metadata is split from the log metadata; a shard identifier of the shard in the split log metadata is modified to shard identifiers of a split shard pair to obtain target split log metadata; after the shard identifier of the shard and the shard identifier of the split shard are written into the shared lists of the log metadata and the target split log metadata, the log metadata and the target split log metadata are stored persistently in the storage system. In the embodiment of the application, after the shard is split, split log metadata of a log is split based on the log metadata of the log corresponding to the shard, so that two new logs are generated, then the shard identifiers of the two logs are written into the shared lists of the data units of the log metadata of the two logs, at the same time, the shard identifier of the shard in the split log metadata is modified to the shard identifiers of the split shard pair to obtain the target split log metadata, and the log metadata and the target split log metadata can be stored persistently in the storage system, the log splitting is completed, no data needs to be copied from a part of data of the log of the shard to another log to implement the log splitting, no data is copied in the log splitting process, and a large number of operations do not need to be involved, the cost of the log splitting is small, the time consumption of the splitting process is short, and various exceptions in the splitting process do not need to be processed.

[0055] In an embodiment of the application, after the shard identifier of the shard and the shard identifier of the split shard are written into the shared lists of the log metadata and the target split log metadata, and the log metadata and the target split log metadata are stored persistently in the storage system, the method can further include:

[0056] A new data unit is written into the log metadata and or the target split log metadata, and the data unit list in the log metadata and or the target split log metadata corresponding to the new data unit.

[0057] In the embodiment of the application, when the log metadata of the journal is stored persistently in the storage system, the two shards (that is, the shard and the split shard) can continue to provide a write service, and the chunk_list of the two journals is no longer the same thereafter. Exemplarily, refer to Figure 4 is a schematic diagram of writing a log after splitting a shard provided in the embodiment of the application, fromFigure 4 It can be seen that the new data of the Journal with shard_id = 1 is written into the new data unit chunk_6, the new data of the Journal with shard_id = 2 is written into the new data unit chunk_7, and chunk_6 and chunk_7 are not shared, so the shared_list of chunk_6 and chunk_7 is empty.

[0058] It should be further noted that the split Journal is split again, because one shard can be split multiple times, so the Journal corresponding to the slice can also be split multiple times, refer to Figure 5 is a schematic diagram of writing a log after the slice is split again in an embodiment of the present application. The process of splitting the split Journal is basically the same as before, and the difference is that the shared_list of the chunk inherits the shared_list before the split. For example: chunk_id = 3 is shared by shard = 1 and shard = 2 before the split, so the shared_list of chunk_id = 3 after the split becomes "1, 2, 3", indicating that the slices of shard = 1, shard = 2 and shard = 3 share the data unit corresponding to chunk_id = 3.

[0059] In an embodiment of the present application, the storage system includes a center node, the center node includes a local shared list and a local deletion list corresponding to the data unit, and the method can further include:

[0060] When the target data unit corresponding to the shared list is not empty, the shared list and the data unit identifier of the target data unit are sent to the center node when the slice or the split slice initiates a deletion request for the target data unit.

[0061] The center node is configured to update the local shared list according to the shared list of the target data unit, and update the local deletion list according to the data unit identifier of the target data unit; when the shard identifier in the local shared list and the local deletion list corresponding to the target data unit is the same, the deletion request of the target data unit is initiated to the shard corresponding to the shard identifier in the local deletion list.

[0062] In a specific implementation, sometimes it is necessary to perform recovery of logs of a storage system, in the embodiment of the present application, the recovery of logs is performed according to chunk granularity, and since some chunks are shared after splitting, if the chunks corresponding to the logs are directly deleted, it will cause other logs sharing the chunks to be abnormal, and therefore the shared chunks cannot be directly deleted. In view of the above problem, the embodiment of the present application introduces an intermediate node (central node) to solve the problem of deleting shared chunks.

[0063] The central node is responsible for recording the sharing information of chunks by recording two lists of chunks, one is a local shared list (shared_list) of the central node, and the other is a local deleted list (deleted_list). In the embodiment of the present application, when a certain shard initiates a deletion request for a target data unit, if the shared list (shared_list) corresponding to the target data unit is not empty, it indicates that the target data unit is a chunk shared by multiple shards, and then the shared list (shared_list) of the target data unit and the data unit identifier (chunk_id) can be sent to the central node, the central node updates the local shared list (shared_list) according to the shared list (shared_list) of the target data unit and the data unit identifier (chunk_id), and updates the local deleted list (deleted_list) according to the data unit identifier (chunk_id) of the target data unit, when the shard identifier in the local shared list (shared_list) and the local deleted list (deleted_list) corresponding to the target data unit is the same, a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deleted list, and the deletion of the target data unit of the shard is completed.

[0064] It can be seen that in the embodiment of the application, when the local shared list (shared_list) of a certain chunk of the center node is the same as the local deleted list (deleted_list), that is, the shard_id in the local shared list (shared_list) and the local deleted list (deleted_list) are the same, it means that the chunk can be truly deleted. For example, the local shared list (shared_list) and the local deleted list (deleted_list) are both "1,2", which indicates that slice 1 and slice 2 agree to delete the shared chunk, and then slice 1 and slice 2 can delete the chunk. Since the center node does not record the shared list of the chunk in the process of log splitting, only when the Journal initiates the deletion of a certain chunk (target data unit), the center node starts to record the local shared list (shared_list) and the local deleted list (deleted_list) corresponding to the chunk.

[0065] Exemplarily, referring to Figures 6a-6c , it is a schematic diagram of deleting a chunk of a log, and a deletion process of deleting chunk=3 by slice shard=2, and the specific process is as follows:

[0066] Referring to Figure 6a , the process of deleting chunk=3 by shard2 includes:

[0067] a. shard=2 initiates a deletion operation of deleting chunk=3.

[0068] b. First, shard=2 judges whether chunk=3 is a shared chunk. If it is not a shared chunk, the deletion is directly initiated.

[0069] c. If it is a shared chunk, the shared list and the deletion list of chunk=3 are sent to the center node. Here, the shared list (shared_list="1,2") and the deletion list (deleted_list="2")

[0070] d. After the center node receives the request, the shared_list="1,2" and the deleted_list="2" of chunk=3 are updated, and then the success is returned. shard=2 deletes chunk=3 from the chunk_list.

[0071] Referring to Figure 6b , the process of deleting chunk=3 by shard1 includes:

[0072] e、shard=3 initiates the delete operation of chunk=3.

[0073] f、First shard=3 judges that chunk=3 is a shared chunk, and sends shard_list=“1,2,3” and deleted_list=“3” of chunk=3 to the center node.

[0074] g、The center node updates shard_list=“1,2,3” and deleted_list=“2,3”.

[0075] Referring to Figure 6c , the process of shard1 deleting chunk=3 includes:

[0076] h、shard=1 initiates the delete operation of chunk=3.

[0077] i、First shard=1 judges that chunk=3 is a shared chunk, and sends shard_list=“1,2,3” and deleted_list=“1” of chunk=3 to the center node.

[0078] j、The center node updates shard_list=“1,2,3” and deleted_list=“1,2,3”, and persists to etcd. Because shard_list is equal to deleted_list, the store layer (the storage layer of chunk) initiates the deletion of chunk.

[0079] k、When the deletion of chunk is successful, the center node deletes shard_list and deleted_list of chunk=3 from etcd, so as to avoid wasting unnecessary storage resources.

[0080] In an embodiment of the present application, the method can further include:

[0081] When the shard or the split shard initiates a deletion request for a target data unit, if a shared list corresponding to the target data unit is empty, the target data unit in the shard or the split shard is deleted.

[0082] In an embodiment of the present application, when a certain shard initiates a deletion request for a target data unit, if a shared list (shared_list) corresponding to the target data unit is empty, it indicates that the target data unit is not a shared chunk, and at this time, the target data unit in the shard or the split shard is deleted.

[0083] The embodiment of the application provides a scheme for splitting one log into two logs quickly and atomically based on a storage system, the core idea of which is that data is not copied during splitting but journal log metadata is split to achieve the purpose of quickly splitting the journal, and the two newly generated journals share data by sharing chunks (data units), and in addition, only the chunks that are not required by each journal need to be filtered out during log playback.

[0084] In order to make the skilled in the art better understand, the following uses a specific example to describe the process of implementing the log splitting function in the distributed KV system.

[0085] 1. Write Journal (log):

[0086] a. The initial state of the entire distributed KV system has only one Shard (slice), which means that all data falls in this Shard. Each Shard has a Journal corresponding thereto, and each data write request is first persisted by writing the Journal.

[0087] Shard{

[0088] Shard_id; 1

[0089] Journal{

[0090] Range{begin="a",end="f"}

[0091] ChunkList{

[0092] {chunk_id=1,chunk_length=12M,is_shared=false,shared_list=""},

[0093] {chunk_id=2,chunk_length=64M,is_shared=false,shared_list=""}

[0094] }

[0095] }

[0096] }

[0097] b. The Journal is composed of multiple Chunks, and the Journal log metadata records the Chunk list

[0098] (chunk metadata), each chunk record in chunk list records the length of chunk, chunk_id, and whether the chunk is shared, which shard shares it and so on. Journal log metadata is persisted through etcd to ensure the reliability of log metadata.

[0099] c. When data is continuously written, Journal will apply for a new chunk, the initial state of the new chunk is not shared, so the shared list is empty. Add the new chunk to ChunkList.

[0100] 2. Journal split:

[0101] a. When the data of a shard is too much, it needs to be split into two shards. Therefore, the Journal corresponding to the shard also needs to be split into two Journal.

[0102] b. Journal split is completed through the replication of Journal log metadata. Copy a new Journal log metadata, then modify the Range (valid data interval) of the two Journal, and change each chunk in the log metadata to a shared chunk and add the shared list of the chunk.

[0103] c. After generating two new Journal log metadata, write the log metadata of the two Journal to etcd at one time for persistence.

[0104] d. After writing to etcd successfully, the entire Journal splitting process is completed. Since the present application writes to etcd at one time, the Journal splitting process is atomic.

[0105] 3. Chunk recycling (log recycling):

[0106] a. After a period of time, the data in some chunks of Journal becomes invalid, so the chunk needs to be deleted to release the storage space of the underlying storage.

[0107] b. After Journal split, some chunks are shared by two Journal. When one Journal deletes it, it needs to be reported to the central node shared_list and deleted_list.

[0108] c. The center node merges the received shared_list and deleted_list with the locally recorded shared_list and deleted_list. If the shared_list == deleted_list is completely consistent, a DeleteChunk request is initiated to the StoreLayer.

[0109] By applying the embodiments of the present application, when the log is split, data does not need to be migrated, but only the log metadata is modified through the shared chunk, and the log split is fast and atomic by persisting the newly generated log metadata to etcd at one time, and the entire split process can be completed in milliseconds. In addition, the embodiments of the present application creatively introduce the center node / shared_list / deleted_list, solve the problem that the deletion of the shared chunk affects other shared slices, and do not produce the problem of mistakenly deleting the chunk and residual chunk.

[0110] It should be noted that for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0111] Reference Figure 7 The structure block diagram of a log splitting device provided in the embodiments of the present application is shown in FIG. 1, which is applied to a storage system, wherein the storage system includes shards, and each shard has a corresponding data interval and a log, the data of the data interval corresponding to each shard is written into the corresponding log, and the log includes a plurality of data units. Figure 7 The device can specifically include the following modules:

[0112] The shard splitting module 701 is configured to, when a split shard is split from a shard, acquire log metadata of a log corresponding to the shard; the log metadata includes shard identification corresponding to the shard and a data unit list; the data unit metadata of a data unit in the data unit list includes data unit identification and a shared list; the shared list includes shard identification sharing the data unit;

[0113] The log splitting module 702 is configured to split split log metadata from the log metadata.

[0114] The sharding identifier modification module 703 is configured to modify the sharding identifier of the shard in the split log metadata as the sharding identifier of the split shard to obtain target split log metadata.

[0115] The persistence module 704 is configured to, after writing the sharding identifier of the shard and the sharding identifier of the split shard in the shared list of the log metadata and the target split log metadata, persistently store the log metadata and the target split log metadata in the storage system.

[0116] In an embodiment of the present application, the apparatus can further include:

[0117] In the log metadata and or the target split log metadata, a new data unit is written, and the new data unit is added to the data unit list in the log metadata and or the target split log metadata.

[0118] In an embodiment of the present application, the storage system includes a center node, the center node includes a local shared list and a local deletion list corresponding to the data unit, and the apparatus further includes:

[0119] When the shard or the split shard initiates a deletion request for a target data unit, if the shared list corresponding to the target data unit is not empty, the shared list and the data unit identifier of the target data unit are sent to the center node.

[0120] The center node is configured to update the local shared list according to the shared list of the target data unit, and update the local deletion list according to the data unit identifier of the target data unit; and when the shard identifier in the local shared list and the local deletion list corresponding to the target data unit is the same, initiate a deletion request for the target data unit to the shard corresponding to the shard identifier in the local deletion list.

[0121] In an embodiment of the present application, the apparatus can further include:

[0122] When the shard or the split shard initiates a deletion request for a target data unit, if the shared list corresponding to the target data unit is empty, the target data unit in the shard or the split shard is deleted.

[0123] In an embodiment of the present application, the center node is configured to, when the shard corresponding to the shard identifier in the local deletion list successfully deletes the target data unit, delete the local shared list and the local deletion list corresponding to the shard corresponding to the shard identifier in the local deletion list.

[0124] In an embodiment of the present application, the log metadata can further include a data interval.

[0125] In an embodiment of the present application, the data unit metadata can further include a data unit length and whether the data unit is shared; and the whether the data unit is shared is closed by default.

[0126] In an embodiment of the present application, the application is applied to a storage system, the storage system includes shards, the shards respectively have corresponding data intervals and logs, data of the data interval corresponding to each shard is written into the corresponding log, the log includes a plurality of data units, when a split shard is split from a shard, log metadata of the log corresponding to the shard is acquired; the log metadata includes a shard identifier corresponding to the shard and a data unit list; data unit metadata of a data unit in the data unit list, the data unit metadata includes a data unit identifier and a sharing list; the sharing list includes a shard identifier of a shared data unit; a split log metadata is split from the log metadata; a shard identifier of the shard in the split log metadata is modified to a shard identifier of a split shard pair to obtain a target split log metadata; after the shard identifier of the shard and the shard identifier of the split shard are written into the sharing list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are stored persistently in the storage system. In the embodiment of the present application, after the shard is split, a split log metadata of a log is split based on the log metadata of the log corresponding to the shard, thereby generating two new logs, then the shard identifiers of the two logs corresponding to the shards are written into the sharing list of the data units of the log metadata of the two logs, at the same time, the shard identifier of the shard in the split log metadata is modified to the shard identifier of the split shard pair to obtain the target split log metadata, and the log metadata and the target split log metadata can be stored persistently in the storage system to complete log splitting. Since it is not necessary to copy part of the data of the log of the shard to another log to implement log splitting, no data is copied in the log splitting process, and a large number of operations are not involved, the cost of log splitting is small, the time consumption of the splitting process is short, and various exceptions in the splitting process do not need to be processed.

[0127] For the above-mentioned device embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and the related part can be referred to the part of the method embodiment.

[0128] It should be noted that the embodiments of the present application can involve the use of user data. In actual application, user-specific personal data can be used in the schemes described herein within the scope allowed by applicable laws and regulations, for example, the user has explicitly agreed, the user has been actually notified, etc.

[0129] The embodiments of the present application also provide an electronic device, such asFigure 8 As shown, the server includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 complete mutual communication through the communication bus 504,

[0130] The memory 503 is used for storing a computer program.

[0131] The processor 501 is used for executing the program stored in the memory 503, and realizes the log splitting method in any of the above embodiments.

[0132] When splitting a split shard from a shard, log metadata of a log corresponding to the shard is acquired; the log metadata includes shard identification and a data unit list corresponding to the shard; data unit metadata of a data unit in the data unit list includes data unit identification and a shared list; the shared list includes shard identification sharing the data unit;

[0133] Splitting split log metadata from the log metadata;

[0134] Modifying shard identification of the shard in the split log metadata to shard identification of the split shard to obtain target split log metadata;

[0135] After writing the shard identification of the shard and the shard identification of the split shard in the shared list of the log metadata and the target split log metadata, persistently storing the log metadata and the target split log metadata in the storage system.

[0136] Optionally, the method further includes:

[0137] Writing a new data unit in the log metadata and / or the target split log metadata, and adding the new data unit in a data unit list in the log metadata and / or the target split log metadata corresponding to the new data unit.

[0138] Optionally, the storage system includes a center node, and the center node includes a local shared list and a local deletion list corresponding to the data unit; the method further includes:

[0139] When the shard or the split shard initiates a deletion request for a target data unit, if a shared list corresponding to the target data unit is not empty, the shared list and data unit identification of the target data unit are sent to the center node.

[0140] The center node is configured to update the local shared list according to a shared list of the target data unit, and update the local deletion list according to a data unit identifier of the target data unit; and when the local shared list and the local deletion list of the target data unit have the same fragment identifier, initiate a deletion request of the target data unit to a fragment corresponding to the fragment identifier in the local deletion list.

[0141] Optionally, the method further comprises:

[0142] When the fragment or the split fragment initiates the deletion request for the target data unit, if the shared list corresponding to the target data unit is empty, the target data unit in the fragment or the split fragment is deleted.

[0143] The communication bus of the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0144] The communication interface is used for communication between the terminal and other devices.

[0145] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0146] The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0147] In yet another embodiment of the present application, a computer readable storage medium is provided, which stores instructions, which when executed on a computer, cause the computer to perform the log splitting method of any of the above embodiments.

[0148] In yet another embodiment of the present application, a computer program product is provided, which contains instructions, which when executed on a computer, cause the computer to perform the log splitting method of any of the above embodiments.

[0149] In the above embodiments, the implementation can be wholly or partially in software, hardware, firmware, or any combination thereof. When implemented in software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer instructions cause the computer to perform the processes or functions described in the embodiments of the present application. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, from a website, a computer, a server, or a data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, solid state disk (SSD)), etc.

[0150] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0151] The various embodiments in the specification are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the system embodiments are described more simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments.

[0152] The above only describes the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A log splitting method, characterized in that, The method is applied to a storage system, which includes shards, each shard having a corresponding data range and a log. Data from the data range corresponding to each shard is written into a corresponding log, the log comprising multiple data units. When splitting a shard from a shard, the log metadata corresponding to the shard is obtained; the log metadata includes the shard identifier and data unit list corresponding to the shard; the data unit metadata of the data units in the data unit list includes the data unit identifier and a sharing list; the sharing list includes the shard identifier that shares the data unit; Split log metadata from the aforementioned log metadata; Modify the fragment identifier of the fragment in the split log metadata to the fragment identifier of the split fragment to obtain the target split log metadata; After writing the shard identifier of the shard and the shard identifier of the split shard into the shared list of the log metadata and the target split log metadata, the log metadata and the target split log metadata are persistently stored in the storage system.

2. The method according to claim 1, characterized in that, After writing the shard identifier of the shard and the shard identifier of the split shard into the shared list of the log metadata and the target split log metadata, and after persistently storing the log metadata and the target split log metadata in the storage system, the method further includes: Write a new data unit into the log metadata and / or the target split log metadata, and add the new data unit to the list of data units in the log metadata and / or the target split log metadata.

3. The method according to claim 1, characterized in that, The storage system includes a central node, which includes a local shared list and a local deletion list corresponding to the data unit. The method further includes: When the shard or split shard initiates a deletion request for a target data unit, if the shared list corresponding to the target data unit is not empty, the shared list and data unit identifier of the target data unit are sent to the central node. The central node is used to update the local shared list according to the shared list of the target data unit, and to update the local deletion list according to the data unit identifier of the target data unit; when the shard identifier in the local shared list and the local deletion list corresponding to the target data unit is the same, a deletion request for the target data unit is initiated to the shard corresponding to the shard identifier in the local deletion list.

4. The method according to claim 3, characterized in that, The method further includes: When a shard or split shard initiates a deletion request for a target data unit, if the shared list corresponding to the target data unit is empty, the target data unit in the shard or split shard will be deleted.

5. The method according to claim 3, characterized in that, The central node is used to delete the local shared list and the local deletion list corresponding to the shard corresponding to the shard identifier in the local deletion list when the target data unit is successfully deleted.

6. The method according to claim 1, characterized in that, The log metadata also includes data ranges.

7. The method according to claim 1, characterized in that, The data unit metadata also includes the data unit length and whether the data units are shared; the data unit sharing is disabled by default.

8. A log splitting device, characterized in that, The device is applied to a storage system, which includes shards, each shard having a corresponding data range and a log. Data in the data range corresponding to each shard is written into a corresponding log, and the log includes multiple data units. The device includes: The sharding module is used to obtain log metadata of the log corresponding to the shard when splitting from a shard; the log metadata includes the shard identifier and data unit list corresponding to the shard; the data unit metadata of the data units in the data unit list includes the data unit identifier and a sharing list; the sharing list includes the shard identifier that shares the data unit; The log splitting module is used to split log metadata from the log metadata; The shard identifier modification module is used to modify the shard identifier of the shard in the split log metadata to the shard identifier of the split shard, so as to obtain the target split log metadata; The persistence module is used to persist the log metadata and the target split log metadata in the storage system after writing the shard identifier of the shard and the shard identifier of the split shard into the shared list of the log metadata and the target split log metadata.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images