Data sharding management method, system, and computing device

Abstract

The present disclosure relates to the technical field of distributed storage, and particularly provides a data shard management method, system and computing device. In the present application, according to the erasure code coding mode, a data object is divided into K data shards, and shard information of each data shard is determined, the shard information including: a version number; metadata of each data shard is stored to a metadata layer, the metadata of the data shard including the version number; and first metadata of a first data shard is queried from the metadata layer according to the version number in the shard information of the first data shard. In the present application, by redesigning the naming structure of the shard information of the data shard, the version number in the shard information can be used to quickly locate the metadata of the data shard, improve the query efficiency of the metadata of the data shard, and facilitate management of the data shard.

Description

Technical Field

[0001] This disclosure relates to the field of distributed storage technology, and in particular to a data sharding management method, system and computing device. Background Technology

[0002] In distributed storage systems, data repair typically employs a replication approach. For example, a replica file corresponding to the file to be repaired can be identified first, and then the file to be repaired can be repaired using the replica file to obtain the repaired file. However, this replication approach is not suitable for data repair in all distributed storage systems, such as Ceph (core) distributed storage systems. Therefore, how to quickly and accurately perform data migration and repair in Ceph distributed storage systems is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This disclosure is made in view of the above-mentioned problems. This disclosure provides a data fragmentation management method, system, and computing device.

[0004] According to a first aspect of this disclosure, a data fragmentation management method is provided, comprising:

[0005] Based on the erasure coding method, the data object is divided into K data fragments, and the fragment information of each data fragment is determined. The fragment information includes: version number;

[0006] The metadata of each data shard is stored in the metadata layer, and the metadata of the data shard includes a version number;

[0007] Based on the version number in the fragment information of the first data fragment, query the first metadata of the first data fragment from the metadata layer. The first data fragment is the data fragment that needs to be managed among the K data fragments of the data object.

[0008] Optionally, the sharding information also includes a bucket identifier and a shard identifier. Based on the version number in the sharding information of the first data shard, the first metadata of the first data shard is retrieved from the metadata layer, including:

[0009] Based on the bucket identifier and version number in the sharding information of the first data shard, locate the first version group to which the first data shard belongs;

[0010] Based on the fragment identifier of the first data fragment, query the first metadata of the first data fragment in the metadata layer from the first version group of the metadata layer.

[0011] Optionally, the chunk information may also include: an upload identifier, used to uniquely identify a chunk upload operation.

[0012] Optionally, the fragmentation information also includes operation status bits, which include at least one of the following:

[0013] The first status bit is used to identify whether the upload method is single object upload or chunked upload;

[0014] The second status bit is used to indicate whether the multi-version feature is enabled;

[0015] The third status bit is used to identify the user type and bucket type to represent data priority.

[0016] Optionally, based on the sharding information of the first data shard, the metadata of the first data shard is queried, including:

[0017] The second status bit of the operation status bit in the fragmentation information of the first data fragment is parsed to obtain the first parsing result;

[0018] If the first parsing result indicates that the multi-version feature is enabled, then use the version number in the fragment information of the first data fragment to query the metadata of the first data fragment;

[0019] If the first parsing result indicates that the multi-version feature is not enabled, then the latest version will be queried according to the preset query logic.

[0020] Optionally, it also includes:

[0021] Parse the first status bit of the operation status bit in the fragmentation information of the first data fragment;

[0022] If the first status bit indicates that the data is uploaded in fragments, and the upload identifier indicates that the data has not been merged in the metadata layer, then the first data fragment will be identified as an unmerged block and will be awaited for repair.

[0023] Optionally, it also includes:

[0024] Analyze the third status bit of the operation status bits in the fragmentation information of the first data fragment;

[0025] Based on the data priority represented by the third state bit, perform the repair and / or migration of the first data fragment.

[0026] Optionally, based on the data priority represented by the third state bit, the repair and / or migration of the first data fragment is performed, including:

[0027] The order of repair and / or migration of the first data fragment is determined based on the data priority represented by the third state bit.

[0028] The repair and / or migration of the first data shard shall be performed in the order of repair and / or migration of the first data shard.

[0029] Optionally, it may also include at least one of the following:

[0030] During the data repair process for the first data shard, repair monitoring is performed, and the repair status of the first data shard is output.

[0031] During the data migration process of the first data shard, a migration event is generated and sent to the client so that the client can display the migration event and provide a prompt that the data migration of the first data shard is being performed.

[0032] According to a second aspect of this disclosure, a data sharding management system is provided, comprising:

[0033] The gateway layer is used to receive data objects sent by clients;

[0034] The data layer is used to divide the data object into K data fragments according to the erasure coding method, and determine the fragment information of each data fragment. The fragment information includes: bucket identifier and fragment identifier, operation status bit and version number; the metadata of each data fragment is stored in the metadata layer, and the metadata of the data fragment includes the version number; according to the version number in the fragment information of the first data fragment, the metadata of the first data fragment is retrieved from the metadata layer. The first data fragment is the data fragment that needs to be managed among the K data fragments of the data object.

[0035] The metadata layer stores the metadata for each data shard.

[0036] According to a third aspect of this disclosure, a computing device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any data fragmentation management method.

[0037] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any data fragmentation management method.

[0038] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any data fragmentation management method.

[0039] As will be described in detail below, according to the technical solution of the embodiments of this disclosure, for a received data object, the data object can be divided into K data fragments according to erasure coding. For each data fragment, fragment information can be determined, including a bucket identifier, a fragment identifier, and a version number. The metadata of each data fragment is stored in the metadata layer. The metadata of the data fragment includes a version number. Since the version number is set in the fragment information of the data fragment, and the version number is also included in the metadata of the data fragment, by redesigning the naming structure of the fragment information of the data fragment, the version number in the fragment information is used to query the first metadata of the first data fragment from the metadata layer. The first data fragment is the data fragment that needs to be managed among the K data fragments of the data object. The version number can quickly locate the metadata of the data fragment, improve the query efficiency of the metadata of the data fragment, and facilitate the management of the data fragment.

[0040] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0041] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0042] Figure 1 This is a schematic diagram illustrating a data fragmentation management system according to an embodiment of the present disclosure;

[0043] Figure 2 This is a flowchart illustrating a data fragmentation management method according to an embodiment of the present disclosure;

[0044] Figure 3 This is an example diagram illustrating fragmentation information according to an embodiment of the present disclosure;

[0045] Figure 4 This is an example diagram illustrating a data repair and / or migration according to an embodiment of the present disclosure;

[0046] Figure 5 This is a schematic diagram of the structure of a data fragmentation management device according to an embodiment of the present disclosure;

[0047] Figure 6 This is a hardware block diagram illustrating a computing device according to an embodiment of the present disclosure. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0049] This application provides a data sharding management method, system, and computing device. By redesigning the naming structure of the data sharding information, the first metadata of the first data shard is queried from the metadata layer using the version number in the sharding information. The version number allows for quick location of the data shard's metadata, improving the efficiency of metadata retrieval and facilitating data sharding management.

[0050] The technical solution of this disclosure will now be described in detail with reference to the accompanying drawings.

[0051] like Figure 1 The diagram shown is a structural schematic of a data fragmentation management system provided in an embodiment of this application. The communication system may include a gateway layer 10, a data layer 20, and a metadata layer 30.

[0052] Gateway layer 10 is used to receive service requests sent by clients.

[0053] Data layer 20 is used to divide the data object into K data fragments according to the erasure coding method, and determine the fragment information of each data fragment. The fragment information includes: bucket identifier and fragment identifier, operation status bit and version number; store the metadata of each data fragment to the metadata layer. The metadata of the data fragment includes the version number; query the metadata of the first data fragment from the metadata layer according to the version number in the fragment information of the first data fragment. The first data fragment is the data fragment that needs to be managed among the K data fragments of the data object.

[0054] Metadata layer 30 is used to store the metadata for each data shard.

[0055] like Figure 2 The diagram shown is a flowchart of a data fragmentation management method provided in an embodiment of this application. The data fragmentation management method may include the following steps:

[0056] S201. Based on the erasure coding method, divide the data object into K data fragments and determine the fragment information of each data fragment, including: version number.

[0057] Alternatively, the data object can be a data object in the Ceph (core) distributed storage system.

[0058] Optionally, erasure coding can be used to divide the data object into K data fragments and M parity fragments.

[0059] Optionally, erasure coding parameters can be preset, such as the total number of fragments. That is, the total number of fragments = K + M. For example, the total number of fragments can be 18. If the total number of fragments is 18, then the K data fragments and M parity fragments written to the data layer can be fragments 1 to 18.

[0060] To improve data management efficiency, the naming structure of data shard information (shard name) in this embodiment is redesigned. It may include at least one of the following: bucket identifier, shard identifier, version number, operation status bit, upload identifier, other reserved bits, and shard sequence number.

[0061] The bucket ID is used to uniquely identify the storage bucket to which a shard belongs.

[0062] The fragment identifier is used to uniquely identify a fragment. It can be a 33-character randomly generated string.

[0063] The version number (version id) can be used to uniquely identify the object version to which the fragment belongs. The generation of the version id is moved forward to the initial fragmentation stage of fragment upload, replacing the existing generation logic in the merging fragmentation stage.

[0064] To make it easier to understand, the following example illustrates the fragmentation information.

[0065] Fragment information 1:

[0066] e41ec2d9-8d0d-4b35-a38a-81a1491312c3bk-std%.447134.2__multipart_.ge udDsyyx5QVZHz3XjsEiCOJjW5KvMKi1111111111111111111111111111111120M.1

[0067] Fragment information 2:

[0068] e41ec2d9-8d0d-4b35-a38a-81a1491312c3bk-std%.447134.2__shadow_.geud Dsyyx5QVZHz3XjsEiCOJjW5KvMKi1111111111111111111111111111111120M.1_2

[0069] Fragment information 3:

[0070] e41ec2d9-8d0d-4b35-a38a-81a1491312c3bk-std%.447134.2__shadow_.eXKfaxSECSZlbE6Wyqhf2NdWIdNxLvPa000000000000000000000000000000001M_0

[0071] To facilitate understanding, we will take fragment information 1 as an example to explain the specific composition of fragment information 1.

[0072] like Figure 3 The image shown is an example diagram of fragmentation information provided in an embodiment of this application. The string preceding the string "__multipart or __shadow" is as follows: Figure 3 In the code, 301 "e41ec2d9-8d0d-4b35-a38a-81a1491312c3bk-std%.447134.2" is the bucket identifier. The 33 characters following the bucket identifier are the fragment identifier, such as:

[0073] Figure 3 The 302 ".geudDsyyx5QVZHz3XjsEiCOJjW5KvMKi" in the image is the fragment identifier. The 32 bits following the fragment identifier are reserved bits, such as "111111111111111111111111111111111111111".

[0074] These 32 reserved bits can be further divided. For example, the first bit can be the operation status bit 307, used to distinguish information such as block splitting and multiple versions. Bits 2-13, that is... Figure 3 303 in the array is used to store the upload identifier. Bits 14-31, that is... Figure 3 The 304 field is used to store the version number. An upload identifier of 0 indicates a single upload. An upload identifier of 1 indicates a multi-part upload. The remaining 7 bits... Figure 3 305 in the value is a reserved bit. These reserved bits can be used to identify whether data fragments are uploaded individually or in fragments.

[0075] 306. “20M.1” is the name of the data object in the fragment.

[0076] S202. Store the metadata of each data shard in the metadata layer. The metadata of the data shard includes the version number.

[0077] Optionally, metadata includes: data storage location, attribute information, and version number.

[0078] The data storage location can refer to the path or node where data fragments are stored. Attribute information may include basic attributes, metadata attributes, storage attributes, security attributes, etc.

[0079] Basic attributes may include information such as name, type, and size. Metadata attributes may include information such as storage location, version number, and user type. Storage attributes may include information such as storage pool, replication parameters / erasure coding parameters, and security attributes may include read / write permissions and encryption information.

[0080] Optionally, metadata can be represented using JSON format. Below are some examples of metadata data structures.

[0081] Format 1: {"olh": {…..}, "obj": {…}, "bi": {….}}.

[0082] Format 2: {"head": {….}, "xid": {…}, "obj": {…..}, "bi": {…..}, "mtime": { ……}}

[0083] In Format 2, the JSON body of the `head` column (such as "xid", "obj", "bi", and "mtime") stores the specific location and attribute information of the object data. In Format 1, the JSON body of the `olh` column (such as "obj" and "bi") is mainly a field used for multiple versions. It allows you to find different version numbers of the object. If the user downloads without specifying a version number, the latest version will be downloaded by default; otherwise, the fixed version number will be downloaded.

[0084] S203. Based on the version number in the fragment information of the first data fragment, query the first metadata of the first data fragment from the metadata layer. The first data fragment is the data fragment that needs to be managed among the K data fragments of the data object.

[0085] Optionally, the first data shard may include one or more.

[0086] Optionally, data shard management can include querying, repairing, and migrating. The first data shard can also be the data shard among the K data shards of a data object that needs to be queried, migrated, or repaired.

[0087] Optionally, prior to S203, the method further includes: in response to a business request from a data object, determining the first data shard to be processed from the K data shards.

[0088] Optionally, in some scenarios, one or more of the K data shards of a data object may need to be managed. For example, in a repair scenario, damaged data shards can be repaired. In this case, determining the first data shard to be processed from the K data shards of the data object may include: selecting the data shard to be managed from the K data shards of the data object to obtain the first data shard.

[0089] Optionally, for some management scenarios, all K data shards of a data object need to be managed, such as in data query or migration scenarios where all data shards need to be queried or migrated. Therefore, based on the sharding order of the K data shards of the data object, the data shards are sequentially determined as the first data shard from the K data shards until the traversal of the K data shards is completed.

[0090] Optionally, during data repair and migration, operations are performed on existing data objects in the distributed storage system. To avoid data residue, it is necessary to query metadata to determine whether the data objects exist.

[0091] Furthermore, to improve the targeting of queries, the first metadata of the first data shard can be queried in a timely manner based on the version number and operation status in the shard information of the first data shard.

[0092] In this embodiment, the received data object can be divided into K data shards according to erasure coding. For each data shard, shard information can be determined, including a bucket identifier, a shard identifier, and a version number. The metadata of each data shard is stored in the metadata layer. The metadata of the data shard includes a version number. Since the version number is set in the shard information and also included in the metadata, the naming structure of the shard information can be redesigned. Using the version number in the shard information, the first metadata of the first data shard can be queried from the metadata layer. The first data shard is the data shard that needs to be managed among the K data shards of the data object. The version number allows for quick location of the data shard's metadata, improving the query efficiency of the data shard's metadata and facilitating the management of the data shards.

[0093] In one possible design, the fragmentation information also includes operation status bits, which include at least one of the following:

[0094] The first status bit is used to identify whether the upload method is single object upload or chunked upload;

[0095] The second status bit is used to indicate whether the multi-version feature is enabled;

[0096] The third status bit is used to identify the user type and bucket type to represent data priority.

[0097] like Figure 3 As shown, the first bit in the reserved bits can be an operation status bit, used to distinguish information such as fragmentation and multiple versions. The operation status bit can be 1 byte / 8 bits, and can be used to identify the operation attributes and priorities of data fragmentation. The status operation bit can, for example, include 0-7 bits.

[0098] The first status bit can be bit 0 in the operation status bit set. The second status bit can be bit 1 in the operation status bit set. The third status bit can be bits 2-7 in the operation status bit set.

[0099] Furthermore, the third status bit may include a user type status bit and a bucket type status bit.

[0100] For example, the 2nd and 3rd bits of the operation status can be user type status bits, specifically used to identify the user type. For example, 00 represents a regular user, 01 represents a VIP user, 10 represents an enterprise user, and 11 represents a core user.

[0101] Bits 4 and 5 of the operation status bits can be the bucket type status bits, specifically used to identify the bucket type, which determines the priority. For example, 00 represents a low-priority bucket, 01 represents a medium-priority bucket, 10 represents a high-priority bucket, and 11 represents a core bucket.

[0102] Bits 6 and 7 of the operation status can be reserved and can be used for extended attribute identifiers.

[0103] Understandably, user type refers to the categories set for different groups of users after dividing them into different groups. For example, based on registration time, users can be divided into new users, active users, dormant users, and churned users. Based on behavior, users can be divided into paying users, free users, content consumers, etc. Based on user value, users can be divided into high-value users, ordinary users (medium-value users), and marginal users (low-value or no-value users), etc.

[0104] A bucket is a top-level container for managing "objects" such as files, images, and videos. Buckets can be categorized into different types based on data access frequency and cost, such as standard buckets, low-frequency access buckets, and archive buckets. Alternatively, buckets can be classified according to parameters such as data popularity and query demand, such as hot data buckets, warm data buckets, cold data buckets, and hash buckets.

[0105] Optionally, after determining the user type and bucket type through the third status bit, the data priority can be determined based on the user type and bucket type corresponding to the third status bit.

[0106] Specifically, multiple priorities are preset, such as first priority P0, second priority P1, third priority P2, fourth priority P3, etc., with P0>P1>P2>P3.

[0107] Priority tables can be pre-built based on multiple user types and multiple bucket types.

[0108] User types can include: high-value users, regular users, and marginal users. Bucket types can include: hot data buckets, warm data buckets, cold data buckets, and hash buckets.

[0109] Optionally, priorities can be set for multiple user types under each bucket type. For example, under multiple bucket types, each bucket type can be associated with multiple user types, each with its own priority.

[0110] Alternatively, it can be described as setting priorities for multiple bucket types under each user type. For example, under multiple user types, each user type is associated with multiple bucket types, each with its own priority.

[0111] For ease of understanding, a priority table can be created with user type as the column and bucket type as the row. The priority table is shown in Table 1 below.

[0112] Table 1

[0113]

[0114] After determining the user type and bucket type using the third status bit, the data priority corresponding to the user type and bucket type indicated by the third status bit can be obtained by querying the priority table. For example, the data priority corresponding to a regular user and a warm data bucket is P2. The data priority corresponding to a regular user and a hash bucket is also P2. Of course, the priority table in Table 1 is only an example and does not constitute a specific limitation.

[0115] In this embodiment of the application, by dividing the operation status bits more finely, it is possible to mark information such as the upload method of data shards, whether it is a multi-version function, user type and bucket type, and perform corresponding management switching on data shards in a timely manner, thereby improving the accuracy of data shard processing.

[0116] In another possible design, the chunk information also includes an upload ID, which uniquely identifies each chunk upload.

[0117] Upload identifiers can include the following two applications:

[0118] Application 1, Repair Management

[0119] If the data shard information contains an upload identifier but is not associated with a merge completion identifier, it is determined to be an unmerged shard. An event notification is triggered via the upload ID to remind the user to complete data repair or clean up shards as soon as possible.

[0120] If the data shard information contains an upload identifier and has been associated with a merge completion identifier, query the metadata using versionid, confirm the object's integrity, and then perform data repair.

[0121] Among them, performing data repair can refer to using data fragmentation to reconstruct lost or damaged data objects.

[0122] Application 2, Migration Management

[0123] If a user migrates a data object, the system iterates through the fragment information of the corresponding data fragment to be migrated. Unmerged fragments are identified by the upload identifier in the fragment information. The system can then notify the user to process these unmerged fragments or perform data repair to prevent data remnants. For merged fragments, the system queries the metadata of the data fragment to be migrated using its version number and migrates the metadata to ensure that the migrated metadata is accessible.

[0124] In this embodiment, an upload identifier is set in the fragment information. The upload identifier is used to uniquely identify the data fragments uploaded in one go, thereby identifying whether the data fragments are unmerged fragments and improving the query efficiency of the metadata of the data fragments.

[0125] Optionally, before dividing the data object into K data fragments according to the erasure coding method and determining the fragment information of each data fragment, the following steps are also included:

[0126] The system receives business requests from clients, including requests for data storage, deletion, update, query, migration, and repair. A data storage request is a request to store a data object in a distributed storage system. A data deletion request is a request to delete a data object from the distributed storage system. A data update request is a request to update the data of a data object. A data query request is a request to query a data object in the distributed storage system. A data migration request is a request to migrate a data object from a source node to a target node in the distributed storage system. A data repair request is a request to repair data objects in the distributed storage system.

[0127] Of course, before executing business requests for data deletion / update / query / migration / repair, a data storage request must be executed first. That is, executing S201-S203 stores the data object in the data layer in the form of shards, and stores the metadata of the data shards in the metadata layer, so as to facilitate subsequent data deletion / update / query / migration / repair processes.

[0128] Depending on the different business requests, the data fragmentation management method provided in this application may include the following embodiments.

[0129] Example 1: Data Query.

[0130] A1. Parse the second status bit of the operation status bit in the fragmentation information of the first data fragment to obtain the first parsing result.

[0131] A2. If the first parsing result indicates that the multi-version function is enabled, then use the version number in the shard information of the first data shard to query the metadata of the first data shard.

[0132] A3. If the first parsing result indicates that the multi-version function is not enabled, then the metadata of the latest version will be queried according to the preset query logic.

[0133] Optionally, the preset query logic can refer to the metadata query logic used in related technologies. For example, instead of using version numbers, it directly uses the shard identifier of the data shard to query the metadata of the data shard. This type of query, which uses the shard identifier directly instead of version numbers, requires a direct query from the metadata database, resulting in a large query volume. However, since the multi-version feature is not enabled, only the metadata of one version needs to be queried, resulting in a smaller query volume compared to the multi-version query with the multi-version feature.

[0134] Optionally, before executing A1, the following steps are also included:

[0135] A4. Receive the business request sent by the client. The business request is a data query business request, which includes the data object to be queried.

[0136] A5. In response to a data query business request, query the first data shard of the data object to be queried and obtain the shard information of the first data shard.

[0137] Optionally, after executing A3, the following may also be included:

[0138] A6. Output the first data shard and its metadata obtained from the query.

[0139] In this embodiment of the application, by parsing the second flag bit, it can be determined whether the multi-version function has been enabled. Thus, by determining whether the multi-version function is enabled, the decision on which metadata query method to use can be made, thereby enabling fast metadata query. Compared with querying metadata directly from multiple versions, a large number of invalid queries can be reduced.

[0140] Example 2: Data Repair.

[0141] B1. Parse the first status bit of the operation status bit in the fragmentation information of the first data fragment;

[0142] B2. If the first status bit indicates that it is a fragmented upload, and it is determined from the upload identifier that it has not been merged in the metadata layer, then the first data fragment is identified as an unmerged block and waits for repair.

[0143] Optionally, before executing B1, the following may also be included:

[0144] B3. Receive the business request sent by the client. The business request is a data merging business request, which includes the data objects to be merged.

[0145] B4. In response to a data merging business request, obtain the first data shard that needs to be merged for the data object to be queried, and obtain the sharding information of the first data shard.

[0146] The steps for repairing data shards can be found in the following embodiments, and will not be repeated here.

[0147] In this embodiment, by parsing the first status bit, if the first status bit indicates that the data is being uploaded in fragments, the metadata layer is determined to be unmerged based on the upload identifier. This allows the first data fragment to be identified as an unmerged block and awaits repair. By identifying the allocation type, fragment management can be more effective, improving the security of data fragmentation.

[0148] Example 3: Data Migration / Repair

[0149] C1. Parse the third status bit of the operation status bit in the fragmentation information of the first data fragment.

[0150] C2. Based on the data priority represented by the third state bit, perform the repair and / or migration of the first data fragment.

[0151] Optionally, based on the data priority represented by the third state bit, the repair and / or migration of the first data fragment is performed, including:

[0152] The order of repair and / or migration of the first data fragment is determined based on the data priority represented by the third state bit.

[0153] The repair and / or migration of the first data shard shall be performed in the order of repair and / or migration of the first data shard.

[0154] Specifically, if the first data shard is determined to be important data based on data priority, then the repair and / or migration of the first data shard will be executed first to complete the data repair and / or migration.

[0155] Furthermore, the data migration steps include: outputting disk check events to prompt the user to check for data loss on the disk; determining the disk check results and troubleshooting results; determining whether the disk meets the security requirements for important data; if not, migrating the first data shard to a disk that meets the security requirements for important data; and ending the data repair and / or migration if the requirements are met.

[0156] If the first data shard is determined to be non-critical based on data priority, the repair task for the first data shard is added to the repair queue, waiting for the repair queue to schedule the repair task for the first data shard, perform data repair, and then end the data repair and / or migration.

[0157] like Figure 4 The diagram shown is an example of data repair and / or migration provided in an embodiment of this application.

[0158] 401. First data fragment detected.

[0159] 402. Determine the data priority represented by the third flag bit of the first data segment.

[0160] 403. Determine whether the first data shard is important based on data priority; if not, execute 404; if so, execute 405 and 406.

[0161] 404. Add the repair task for the first data shard to the repair queue, wait for the repair queue to schedule the repair task for the first data shard, and execute 405.

[0162] 405. Perform data repair, then proceed to 410.

[0163] 406. Output disk check event. The disk check event is used to prompt the user to check whether there is any data loss on the disk.

[0164] 407. Determine the disk check results and troubleshooting results.

[0165] 408. Determine if the disk meets the security requirements for important data. If not, proceed to 409. If it does, proceed to 410.

[0166] 409. Migrate the first data shard to a disk that meets the important data security requirements, and then execute 410.

[0167] 410. End data repair and / or migration.

[0168] Optionally, it may also include at least one of the following:

[0169] During the data repair process for the first data shard, repair monitoring is performed, and the repair status of the first data shard is output.

[0170] During the data migration process of the first data shard, a migration event is generated and sent to the client so that the client can display the migration event and provide a prompt that the data migration of the first data shard is being performed.

[0171] Unmerged data blocks, if not detected and merged in a timely manner, will lead to data loss. In data migration scenarios, such as when the current data pool is taken offline or decommissioned, data migration is necessary. If unmerged data blocks are detected during the migration process, users can be notified through migration events, allowing them to merge the blocks or clean up fragments in a timely manner.

[0172] In this embodiment, the third status bit can be parsed, and the data priority represented by the third status bit can be used to perform the repair and / or migration of the first data fragment. During data repair, data with high security requirements is repaired first to ensure the security of important data.

[0173] like Figure 5 The diagram shown is a structural schematic of a data fragmentation management device 500 provided in an embodiment of this application. The data fragmentation management device 500 includes:

[0174] Sharding unit 501 is used to divide the data object into K data shards according to the erasure coding method, and to determine the sharding information of each data shard. The sharding information includes: bucket identifier, shard identifier and version number.

[0175] Storage unit 502 is used to store the metadata of each data shard to the metadata layer. The metadata of the data shard includes a version number.

[0176] The query unit 503 is used to query the first metadata of the first data shard from the metadata layer based on the version number in the shard information of the first data shard. The first data shard is the data shard that needs to be managed among the K data shards of the data object.

[0177] As one embodiment, the query unit 503 includes:

[0178] The positioning module is used to locate the first version group to which the first data shard belongs based on the bucket identifier and version number in the shard information of the first data shard.

[0179] The query module is used to query the first metadata of the first data shard in the metadata layer from the first version group of the metadata layer based on the shard identifier of the first data shard.

[0180] As another embodiment, the fragmentation information also includes an operation status bit, which includes at least one of the following:

[0181] The first status bit is used to identify whether the upload method is single object upload or chunked upload;

[0182] The second status bit is used to indicate whether the multi-version feature is enabled;

[0183] The third status bit is used to identify the user type and bucket type to represent data priority.

[0184] As another embodiment, the fragment information also includes: an upload identifier, used to uniquely identify a fragment upload operation.

[0185] As another embodiment, the query unit includes:

[0186] The parsing module is used to parse the second status bit of the operation status bit in the fragmentation information of the first data fragment to obtain the first parsing result;

[0187] The first query module is used to query the metadata of the first data shard by using the version number in the shard information of the first data shard if the first parsing result indicates that the multi-version function is enabled.

[0188] The second query module is used to query the metadata of the latest version according to the preset query logic if the first parsing result indicates that the multi-version function is not enabled.

[0189] As yet another embodiment, it also includes:

[0190] The first parsing unit is used to parse the first status bit of the operation status bit in the fragmentation information of the first data fragment.

[0191] The chunk identification unit is used to identify the first data chunk as an unmerged chunk and wait for repair if the first status bit indicates that it is a chunked upload and the upload identifier indicates that it has not been merged in the metadata layer.

[0192] As yet another embodiment, it also includes:

[0193] The second parsing unit is used to parse the third status bit of the operation status bit in the fragmentation information of the first data fragment.

[0194] The migration and repair unit is used to perform repair and / or migration of the first data fragment according to the data priority represented by the third state bit.

[0195] Furthermore, the migration repair unit includes:

[0196] The sequence determination module is used to determine the order of repair and / or migration of the first data fragment based on the data priority represented by the third status bit.

[0197] The migration repair module is used to perform the repair and / or migration of the first data shard according to the order of repair and / or migration of the first data shard.

[0198] Furthermore, it also includes at least one of the following:

[0199] The status monitoring unit is used to perform repair monitoring and output the repair status of the first data fragment during the data repair process of the first data fragment.

[0200] The migration notification unit is used to generate a migration event during the data migration process of the first data shard and send the migration event to the client so that the client can display the migration event and provide a notification of the data migration of the first data shard.

[0201] The specific steps and technical effects performed by each unit, module, and sub-module in the device in this embodiment are the same as those in the embodiment shown in the method, and will not be repeated here.

[0202] Figure 6 The diagram shows a schematic of a computing device 600 provided in an embodiment of the present disclosure. The computing device 600 may include a processor 601 and a memory 602, wherein the processor 601 and the memory 602 are connected via a bus 603.

[0203] The memory 602 can be used to store computer programs, and the processor 601 can be used to execute the computer programs stored in the memory 602 to implement any of the data fragmentation management methods in the above embodiments.

[0204] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0205] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0206] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0207] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0208] Various changes, substitutions, and modifications can be made to the technology herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0209] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0210] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A data fragmentation management method, characterized in that, include: Based on erasure coding, the data object is divided into K data shards, and the shard information of each data shard is determined. The shard information includes: bucket identifier and shard identifier, version number, and operation status bit. The bucket identifier is used to uniquely identify the storage bucket to which the shard belongs. The shard identifier is used to uniquely identify the shard. The version number is used to uniquely identify the object version of the data object to which the shard belongs. The operation status bit includes: a second status bit, used to indicate whether the multi-version function is enabled. The metadata of each data shard is stored in the metadata layer, and the metadata of the data shard includes a version number; Receive business requests sent by clients, including data storage / deletion / update / query / migration / repair requests; In response to a business request from the data object, a first data shard to be processed is determined from the K data shards; The second status bit of the operation status bit in the fragmentation information of the first data fragment is parsed to obtain the first parsing result; If the first parsing result indicates that the multi-version function is enabled, then based on the version number in the sharding information of the first data shard, the first metadata of the first data shard is queried from the metadata layer. The first data shard is the data shard that needs to be managed among the K data shards of the data object. The step of querying the first metadata of the first data shard from the metadata layer based on the version number in the shard information of the first data shard includes: Based on the bucket identifier and version number in the fragment information of the first data fragment, locate the first version group to which the first data fragment belongs; Based on the fragment identifier of the first data fragment, query the first metadata of the first data fragment in the metadata layer from the first version group of the metadata layer; The operation status bit also includes a third status bit, which is used to identify the user type and bucket type to characterize the data priority; The method further includes: If the first data fragment is determined to be important data based on the data priority represented by the third flag bit of the first data fragment, and if the disk does not meet the security requirements of the important data, the first data fragment is migrated to a disk that meets the security requirements of the important data.

2. The method according to claim 1, characterized in that, The segmentation information also includes: an upload identifier, which is used to uniquely identify a segmented upload operation.

3. The method according to claim 1 or 2, characterized in that, The fragmentation information also includes an operation status bit, which includes at least one of the following: The first status bit is used to identify whether the upload method is single object upload or chunked upload.

4. The method according to claim 3, characterized in that, The step of querying the metadata of the first data shard based on its sharding information further includes: If the first parsing result indicates that the multi-version feature is not enabled, then the metadata of the latest version will be queried according to the preset query logic.

5. The method according to claim 3, characterized in that, Also includes: Parse the first status bit of the operation status bit in the fragmentation information of the first data fragment; If the first status bit indicates that the data is uploaded in fragments, and the upload identifier indicates that the data has not been merged in the metadata layer, then the first data fragment will be identified as an unmerged block and will be awaited for repair.

6. The method according to claim 3, characterized in that, Also includes: Parse the third status bit of the operation status bit in the fragmentation information of the first data fragment; Based on the data priority represented by the third state bit, the repair and / or migration of the first data fragment is performed.

7. The method according to claim 6, characterized in that, The step of repairing and / or migrating the first data fragment according to the data priority represented by the third state bit includes: The order of repairing and / or migrating the first data fragment is determined based on the data priority represented by the third state bit. The repair and / or migration of the first data shard are performed according to the order of repair and / or migration of the first data shard.

8. The method according to claim 6, characterized in that, It also includes at least one of the following: During the data repair process of the first data shard, repair monitoring is performed, and the repair status of the first data shard is output. During the data migration process of the first data shard, a migration event is generated and sent to the client so that the client can display the migration event and provide a prompt that the first data shard is undergoing data migration.

9. A data sharding management system, characterized in that, include: The gateway layer is used to receive business requests sent by clients. These business requests include data storage business requests, which refer to requests to store data objects in a distributed storage system. The data layer is used to divide the data object into K data shards according to the erasure coding scheme, and determine the shard information of each data shard. The shard information includes: bucket identifier and shard identifier, operation status bit and version number. The bucket identifier is used to uniquely identify the storage bucket to which the shard belongs, the shard identifier is used to uniquely identify the shard, the version number is used to uniquely identify the object version of the data object to which the shard belongs, and the operation status bit also includes: a second status bit, used to indicate whether the multi-version function is enabled; the metadata of each data shard is stored in the metadata layer, and the metadata of the data shard includes the version number; the second status bit of the operation status bit in the shard information of the first data shard is parsed to obtain a first parsing result; if the first parsing result indicates that the multi-version function is enabled, the metadata of the first data shard is queried from the metadata layer according to the version number in the shard information of the first data shard, and the first data shard is the data shard that needs to be managed among the K data shards of the data object; The metadata layer stores the metadata for each data shard. Specifically, the data layer performs the step of querying the first metadata of the first data shard from the metadata layer based on the version number in the shard information of the first data shard, which includes: Based on the bucket identifier and version number in the fragment information of the first data fragment, locate the first version group to which the first data fragment belongs; Based on the fragment identifier of the first data fragment, query the first metadata of the first data fragment in the metadata layer from the first version group of the metadata layer; The sharding information also includes a third status bit, which is used to identify the user type and bucket type to characterize the data priority; The data layer is also used for: If the first data fragment is determined to be important data based on the data priority represented by the third flag bit of the first data fragment, and if the disk does not meet the security requirements of the important data, the first data fragment is migrated to a disk that meets the security requirements of the important data.

10. A computing device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the data fragmentation management method as described in any one of claims 1-8.

Images