An ordered patching and deletion method for a distributed storage system

Abstract

The application discloses an ordered repairing and deleting method of a distributed storage system, and comprises the following steps: constructing a repairing partition key for repairing information, and storing the repairing information in the distributed storage in order based on the repairing partition key; constructing a deleting partition key for deleting information, and storing the deleting information in the distributed storage in order based on the deleting partition key; distributing each erasure code group to a corresponding repairing work process and deleting work process, so that the repairing work process is configured with the repairing partition key, the corresponding repairing information is pulled from the distributed storage, a data repairing operation is performed, the deleting work process is configured with the deleting partition key, the corresponding deleting information is pulled from the distributed storage, and a data deleting operation is performed. The application effectively avoids problems such as excessive system memory pressure, data reliability decline and unstable system operation caused by unordered organization and unordered scheduling, and improves the stability and scheduling efficiency of the repairing and deleting tasks under the Quorum intelligent read-write architecture.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to an orderly repair and deletion method, apparatus, device, medium and product for a distributed storage system. Background Technology

[0002] The cross-domain online erasure coding Quorum (majority arbitration) intelligent read-write architecture is a majority arbitration consistency mechanism for distributed storage. To improve data reliability, system availability, and reduce storage costs, distributed storage systems use erasure coding redundancy groups (Poolsets) as the smallest unit of data management. The original business data is split into K data shards, and M verification shards are generated simultaneously. The system adopts the Quorum intelligent read-write mechanism, abandoning traditional strong consistency writes. Only when a majority of members in the erasure coding group complete the read-write operation can the business read-write operation be considered successful. Simultaneously, it will detect business write failures or scans. When a data shard is missing, repair information is generated. During system operation, if some members of the erasure coding group fail to execute during a business write operation, or if the background scan and comparison finds that a data shard is missing, repair information will be automatically generated. If a business write fails and residual garbage data needs to be cleaned up, or if an active deletion instruction is received from the business side, deletion information will be generated accordingly. Both the repair information and the deletion information are stored in a distributed key-value store or a distributed message queue (MQ). The background repair task and deletion task consume the information asynchronously and execute the corresponding shard repair and data deletion operations to ensure the data integrity within the system and maintain the storage cleanliness of the storage system.

[0003] In existing technologies, repair and deletion information in distributed KV or MQ are organized using the partition keys of the storage medium itself, without being ordered according to the erasure code group dimension. This leads to the inability of background repair and deletion tasks to accurately identify the erasure code group to which the information belongs during high-concurrency scheduling. Consequently, tasks cannot be started on demand, and the number of concurrent tasks cannot be flexibly controlled. In extreme scenarios, multiple tasks may simultaneously perform operations on the disk of the same erasure code group. Due to the IOPS performance and disk-level QoS limitations of HDD mechanical hard drives, a large number of read, write, and delete operations will accumulate in the memory of a single storage engine, severely impacting memory stability. At the same time, if most information belongs to a single erasure code group, the repair and deletion information of other erasure code groups will be delayed due to task congestion. Missing fragments cannot be repaired in a timely manner, and garbage data will remain for a long time, significantly reducing the data reliability of the storage system and causing unpredictable negative impacts on the stable operation of the distributed storage system. Summary of the Invention

[0004] This invention provides an ordered repair and deletion method for a distributed storage system, which can solve the problems of single-machine storage engine stability and data reliability caused by disordered organization of deleted / repaired information and chaotic task scheduling in the prior art.

[0005] In a first aspect, embodiments of the present invention provide an ordered repair and deletion method for a distributed storage system, comprising: For the repair information generated by the repair producer, a repair partition key is constructed by combining the number of missing fragments, the erasure code group identifier, and the distributed storage sub-partition number. The repair information is then stored in the distributed storage in an orderly manner based on the repair partition key. For deletion information generated by the deletion producer, a deletion partition key is constructed by combining erasure code group identifier and distributed storage sub-partition number, and the deletion information is stored in the distributed storage in an orderly manner based on the deletion partition key; Obtain a list of erasure coding groups in the distributed storage system, and uniquely assign each erasure coding group in the list to a corresponding repair worker process and a deletion worker process. The repair worker process configures a patch partition key based on the assigned erasure coding group identifier, and retrieves the corresponding patch information from the distributed storage based on the patch partition key to perform a data patching operation. Similarly, the deletion worker process configures a deletion partition key based on the assigned erasure coding group identifier, and retrieves the corresponding deletion information from the distributed storage based on the deletion partition key to perform a data deletion operation.

[0006] Furthermore, the sub-partition number is a sub-partition identifier assigned to a single erasure coding group in the distributed storage, and the value range of the sub-partition number is determined by the ratio of the maximum number of objects carried by a single erasure coding group to the maximum number of key-value pairs carried by a single distributed storage partition.

[0007] Furthermore, the step of retrieving corresponding repair information from distributed storage based on the repair partition key and performing data repair operations includes: The repair priority is determined according to the number of missing fragments in descending order; wherein, the more missing fragments there are, the higher the repair priority. According to the repair priority, the corresponding repair information is pulled from the distributed storage, and the data repair operation is performed.

[0008] Furthermore, the method also includes: After the data repair operation is successfully executed, the corresponding repair partition key and repair information will be deleted from the distributed storage. After the data deletion operation is successfully executed, the corresponding deletion partition key and deletion information will be removed from the distributed storage.

[0009] Furthermore, the method also includes: Monitor the load of the repair and deletion processes in real time; When the load of any worker process exceeds a preset threshold, the erasure coding group corresponding to that worker process is reassigned to other worker processes with suitable loads.

[0010] Furthermore, the method also includes: The allocation information of erasure coding groups is persistently stored in distributed storage, and the consistency mechanism of distributed storage ensures the data consistency of the allocation information in the cluster.

[0011] Secondly, embodiments of the present invention provide an ordered repair and deletion apparatus for a distributed storage system, comprising: The patch information organization module is used to construct a patch partition key for the patch information generated by the patch producer, using a combination of the number of missing fragments, erasure code group identifier and distributed storage sub-partition number, and to store the patch information in an orderly manner in the distributed storage based on the patch partition key; The deletion information organization module is used to construct a deletion partition key for deletion information generated by the deletion producer by using a combination of erasure code group identifier and distributed storage sub-partition number, and to store the deletion information in the distributed storage in an orderly manner based on the deletion partition key; An ordered scheduling execution module is used to obtain a list of erasure coding groups in the distributed storage system, and uniquely assign each erasure coding group in the list to a corresponding repair worker process and a deletion worker process. The repair worker process configures a patch partition key based on the assigned erasure coding group identifier, and retrieves the corresponding patch information from the distributed storage based on the patch partition key to perform data patching operations. Similarly, the deletion worker process configures a deletion partition key based on the assigned erasure coding group identifier, and retrieves the corresponding deletion information from the distributed storage based on the deletion partition key to perform data deletion operations.

[0012] Thirdly, embodiments of the present invention provide an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program; Wherein, when the processor executes the computer program, it implements the orderly repair and deletion method of the distributed storage system described in any of the first aspects above.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed, implements the ordered repair and deletion method for the distributed storage system described in any of the first aspects above.

[0014] Fifthly, embodiments of the present invention provide a computer program product, including computer instructions, which, when executed by a processor, implement the ordered repair and deletion method for a distributed storage system as described in any of the first aspects above.

[0015] Compared with existing technologies, the orderly repair and deletion method for a distributed storage system provided by this invention has the following advantages: By constructing partition keys for repair information and deletion information respectively, and dividing the two types of task information into orderly storage according to the erasure code group dimension and sub-partitions, the method relies on the scheduling management component to uniquely allocate erasure code groups to the corresponding repair and deletion work processes. This allows the work processes to accurately concatenate partition keys, retrieve dedicated tasks, and execute corresponding operations, reducing read and write congestion in the distributed storage partitions. Furthermore, it enables on-demand start and stop, orderly scheduling, and efficient coordination of repair and deletion tasks, making the entire task scheduling process more predictable and manageable. This effectively avoids problems such as excessive system memory pressure, decreased data reliability, and system instability caused by disordered organization and scheduling, ultimately improving the stability and scheduling efficiency of repair and deletion tasks under the Quorum intelligent read-write architecture. Attached Figure Description

[0016] To more clearly illustrate the technical features of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an ordered repair and deletion method for a distributed storage system provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an ordered repair and deletion device for a distributed storage system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0021] It should be noted that this invention is applied to a distributed storage system with a cross-domain online erasure coding Quorum intelligent read-write architecture. This system is deployed in three availability zones (AZs) and adopts an erasure coding (EC, K+M) redundancy strategy to achieve low-cost, low-tail latency storage services. The system uses an erasure coding poolset as the smallest unit of data management, splits the original business data into K data fragments and generates M verification fragments. The Quorum read-write mechanism enables efficient business read and write operations. At the same time, it relies on the collaboration of multiple components to complete data repair, deletion, and scheduling management, ensuring the integrity of system data and operational stability.

[0022] The core implementing entities and related relationships of this plan are as follows: The scheduling management component (SchedulerMgr) serves as the main control unit for task scheduling, responsible for obtaining the full list of erasure coding groups within the cluster, allocating tasks at the erasure coding group granularity, and monitoring the workload of worker processes. The repair worker process (MissingSliceRepair) and the deletion worker process (SegmentDeleter) are deployed on the SliceNode single-machine storage engine. They are stateless worker processes responsible for pulling tasks and performing the corresponding data repair and deletion operations. Distributed key-value storage serves as a carrier of task information, used to store repair information, deletion information, and related partition keys; The above-mentioned entities work together to achieve the orderly organization, scheduling and execution of repair and deletion tasks, adapting to the high reliability and high availability requirements of distributed storage systems.

[0023] In a first aspect, embodiments of the present invention provide an ordered repair and deletion method for a distributed storage system, see [link to relevant documentation]. Figure 1 This is a flowchart illustrating an embodiment of an ordered repair and deletion method for a distributed storage system provided by the present invention.

[0024] like Figure 1As shown, the method includes the following steps: S1: For the repair information generated by the repair producer, a repair partition key is constructed by combining the number of missing fragments, the erasure code group identifier and the distributed storage sub-partition number. The repair information is then stored in the distributed storage in an orderly manner based on the repair partition key. Specifically, the format for repairing a partition key is num_missing.poolset.partition_num, and the specific definitions and configurations of each component of the repaired partition key are as follows: 1. Number of missing fragments (num_missing): This represents the number of fragments lost by a segment object in its corresponding erasure coding group (poolset). Its value range is related to the system erasure coding EC (K+M) configuration. The maximum number of missing fragments corresponding to an erasure coding group is K+M (i.e., the total number of fragments in the erasure coding group). Here, segment is the smallest logical data segment after the business data is split in the distributed storage system.

[0025] 2. Erasure code group identifier (poolset): This is a unique identifier for the erasure code group in the entire distributed storage cluster. It is used to uniquely locate the erasure code group to which the patch information belongs, ensuring that all patch information in the same erasure code group is categorized and stored based on this identifier, providing an accurate basis for subsequent work processes to pull tasks by erasure code group.

[0026] 3. Distributed storage sub-partition number (partition_num): This is the sub-partition identifier for a single erasure code group in the distributed KV storage. It is used to distribute the repair information corresponding to a single erasure code group to different sub-partitions of the distributed KV, so as to avoid read and write performance bottlenecks caused by a single KV partition carrying too many key-value pairs.

[0027] In this embodiment, the producers of repair information include two types, each corresponding to different repair information generation scenarios, specifically: Business Gateway Accesser: When the business gateway receives and processes the write data request from the business side, according to the Quorum intelligent read and write mechanism, if the overall business write operation is judged to be successful, but some erasure coding group members experience write timeouts or write failures, resulting in missing fragments of the corresponding segment object, the business gateway will automatically generate repair information, recording core information such as the erasure coding group to which the object belongs, the number of missing fragments, and the missing fragment number. Scour, the scanning and comparison component, scans and compares the shard information of all erasure coding groups in the distributed storage cluster at a preset period. By verifying the shard integrity of each object, if a shard is found to be missing in an object, repair information is automatically generated and the relevant core information of the object is recorded synchronously to ensure that all shard missing situations can be detected in a timely manner and the repair process is triggered.

[0028] The repair information generated by the two types of producers mentioned above is stored according to the repair partition key in the format of num_missing.poolset.partition_num. This ensures that the repair information is organized in an orderly manner in the distributed key-value store according to the priority of the number of missing fragments, the dimension of erasure code groups, and the division of sub-partitions. This lays the foundation for the accurate retrieval and sequential execution of repair tasks in the subsequent repair process.

[0029] S2: For deletion information generated by the deletion producer, a deletion partition key is constructed by combining the erasure code group identifier and the distributed storage sub-partition number, and the deletion information is stored in the distributed storage in an orderly manner based on the deletion partition key; Specifically, the format for deleting a partition key is poolset.partition_num, where the definitions of poolset and partition_num are the same as in step S1.

[0030] In this implementation, the producers of deleted information include two types, each corresponding to different deleted information generation scenarios, ensuring timely cleanup of junk data within the distributed storage system and accurate response to business deletion requests. Specifically: Business Gateway Accesser: When the business gateway receives and processes write data requests from the business side, according to the Quorum intelligent read and write mechanism, if most members of the erasure coding group experience write timeouts or write failures during the business write operation, causing the business write operation to fail to meet the Quorum read and write requirements and be judged as an overall failure, in order to prevent the unsuccessfully written fragments from becoming garbage data remaining in the system, the Accesser automatically generates deletion information, records the erasure coding group to which the segment object corresponding to this write operation belongs, the fragment information to be deleted, and other core content, triggering the garbage data cleanup process; On the business side: When the business side has a data deletion requirement, it initiates an active deletion operation to the distributed storage system. After receiving the deletion instruction, the system generates deletion information by the corresponding component, which clarifies the erasure code group to which the segment object to be deleted belongs, the object identifier, and the relevant fragment information, so as to ensure that the target data corresponding to the business requirement can be deleted accurately.

[0031] The deletion information generated by the two types of producers mentioned above is stored in an orderly manner according to the deletion partition key in the aforementioned poolset.partition_num format. This achieves a standardized organization of deletion information by erasure code group dimension and sub-partition division, which not only ensures the orderliness and efficiency of deletion information storage, but also accurately adapts to the erasure code group allocation strategy of the scheduling management component and the task retrieval logic of the deletion process. This lays a solid foundation for the subsequent deletion process to concatenate the deletion partition key, retrieve tasks, and execute deletion operations.

[0032] S3: Obtain the erasure coding group list in the distributed storage system, and uniquely assign each erasure coding group in the list to the corresponding repair worker process and deletion worker process. The repair worker process configures the repair partition key based on the assigned erasure coding group identifier, and pulls the corresponding repair information from the distributed storage based on the repair partition key to perform data repair operation. The deletion worker process configures the deletion partition key based on the assigned erasure coding group identifier, and pulls the corresponding deletion information from the distributed storage based on the deletion partition key to perform data deletion operation.

[0033] Specifically, the scheduling and management component first obtains a full list of all erasure coding groups in the distributed storage system. This list contains the unique identifiers and related configuration information of all erasure coding groups in the cluster. According to the granularity of the erasure coding group, each erasure coding group in the list is uniquely assigned to the corresponding repair worker process (MissingSliceRepair) and deletion worker process (SegmentDeleter). That is, one erasure coding group corresponds to only one repair worker process and one deletion worker process, ensuring that the repair task and deletion task of the same erasure coding group are handled by dedicated worker processes, avoiding task conflicts and disk I / O congestion caused by different worker processes operating the same erasure coding group at the same time.

[0034] After allocation, the repair and deletion processes, based on their assigned erasure coding group identifiers, configure the corresponding partition keys and retrieve tasks to perform relevant operations. The specific process is as follows: As a stateless worker process deployed in the SliceNode single-machine storage engine, after obtaining the erasure coding group identifier assigned by the scheduling management component, it automatically concatenates the system's preset EC (K+M) configuration and the corresponding sub-partition number (partition_num) configuration to form a patch partition key in the format num_missing.poolset.partition_num. After concatenation, the repair worker process accurately pulls the patch information of the corresponding erasure coding group from the distributed KV storage based on the patch partition key. After retrieval, it performs data patching operations on the disk corresponding to the single-machine storage engine. Specifically, according to the erasure coding EC (K+M) rules, it reads K normal data fragments in the erasure coding group, calculates the missing fragment using the erasure coding algorithm, and writes the calculated fragment to the corresponding storage node to complete the repair of a single missing fragment. This process is repeated until all missing fragments of the target object are repaired.

[0035] For the deletion worker process, which is also a stateless worker process deployed in a single-machine storage engine, after obtaining the erasure coding group identifier assigned by the scheduling management component, it automatically concatenates the corresponding sub-partition number (partition_num) configuration of the erasure coding group to form a deletion partition key in the format of poolset.partition_num. After concatenation, the deletion worker process pulls the deletion information of the corresponding erasure coding group from the distributed KV storage based on the deletion partition key. After retrieval, it performs data deletion operation on the disk corresponding to the single-machine storage engine. Specifically, according to the erasure coding EC(K+M) rule, it performs a full deletion operation on all K data fragments and M parity fragments of the target object in the erasure coding group to ensure that the target data and its parity information are thoroughly cleaned up and to avoid residual garbage data.

[0036] In summary, this invention constructs partition keys for repair and deletion information respectively, and achieves orderly storage of two types of task information by dividing them according to erasure coding groups and sub-partitions. At the same time, relying on the scheduling management component, erasure coding groups are uniquely assigned to the corresponding repair and deletion work processes, enabling the work processes to accurately concatenate partition keys, retrieve dedicated tasks, and execute corresponding operations. This reduces read and write congestion in distributed storage partitions and enables on-demand start and stop, orderly scheduling, and efficient coordination of repair and deletion tasks. This makes the entire task scheduling process more predictable and manageable, effectively avoiding problems such as excessive system memory pressure, decreased data reliability, and system instability caused by disordered organization and scheduling. Ultimately, it improves the stability and scheduling efficiency of repair and deletion tasks under the Quorum intelligent read-write architecture.

[0037] In one optional implementation, the sub-partition number is a sub-partition identifier assigned to a single erasure coding group in the distributed storage, and the value range of the sub-partition number is determined by the ratio of the maximum number of objects carried by a single erasure coding group to the maximum number of key-value pairs carried by a single distributed storage partition.

[0038] Specifically, the range of sub-partition numbers is determined by the ratio of the maximum number of objects that a single erasure coding group can carry to the maximum number of key-value pairs that a single distributed KV partition can carry. That is, the number of sub-partition numbers for a single erasure coding group is equal to the maximum number of objects carried by that erasure coding group divided by the maximum number of key-value pairs carried by a single distributed KV partition. When inserting repair information or deletion information into the distributed KV storage, the corresponding KV partition can be selected according to the sub-partition number to complete the information writing in a hash mapping manner.

[0039] For example, if a single erasure coding group can hold a maximum of 100,000 objects and a single distributed KV partition can hold a maximum of 10,000 key-value pairs, then the number of sub-partition numbers of the erasure coding group is 100,000 divided by 10,000, which is 10.

[0040] Based on the sub-partition numbers, the partition key configuration and number of partitions for patching information in the distributed KV storage can be further determined. The number of patching information partitions required for a single erasure coding group is the product of the total number of missing fragments (num_missing_count), the number of erasure coding groups (poolset_count), and the number of sub-partition numbers (partition_num_count), using erasure coding (EC)... Taking the 10+4 configuration as an example, when a single erasure code group can hold a maximum of 100,000 objects and a single KV partition can hold a maximum of 10,000 key-value pairs, the number of repair information partitions is (10+4)×1×(100000 / 10000)=140. The total number of repair information partitions required by all erasure code groups in the distributed KV storage is the product of the number of repair information partitions in a single erasure code group and the total number of erasure code groups in the cluster. If there are 1 million erasure code groups in the cluster under the above configuration, the total number of repair information partitions in the distributed KV storage is 140×1000000=140000000, thus realizing the orderly partitioning and storage of massive repair and deletion information in the distributed KV storage.

[0041] Similarly, the partition key configuration and number of partitions for deleting information in the distributed KV storage can be further determined. The number of partitions for deleting information required by a single erasure coding group is the product of the number of erasure coding groups (poolset_count) and the number of sub-partition numbers (partition_num_count). For example, if a single data cluster contains a total of 1 million erasure coding groups, and the number of KV partitions required by a single erasure coding group is 10, then the total number of partitions for deleting information in the distributed KV storage is 10 × 1,000,000 = 10,000,000.

[0042] This embodiment can distribute the repair and deletion information of a single erasure code group to multiple KV sub-partitions, effectively avoiding the read and write performance bottleneck caused by a single KV partition carrying too many key-value pairs. Furthermore, through standardized partition number calculation rules, the partition planning of distributed KV storage for repair and deletion information has quantifiable and configurable characteristics, adapting to the needs of distributed storage systems with different erasure code configurations and different cluster sizes, further supporting the stability and reliability of the entire Quorum intelligent read and write architecture distributed storage system.

[0043] In one optional implementation, the step of retrieving corresponding repair information from distributed storage based on the repair partition key and performing data repair operation includes: The repair priority is determined according to the number of missing fragments in descending order; wherein, the more missing fragments there are, the higher the repair priority. According to the repair priority, the corresponding repair information is pulled from the distributed storage, and the data repair operation is performed.

[0044] Specifically, the repair process first determines the execution priority of data repair according to the order of the number of missing fragments in the repair partition key from largest to smallest. The number of missing fragments is the number of fragments lost by the object in the corresponding erasure code group. The more fragments lost, the higher the repair priority of the object.

[0045] The repair process is based on the above repair priority, and the repair information of the corresponding erasure code group is pulled from the distributed KV storage in sequence, and the data repair operation is performed.

[0046] This embodiment links the number of missing shards with the repair priority, allowing objects with a large number of missing shards to be repaired first. This effectively avoids the risk of data unrecoverability due to excessive shard loss and significantly improves the data reliability of the distributed storage system.

[0047] In an optional implementation, the method further includes: After the data repair operation is successfully executed, the corresponding repair partition key and repair information will be deleted from the distributed storage. After the data deletion operation is successfully executed, the corresponding deletion partition key and deletion information will be removed from the distributed storage.

[0048] Specifically, when all missing fragments of the target object in the erasure coding group are successfully repaired, the data repair operation is considered complete. At this point, the repair partition key corresponding to the object and the associated repair information are synchronously deleted from the distributed storage.

[0049] When all K+M fragments of the target object in the erasure coding group have completed the deletion operation, the data deletion operation is considered complete. At this time, the deletion partition key corresponding to the object and the associated deletion information in the distributed storage are synchronously deleted.

[0050] This embodiment effectively avoids redundant accumulation of invalid repair and deletion information in the distributed storage by synchronously cleaning up the corresponding partition keys and information in the distributed storage after the repair and deletion operations are successfully executed. This reduces the storage occupation of key-value pairs in the distributed KV storage and improves the utilization efficiency of storage resources.

[0051] In an optional implementation, the method further includes: Monitor the load of the repair and deletion processes in real time; When the load of any worker process exceeds a preset threshold, the erasure coding group corresponding to that worker process is reassigned to other worker processes with suitable loads.

[0052] Specifically, the scheduling management component SchedulerMgr monitors the runtime load of all repair and deletion processes deployed on single-machine storage engines within the cluster in real time. When the scheduling management component detects that the load index of any repair or deletion process exceeds a preset threshold, it immediately triggers the erasure coding group reassignment mechanism. All erasure coding groups handled by the overloaded process are redistributed to other normal working processes of the same type within the cluster that meet the preset load index, according to the load balancing principle. At the same time, the allocation information of the erasure coding groups is updated and the updated allocation information is synchronously persisted to the distributed storage of the cluster management component. This ensures that after the reassignment, each working process can pull and execute the tasks of the corresponding erasure coding group based on the latest allocation information, and the original overloaded working process will no longer process the tasks related to the reassigned erasure coding group.

[0053] This embodiment achieves load balancing scheduling of worker processes within the cluster by performing real-time load monitoring of the repair and deletion processes across all dimensions and triggering dynamic reallocation of erasure coding groups when a process is overloaded. This effectively avoids problems such as running lag and task execution delays caused by a single worker process carrying too many tasks, ensuring the execution efficiency of repair and deletion tasks. At the same time, the reallocation strategy based on the load balancing principle can fully utilize the processing power of all worker processes within the cluster, improving the resource utilization of the entire distributed storage system.

[0054] In an optional implementation, the method further includes: The allocation information of erasure coding groups is persistently stored in distributed storage, and the consistency mechanism of distributed storage ensures the data consistency of the allocation information in the cluster.

[0055] Specifically, after the scheduling and management component completes the allocation of erasure coding groups to the repair and deletion worker processes, it persistently stores the allocation information of all erasure coding groups in the distributed storage of the cluster management component. The allocation information includes, but is not limited to, the unique identifier of the erasure coding group, the corresponding identifier of the repair worker process, the corresponding identifier of the deletion worker process, the task allocation time, the task execution status, and the allocation effective time. At the same time, relying on the consistency mechanism of the distributed storage itself (such as the Raft protocol), the persistently stored allocation information is checked and synchronized to ensure that the allocation information is consistent across all nodes in the entire distributed storage cluster, avoiding the loss, tampering, or inconsistency of allocation information between nodes due to cluster node failures, network fluctuations, or other abnormal situations.

[0056] This embodiment persists the erasure coding group allocation information and relies on the consistency mechanism of distributed storage to ensure the consistency of information within the cluster. This effectively avoids task scheduling chaos caused by loss or tampering of allocation information or inconsistency between nodes, and ensures the accuracy of the scheduling management component's control over the work process and erasure coding groups.

[0057] Secondly, embodiments of the present invention provide an ordered repair and deletion device for a distributed storage system, see [link to relevant documentation]. Figure 2 This is a schematic diagram of an embodiment of an ordered repair and deletion device for a distributed storage system provided by the present invention.

[0058] like Figure 2 As shown, the device includes: The repair information organization module 21 is used to construct a repair partition key for the repair information generated by the repair producer by using a combination of the number of missing fragments, the erasure code group identifier and the distributed storage sub-partition number, and to store the repair information in an orderly manner in the distributed storage based on the repair partition key; The deletion information organization module 22 is used to construct a deletion partition key for deletion information generated by the deletion producer by using a combination of erasure code group identifier and distributed storage sub-partition number, and to store the deletion information in the distributed storage in an orderly manner based on the deletion partition key; The ordered scheduling execution module 23 is used to obtain a list of erasure coding groups in the distributed storage system, and uniquely assign each erasure coding group in the list to a corresponding repair worker process and a deletion worker process. The repair worker process configures a patch partition key based on the assigned erasure coding group identifier, and pulls the corresponding patch information from the distributed storage based on the patch partition key to perform data patching operation. The deletion worker process configures a deletion partition key based on the assigned erasure coding group identifier, and pulls the corresponding deletion information from the distributed storage based on the deletion partition key to perform data deletion operation.

[0059] In one optional implementation, the sub-partition number is a sub-partition identifier assigned to a single erasure coding group in the distributed storage, and the value range of the sub-partition number is determined by the ratio of the maximum number of objects carried by a single erasure coding group to the maximum number of key-value pairs carried by a single distributed storage partition.

[0060] In one optional implementation, the step of retrieving corresponding repair information from distributed storage based on the repair partition key and performing data repair operation includes: The repair priority is determined according to the number of missing fragments in descending order; wherein, the more missing fragments there are, the higher the repair priority. According to the repair priority, the corresponding repair information is pulled from the distributed storage, and the data repair operation is performed.

[0061] In an optional embodiment, the device is further configured to: After the data repair operation is successfully executed, the corresponding repair partition key and repair information will be deleted from the distributed storage. After the data deletion operation is successfully executed, the corresponding deletion partition key and deletion information will be removed from the distributed storage.

[0062] In an optional embodiment, the device is further configured to: Monitor the load of the repair and deletion processes in real time; When the load of any worker process exceeds a preset threshold, the erasure coding group corresponding to that worker process is reassigned to other worker processes with suitable loads.

[0063] In an optional embodiment, the device is further configured to: The allocation information of erasure coding groups is persistently stored in distributed storage, and the consistency mechanism of distributed storage ensures the data consistency of the allocation information in the cluster.

[0064] It should be noted that the ordered repair and deletion device for a distributed storage system provided in this embodiment of the invention is used to execute all the process steps of the ordered repair and deletion method for a distributed storage system in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0065] Thirdly, embodiments of the present invention provide an electronic device, see [link to previous document]. Figure 3 The diagram shown is a structural schematic of an electronic device provided in an embodiment of the present invention.

[0066] like Figure 3 As shown, the device includes: Memory 31 is used to store computer programs; Processor 32 is used to execute the computer program; When the processor 32 executes the computer program, it implements the orderly repair and deletion method of the distributed storage system as described in any of the above embodiments.

[0067] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0068] The processor 32 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0069] The memory 31 can be used to store the computer programs and / or modules. The processor 32 implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory 31 and calling the data stored in the memory 31. The memory 31 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 31 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0070] It should be noted that the aforementioned electronic devices include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 3 The structural diagram is merely an example of the electronic device described above and does not constitute a limitation on the electronic device. It may include more components than shown in the diagram, or combine certain components, or use different components.

[0071] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed, implements the ordered repair and deletion method of the distributed storage system described in any of the above embodiments.

[0072] It should be understood that the implementation of all or part of the processes in the above-described orderly repair and deletion method for a distributed storage system can also be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the above-described orderly repair and deletion method for a distributed storage system. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0073] Fifthly, embodiments of this application also provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the ordered repair and deletion method of the distributed storage system described in any of the above embodiments.

[0074] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. It should be noted that, for those skilled in the art, several equivalent obvious modifications and / or equivalent substitutions can be made without departing from the technical principles of the present invention, and these obvious modifications and / or equivalent substitutions should also be considered within the scope of protection of the present invention.

Claims

1. An ordered repair and deletion method for a distributed storage system, characterized in that, include: For the repair information generated by the repair producer, a repair partition key is constructed by combining the number of missing fragments, the erasure code group identifier, and the distributed storage sub-partition number. The repair information is then stored in the distributed storage in an orderly manner based on the repair partition key. For deletion information generated by the deletion producer, a deletion partition key is constructed by combining erasure code group identifier and distributed storage sub-partition number, and the deletion information is stored in the distributed storage in an orderly manner based on the deletion partition key; Obtain a list of erasure coding groups in the distributed storage system, and uniquely assign each erasure coding group in the list to a corresponding repair worker process and a deletion worker process. The repair worker process configures a patch partition key based on the assigned erasure coding group identifier, and retrieves the corresponding patch information from the distributed storage based on the patch partition key to perform a data patching operation. Similarly, the deletion worker process configures a deletion partition key based on the assigned erasure coding group identifier, and retrieves the corresponding deletion information from the distributed storage based on the deletion partition key to perform a data deletion operation.

2. The ordered repair and deletion method for a distributed storage system as described in claim 1, characterized in that, The sub-partition number is the identifier of a sub-partition for a single erasure coding group in the distributed storage. The value range of the sub-partition number is determined by the ratio of the maximum number of objects carried by a single erasure coding group to the maximum number of key-value pairs carried by a single distributed storage partition.

3. The ordered repair and deletion method for a distributed storage system as described in claim 1, characterized in that, The step of retrieving corresponding repair information from distributed storage based on the repair partition key and performing data repair operations includes: The repair priority is determined according to the number of missing fragments in descending order; wherein, the more missing fragments there are, the higher the repair priority. According to the repair priority, the corresponding repair information is pulled from the distributed storage, and the data repair operation is performed.

4. The ordered repair and deletion method for a distributed storage system as described in claim 1, characterized in that, The method further includes: After the data repair operation is successfully executed, the corresponding repair partition key and repair information will be deleted from the distributed storage. After the data deletion operation is successfully executed, the corresponding deletion partition key and deletion information will be removed from the distributed storage.

5. The ordered repair and deletion method for a distributed storage system as described in claim 1, characterized in that, The method further includes: Monitor the load of the repair and deletion processes in real time; When the load of any worker process exceeds a preset threshold, the erasure coding group corresponding to that worker process is reassigned to other worker processes with suitable loads.

6. The ordered repair and deletion method for a distributed storage system as described in claim 1, characterized in that, The method further includes: The allocation information of erasure coding groups is persistently stored in distributed storage, and the consistency mechanism of distributed storage ensures the data consistency of the allocation information in the cluster.

7. An ordered repair and deletion device for a distributed storage system, characterized in that, include: The patch information organization module is used to construct a patch partition key for the patch information generated by the patch producer, using a combination of the number of missing fragments, erasure code group identifier and distributed storage sub-partition number, and to store the patch information in an orderly manner in the distributed storage based on the patch partition key; The deletion information organization module is used to construct a deletion partition key for deletion information generated by the deletion producer by using a combination of erasure code group identifier and distributed storage sub-partition number, and to store the deletion information in the distributed storage in an orderly manner based on the deletion partition key; An ordered scheduling execution module is used to obtain a list of erasure coding groups in the distributed storage system, and uniquely assign each erasure coding group in the list to a corresponding repair worker process and a deletion worker process. The repair worker process configures a patch partition key based on the assigned erasure coding group identifier, and retrieves the corresponding patch information from the distributed storage based on the patch partition key to perform data patching operations. Similarly, the deletion worker process configures a deletion partition key based on the assigned erasure coding group identifier, and retrieves the corresponding deletion information from the distributed storage based on the deletion partition key to perform data deletion operations.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program; Wherein, when the processor executes the computer program, it implements the orderly repair and deletion method of the distributed storage system as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the ordered repair and deletion method for a distributed storage system as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the ordered patching and deletion method for a distributed storage system as described in any one of claims 1 to 6.

Images