A method, device, electronic device and storage medium for volume-first reconfiguration

Abstract

The application discloses a method for volume priority reconstruction, comprising: creating at least two priority containers and associating storage volumes to direct data storage; scanning gc objects and associating storage objects by GC to record garbage amount and garbage bitmap of the storage objects for triggering garbage collection of the storage objects and identifying valid data; calculating target disks of the storage objects and their states by a layout algorithm and marking objects with target disk failure as to-be-reconstructed objects; periodically iterating the gc objects by a GC producer to add the to-be-reconstructed objects to a failure hash bucket and increase reference counts; extracting the to-be-reconstructed objects by a GC consumer to move valid data of the to-be-reconstructed objects to new storage objects with healthy target disks through an aggregation module; reducing counts and removing the objects after the moving is completed; and reporting reconstruction progress based on hash bucket count states of each node. The application realizes specified data priority reconstruction and guarantees important business data priority recovery safety through priority container association and GC directed moving.

Description

Technical Field

[0001] This application relates to the field of hard disk reconstruction technology, and in particular to a method, apparatus, electronic device and storage medium for volume-first reconstruction. Background Technology

[0002] To improve hard drive write performance, the industry commonly uses Redirect-On-Write (ROW) technology. This technology redirects and aggregates data to be written into large I / O operations before sequentially writing them to the hard drive pool objects, thus optimizing write efficiency. In distributed storage scenarios, append-only write technology can further improve data write performance, so ROW adopts this approach. However, appending to the same data block can generate redundant garbage data. To address this, the industry typically uses garbage collection (GC) algorithms to reclaim garbage space. Since ROW often uses large objects (e.g., a single object of 128MB), the conventional approach is to scan such objects and perform GC relocation on objects whose garbage volume reaches a set threshold—that is, read valid data from the object's old address and write it to the new object to complete the deletion of the old object, thereby releasing the space it occupies.

[0003] In distributed storage clusters, when a storage node (or hard drive) fails, the data on the failed node (hard drive) needs to be reconstructed (recovered). If the number of nodes (hard drives) involved in the reconstruction is large, or the amount of data to be reconstructed is large, especially in continuous foreground business scenarios, to avoid excessive consumption of CPU, network, and hard drive resources during the reconstruction process and impacting business operations, reconstruction bandwidth and IOPS (input / output per second) are typically limited, resulting in restricted reconstruction speed. Furthermore, existing reconstruction methods are generally carried out at the storage pool level, and the cluster global view information is generated based on the pool granularity to create a poolmap, making it difficult to prioritize the reconstruction of specific data. However, user data and business operations differ in importance, and different types of data and business operations have varying timeliness and urgency requirements for reconstructing failed data. Based on this requirement, in ROW scenarios, a method supporting priority reconstruction of storage volumes is urgently needed. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this application proposes a novel volume-first reconstruction method, which aims to meet the different timeliness and urgency requirements of different types of data and business for reconstructing faulty data.

[0005] Specifically, this application provides the following technical solutions:

[0006] A first aspect of this application provides a volume-first reconstruction method, the method comprising:

[0007] S1. Create at least two containers with different priorities in the Redirect Write (ROW) storage pool, and establish an association between the storage volume and the containers, so that data written to the storage volume is directed to the associated container;

[0008] S2. Scan the GC objects in each container through the garbage collection (GC) mechanism and extract their associated storage objects, wherein the GC objects record the amount of garbage and the garbage bitmap of the storage objects;

[0009] S3. Identify whether there is a faulty disk in the target disk of the storage object using a layout algorithm (such as jump). If so, mark the storage object as an object to be reconstructed.

[0010] S4. The GC producer periodically iterates through the GC objects on each node. When an object to be reconstructed is detected, it is added to the faulty object hash bucket and the reference count is increased.

[0011] S5. The GC consumer extracts the objects to be reconstructed from the hash bucket of the faulty objects, and moves their valid data to new storage objects through the aggregation module. The target disks of the new storage objects are all healthy disks. After all the valid data of the objects to be reconstructed has been moved, their reference count is reduced and they are removed from the hash bucket.

[0012] S6. Based on the reference count status of the hash buckets of faulty objects on each node, report the reconstruction progress of the storage volume to the user.

[0013] Furthermore, in the method of this application, the container in step S1 includes a normal container and a priority container; the normal volume is associated with the normal container, and the high-priority volume is associated with the priority container.

[0014] Furthermore, in the method of this application, the creation of at least two containers with different priorities in the redirected write (ROW) storage pool in step S1 includes:

[0015] The regular container and priority container are created simultaneously when the ROW pool is created, or priority containers are dynamically added during the operation of the ROW pool.

[0016] Furthermore, the method of this application also includes:

[0017] When data aggregation is written to the storage object, the gc object establishes its association with the storage object;

[0018] The jump layout algorithm is used to calculate the fault status of the target disk of the storage object in real time.

[0019] Furthermore, in the method of this application, step S5 includes:

[0020] S51. Extract the objects to be reconstructed from the hash bucket of the faulty objects using a GC consumer;

[0021] S52. Parse the garbage bitmap of the object to be reconstructed and identify the uncovered valid data within it;

[0022] S53. Aggregate valid data into large I / O blocks using the aggregation module and request new storage objects;

[0023] S54. If the target disk of the new storage object has a faulty disk, then re-apply for the new storage object until all its target disks are healthy disks;

[0024] S55. Write the valid data to the new storage object.

[0025] Furthermore, in the method of this application, step S6 includes:

[0026] When the reference count of the hash bucket of a faulty node reaches zero, the node reports "reconstruction complete";

[0027] When all nodes report "reconstruction complete", the system returns the final "reconstruction complete" status to the user.

[0028] A second aspect of this application provides an apparatus for volume-first reconstruction, wherein the apparatus, when operating, implements the steps of the aforementioned volume-first reconstruction method, the apparatus comprising:

[0029] A container creation unit is used to create at least two containers with different priorities in a redirected write (ROW) storage pool and establish an association between the storage volume and the containers, so that data written to the storage volume is directed to the associated container.

[0030] The GC scanning unit is used to scan the GC objects in each container through the garbage collection mechanism and extract their associated storage objects. The GC objects record the amount of garbage and the garbage bitmap of the storage objects.

[0031] The fault identification unit is used to identify whether there is a faulty disk in the target disk of the storage object through a layout algorithm (such as jump). If there is a faulty disk, the storage object is marked as an object to be reconstructed.

[0032] The GC production unit is used to periodically iterate through the GC objects on each node through the GC producer. When an object to be reconstructed is detected, it is added to the faulty object hash bucket and the reference count is increased.

[0033] The GC consumer unit is used to extract objects to be reconstructed from the hash bucket of the faulty objects through the GC consumer, and move their valid data to a new storage object through the aggregation module. The target disks of the new storage objects are all healthy disks. After all the valid data of the objects to be reconstructed has been moved, their reference count is reduced and they are removed from the hash bucket.

[0034] The status reporting unit is used to report the reconstruction progress of the storage volume to the user based on the reference count status of the hash bucket of the faulty object on each node.

[0035] Furthermore, in the device of this application, the containers created by the container creation unit include ordinary containers and priority containers; ordinary volumes are associated with ordinary containers, and high-priority volumes are associated with priority containers;

[0036] When the GC consumer unit moves data, if the target disk of the new storage object has a faulty disk, the aggregation module is triggered to re-apply for storage objects until all target disks are healthy disks.

[0037] The status reporting unit reports "reconstruction complete" when the reference count of the hash bucket of the node failure object reaches zero, and after summarizing the status of all nodes, it feeds back the final reconstruction result to the user.

[0038] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0039] Memory: Used to store computer programs;

[0040] Processor: Used to execute the computer program to implement the steps of the aforementioned volume-first reconstruction method.

[0041] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned volume-first reconstruction method.

[0042] In summary, compared with existing technologies, the technical advantages of this invention are as follows: By establishing a priority partitioning mechanism for storage volumes, storage volumes of different priorities are associated with containers of corresponding priorities (such as normal containers and priority containers), enabling differentiated reconstruction management of specified data; based on the garbage collection (GC) mechanism, storage objects within containers are scanned to identify objects to be reconstructed associated with faulty target disks (tgt), and the effective data of faulty objects is preferentially moved to new storage objects in a healthy state on the target disk through the GC producer-consumer model, forming a closed-loop processing flow of "priority identification - fault location - targeted migration". Compared with traditional full reconstruction at the storage pool level, this solution breaks through the limitation of globally unified reconstruction, and can implement priority reconstruction for container data associated with high-priority volumes, accurately meeting the differentiated needs of different businesses for the timeliness and urgency of data reconstruction, ensuring the priority recovery of important business data, and simultaneously balancing reconstruction efficiency and overall system stability through distributed node independent processing and status aggregation reporting mechanisms.

[0043] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0044] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0045] Figure 1 The overall implementation flowchart of the method for prioritizing the reconstruction of this application volume is shown.

[0046] Figure 2 This is a diagram of the cluster topology involved in the embodiments of this application (this example is a 3-node cluster, with 4 disks under each node).

[0047] Figure 3 This is a diagram illustrating the relationship between the volume, pool, and container in this application.

[0048] Figure 4 The architecture diagrams of the various functional modules involved in the priority restructuring of the scheme volume in this application are shown below.

[0049] Figure 5 This is a diagram illustrating the implementation method of the priority reconstruction process for the proposed solution volume in this application.

[0050] Figure 6 This is a structural diagram of the device that is prioritized for reconstruction in this application.

[0051] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0053] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0054] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0055] Figure 1 The diagram shows the overall implementation flow of the volume-first reconstruction method provided in this application, including the following steps:

[0056] S1. Create at least two containers with different priorities in the Redirect Write (ROW) storage pool, and establish an association between the storage volume and the containers, so that data written to the storage volume is directed to the associated container;

[0057] S2. Scan the GC objects in each container through the garbage collection mechanism and extract their associated storage objects, wherein the GC objects record the amount of garbage and the garbage bitmap of the storage objects;

[0058] S3. Identify whether there is a faulty disk in the target disk of the storage object using a layout algorithm (such as jump). If so, mark the storage object as an object to be reconstructed.

[0059] S4. The GC producer periodically iterates through the GC objects on each node. When an object to be reconstructed is detected, it is added to the faulty object hash bucket and the reference count is increased.

[0060] S5. The GC consumer extracts the objects to be reconstructed from the hash bucket of the faulty objects, and moves their valid data to new storage objects through the aggregation module. The target disks of the new storage objects are all healthy disks. After all the valid data of the objects to be reconstructed has been moved, their reference count is reduced and they are removed from the hash bucket.

[0061] S6. Based on the reference count status of the hash buckets of faulty objects on each node, report the reconstruction progress of the storage volume to the user.

[0062] Example: A volume-first reconstruction method

[0063] To more clearly illustrate the technical solution of the present invention, the following will provide further explanation through specific scenario embodiments.

[0064] Assuming a 3-node cluster, with each node having 4 disks, the cluster topology is as follows: Figure 2 As shown.

[0065] The implementation process of this plan is as follows:

[0066] 1) Create high-priority containers: When creating the ROW pool, both normal and priority containers are created. The normal container is used to store ordinary user data, while the priority container is used to store high-priority data specified by the user.

[0067] 2) Creation and association of priority volumes: When deploying a cluster, when creating storage volumes, specify normal volumes and priority volumes. Normal volumes are associated with normal containers, and priority volumes are associated with priority containers. When writing data to storage volumes, the data falls in the associated containers.

[0068] 3) Data Aggregation: When user data is written, a storage object is requested in the associated container. The storage object uses a layout algorithm such as Jump to calculate the tgt disk to be written to and completes the data aggregation. Taking 3 replicas as an example, when a client's small IO (A1 B1 C1...) falls on node0, a storage object obj0 is requested on node0. obj0 uses a layout algorithm to calculate the actual disk to be written to as tgt0 / tgt4 / tgt8. With a preset aggregation granularity of 1M, when the user's data is sent in 4K small IOs, it needs to go through 256 aggregations to become a 1M large IO before being flushed to the actual disk, thereby improving the disk flushing efficiency.

[0069] 4) GC Scan: When data is aggregated by the aggregation module, the corresponding GC object is associated with the storage object when it is requested. For example, the storage object obj0 is associated with the GC object gc_obj0. The GC object is mainly used to record information such as the capacity of olog, the amount of garbage, and the garbage bitmap. This GC object can be scanned by the GC module. That is, the GC module scans the normal and priority containers to obtain the storage object data associated with the GC object of each container.

[0070] 5) Fault Identification Module: For storage objects in the priority container, the module calculates the target (disk) where the storage object data is written using layout algorithms such as Jump, and identifies the faulty target. For example, if tgt4 in obj0 is offline due to hard disk failure and the hard disk is unwritable, tgt4 will be marked as faulty because the 3-replica redundancy condition can no longer be met, meaning that the storage object is faulty and data recovery, i.e., reconstruction, is required.

[0071] 6) GC production iteration: The GC module under each node will periodically (the scanning period is determined by the number of objects, such as 10,000 stored objects, the iteration period is 30 seconds) iterate out the GC objects written on that node. For example, obj0 will be iterated every time the cycle is repeated.

[0072] 7) Reference count: A hash bucket is created for storing objects. When the GC producer first iterates to the storage object associated with the GC object, which is a faulty target object (such as obj0), and inserts it into the hash bucket, the reference count is incremented by 1. When this storage object is moved to a new storage object by the GC consumer, it is deleted from the hash bucket, and the reference count is decremented by 1.

[0073] 8) GC Consumer: The GC consumer continuously retrieves GC objects from the GC producer bucket. When it identifies a faulty storage object associated with it, the GC reclamation module moves the valid data in the storage object to a new, healthy storage object via the aggregation module. Specifically, if the target objects allocated by the aggregation module are all healthy according to the layout calculation, the data to be moved can be written; otherwise, the aggregation module re-allocates objects until a healthy storage object is found, and then writes the data. For example, if obj0 is a faulty storage object, valid data is determined through the bitmap garbage bitmap and reverse lookup information, and the valid data is written to a new target disk via a newly allocated storage object (e.g., obj00) from the aggregation module. If the actual target disks for obj00 are tgt1 / tgt6 / tgt9 calculated by the layout algorithm, and all disks are healthy, then the data can be written; otherwise, if any disk is unhealthy, a new storage object is allocated, and the operation is repeated.

[0074] 9) Status Query and Reporting: When the count of faulty objects in the GC hash bucket of a node decreases to the initial value, it means that all storage objects to be moved in the high-priority hash bucket have been moved, and the node can report "Reconstruction Completed". Otherwise, it reports "Reconstruction in Progress". Due to the distributed processing, each node processes independently. Only when all nodes report "Reconstruction Completed" can the final "Reconstruction Completed" status be considered. Otherwise, they are all considered to be in the "Reconstruction in Progress" status. For example, all faulty objects on node0 have been processed, and it reports "Reconstruction Completed". However, because the processing of faulty objects on node1 / node2 is slower, it has not yet been completed, and it reports "Reconstruction in Progress". After aggregation, the final status presented to the client is "Reconstruction in Progress". When node1 / node2 / node3 have all processed their respective storage objects, the final status presented to the client is "Reconstruction Completed".

[0075] Figure 3 The diagram shows the volume-pool-container relationship in this solution. As shown, in block operations, this solution establishes a volume-pool-container relationship, where IO data of different priorities in the same ROW pool are written to different priority containers through priority volumes.

[0076] Figure 4 The diagram shows the functional modules involved in the volume-priority refactoring of this solution, including:

[0077] (1) Volume priority configuration module: This module allows customers to configure the storage of data of different importance levels;

[0078] (2) Aggregation module: The core module in ROW technology used to aggregate discrete small I / Os into a large I / O, thereby improving disk flushing efficiency and I / O performance;

[0079] (3) Layout computing module: mainly used for the disk persistence method of distributed computing data to achieve redundant storage. For example, in the above cluster topology, 3-replica redundancy requires selecting 1 disk from each of the 3 nodes to achieve backup and secure storage of data, ensuring data availability in case of failure.

[0080] (4) Counting module: mainly used for counting faulty storage objects, driving the GC process to move faulty storage objects to new healthy storage objects, and realizing accurate reporting of reconstruction status;

[0081] (5) GC moving module: In ROW technology, because the data is written in an append-only manner, that is, the same piece of data is first written to a storage object obj1, and then written to a new storage object obj2. The data on obj1 will inevitably become overwritten garbage. The GC algorithm commonly used in the industry is to reclaim the garbage space on the old storage object obj1.

[0082] (6) Status query and reporting module: This module provides customers with a means to query whether the current high-quality volume has been reconstructed, thereby determining whether important data has been restored normally.

[0083] Figure 5 The diagram shows the implementation method of the volume-priority reconstruction process in this scheme, which is divided into GC producer process, GC consumer process, and reconstruction status query and reporting process.

[0084] 1. GC Producer Process: The GC producer scans the storage object and calculates the layout of its associated storage object tgt using an algorithm such as the jump layout. It then determines whether there is a fault in the tgt. If there is a fault in the tgt, the reference count of the faulty object is increased and it is then put into the GC producer's hash bucket.

[0085] 2. GC Consumer Process: If the GC consumer determines that the reference count of the current faulty object is not 0, it considers that there is still a faulty storage object waiting to be GCd in the hash bucket. It continuously iterates through the hash bucket to find the faulty storage object, and unconditionally moves the data in it to a new healthy storage object through the aggregation write module. When all the valid data on the faulty object has been moved, it is considered that the faulty object has been processed and the reference count of the faulty object is decremented by 1.

[0086] 3. Reconstruction status query and reporting process: When the fault object count on a node is reduced to 0 (initial value), the node can report the "reconstruction completed" status; otherwise, it reports the "reconstruction in progress" status.

[0087] Figure 6 The diagram shows a volume-first reconstruction apparatus according to this application, the apparatus comprising:

[0088] A container creation unit is used to create at least two containers with different priorities in a redirected write (ROW) storage pool and establish an association between the storage volume and the containers, so that data written to the storage volume is directed to the associated container.

[0089] The GC scanning unit is used to scan the GC objects in each container through the garbage collection mechanism and extract their associated storage objects. The GC objects record the amount of garbage and the garbage bitmap of the storage objects.

[0090] The fault identification unit is used to identify whether there is a faulty disk in the target disk of the storage object through a layout algorithm (such as jump). If there is a faulty disk, the storage object is marked as an object to be reconstructed.

[0091] The GC production unit is used to periodically iterate through the GC objects on each node through the GC producer. When an object to be reconstructed is detected, it is added to the faulty object hash bucket and the reference count is increased.

[0092] The GC consumer unit is used to extract objects to be reconstructed from the hash bucket of the faulty objects through the GC consumer, and move their valid data to a new storage object through the aggregation module. The target disks of the new storage objects are all healthy disks. After all the valid data of the objects to be reconstructed has been moved, their reference count is reduced and they are removed from the hash bucket.

[0093] The status reporting unit is used to report the reconstruction progress of the storage volume to the user based on the reference count status of the hash bucket of the faulty object on each node.

[0094] The above-mentioned device implements the steps of the volume-first reconstruction method disclosed in this application when it is in operation.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0096] like Figure 7 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the volume-first reconstruction method described above.

[0097] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0098] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0099] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0100] Conventional procedural programming languages, such as "C" or similar programming languages.

[0101] The execution methods of program code include, but are not limited to:

[0102] It runs entirely on the user's computer;

[0103] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0104] Execute as a standalone software package;

[0105] It is executed entirely on a remote computer or server.

[0106] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0107] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the volume-first reconstruction method disclosed in this application.

[0108] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0109] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0110] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0111] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0112] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0113] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0114] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0115] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0116] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0117] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0118] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0119] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0120] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. The computer program contains program code for performing the volume-first refactoring method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0121] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0122] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A volume-first reconstruction method, characterized in that, The method includes: S1. Create at least two containers with different priorities in the redirected write storage pool, and establish an association between the storage volume and the containers, so that data written to the storage volume is redirected to the associated container; S2. Scan the GC objects in each container through the garbage collection mechanism and extract their associated storage objects, wherein the GC objects record the amount of garbage and the garbage bitmap of the storage objects; S3. Identify whether there is a faulty disk in the target disk of the storage object through a layout algorithm. If so, mark the storage object as an object to be reconstructed. S4. The GC producer periodically iterates through the GC objects on each node. When an object to be reconstructed is detected, it is added to the faulty object hash bucket and the reference count is increased. S5. The GC consumer extracts the objects to be reconstructed from the hash bucket of the faulty objects, and moves their valid data to new storage objects through the aggregation module. The target disks of the new storage objects are all healthy disks. After all the valid data of the objects to be reconstructed has been moved, their reference count is reduced and they are removed from the hash bucket. S6. Based on the reference count status of the hash buckets of faulty objects on each node, report the reconstruction progress of the storage volume to the user.

2. The method according to claim 1, characterized in that, The containers mentioned in step S1 include ordinary containers and priority containers; ordinary volumes are associated with ordinary containers, and high-priority volumes are associated with priority containers.

3. The method according to claim 2, characterized in that, Step S1, which describes creating at least two containers with different priorities in the redirected write storage pool, includes: The regular container and priority container are created simultaneously when the ROW pool is created, or priority containers are dynamically added during the operation of the ROW pool.

4. The method according to claim 1, characterized in that, The method also includes: When data aggregation is written to the storage object, the gc object establishes its association with the storage object; The jump layout algorithm is used to calculate the fault status of the target disk of the storage object in real time.

5. The method according to claim 1, characterized in that, Step S5 includes: S51. Extract the objects to be reconstructed from the hash bucket of the faulty objects using a GC consumer; S52. Parse the garbage bitmap of the object to be reconstructed and identify the uncovered valid data within it; S53. Aggregate valid data into large I / O blocks using the aggregation module and request new storage objects; S54. If the target disk of the new storage object has a faulty disk, then re-apply for the new storage object until all its target disks are healthy disks; S55. Write the valid data to the new storage object.

6. The method according to claim 1, characterized in that, Step S6 includes: When the reference count of the hash bucket of a faulty node reaches zero, the node reports "Reconstruction complete"; When all nodes report "reconstruction complete", the system returns the final "reconstruction complete" status to the user.

7. A device for volume-first reconfiguration, characterized in that, The apparatus, when operating, implements the volume-first reconstruction method as described in any one of claims 1-6, the apparatus comprising: A container creation unit is used to create at least two containers with different priorities in a redirected write storage pool, and to establish an association between the storage volume and the containers, so that data written to the storage volume is directed to the associated container. The GC scanning unit is used to scan the GC objects in each container through the garbage collection mechanism and extract their associated storage objects. The GC objects record the amount of garbage and the garbage bitmap of the storage objects. The fault identification unit is used to identify whether there is a faulty disk in the target disk of the storage object through a layout algorithm. If there is, the storage object is marked as an object to be reconstructed. The GC production unit is used to periodically iterate through the GC objects on each node through the GC producer. When an object to be reconstructed is detected, it is added to the faulty object hash bucket and the reference count is increased. The GC consumer unit is used to extract objects to be reconstructed from the hash bucket of the faulty objects through the GC consumer, and move their valid data to a new storage object through the aggregation module. The target disks of the new storage objects are all healthy disks. After all the valid data of the objects to be reconstructed has been moved, their reference count is reduced and they are removed from the hash bucket. The status reporting unit is used to report the reconstruction progress of the storage volume to the user based on the reference count status of the hash bucket of the faulty object on each node.

8. The apparatus according to claim 7, characterized in that, The containers created by the container creation unit include ordinary containers and priority containers; ordinary volumes are associated with ordinary containers, and priority volumes are associated with priority containers; When the GC consumer unit moves data, if the target disk of the new storage object has a faulty disk, the aggregation module is triggered to re-apply for storage objects until all target disks are healthy disks. The status reporting unit reports "reconstruction complete" when the reference count of the hash bucket of the node failure object reaches zero, and after summarizing the status of all nodes, it feeds back the final reconstruction result to the user.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the volume-first reconstruction method as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the volume-first reconstruction method as described in any one of claims 1-6.

Images