Data reconstruction method and electronic device

Abstract

The application discloses a data reconstruction method and electronic equipment, and relates to the technical field of data processing, and comprises the following steps: according to the last written internal object identifier in the main homing group and the to-be-recovered homing group, dividing the missing range and the residual range of the to-be-recovered homing group; performing residual object cleaning on the missing range; performing to-be-reconstructed object screening and garbage data filtering on the residual range, and performing data reconstruction on the effective objects obtained after filtering; when the missing range and the residual range are both empty, updating the last written internal object identifier in the to-be-recovered homing group to be consistent with the last written internal object identifier in the main homing group. In this way, the amount of reconstructed data and metadata overhead can be reduced without relying on modification logs and version numbers, the reconstruction and writing performance is improved, and the data consistency between homing groups and the system reliability are guaranteed.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data reconstruction method and an electronic device. Background Technology

[0002] Distributed storage systems often employ append-only write mechanisms to improve write performance, with failed data relying on background garbage collection for delayed release. Post-failure data reconstruction schemes largely depend on modification logs in placement groups (PGs), supporting only short-term incremental recovery. If a node remains unresponsive for an extended period, a full recovery is required. Traditional full recovery mechanisms reconstruct all objects, resulting in large data volumes, lengthy reconstruction times, and the system remaining in a low-redundancy state for extended periods, increasing the risk of data loss. Furthermore, this mechanism relies on modification logs and object version numbers, increasing metadata overhead during the write process and reducing system write performance and reliability. Summary of the Invention

[0003] This invention provides a data reconstruction method and electronic device that can reduce the amount of reconstructed data and metadata overhead, improve reconstruction and write performance, and ensure data consistency and system reliability between placement groups without relying on modification logs and version numbers.

[0004] This invention provides a data reconstruction method, comprising: Based on the internal object identifiers written at the end of the main placement group and the placement group to be restored, the missing range and residual range of the placement group to be restored are divided. Perform residual object cleanup on the missing range; The residual range is subjected to object screening and garbage data filtering, and data reconstruction is performed on the valid objects obtained after filtering. When both the missing range and the residual range are empty, the internal object identifier written at the end of the restoring group is updated to match the internal object identifier written at the end of the main restoring group.

[0005] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described data reconstruction methods when executing the computer program.

[0006] This invention divides the missing and residual ranges of the recovery placement group based on the internal object identifiers at the end of the main placement group and the placement group to be recovered, enabling precise location of the data area to be processed. Cleaning up residual objects in the missing range removes invalid and redundant data. Sequentially filtering objects to be reconstructed and filtering garbage data in the residual range eliminates invalid objects and retains valid data, significantly reducing the scale of data involved in reconstruction. Data reconstruction is performed on the filtered valid objects, reducing reconstruction overhead and improving reconstruction efficiency. Synchronously updating the internal object identifiers at the end of the placement group when both the missing and residual ranges are empty ensures eventual data consistency between the placement group and the main placement group. The entire solution does not rely on the modification logs and object modification version numbers of the placement groups, eliminating unnecessary metadata commit operations during normal write operations, effectively improving system write performance, and optimizing the overall operating efficiency and data reliability of the distributed storage system.

[0007] In addition, the present invention also provides a corresponding electronic device for the data reconstruction method, which has the same or corresponding technical features as the data reconstruction method mentioned above, and has the same effect. Attached Figure Description

[0008] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart of the data reconstruction method provided in the embodiments of the present invention; Figure 2 A detailed flowchart of the data reconstruction method provided in this embodiment of the invention; Figure 3 This is a schematic diagram illustrating the relationship between the metadata pool placement group and the data pool placement group provided in an embodiment of the present invention; Figure 4 A schematic diagram of the data reconstruction device provided in an embodiment of the present invention. Detailed Implementation

[0010] Distributed storage systems achieve high capacity, high availability, and elastic scalability by distributing data across multiple physical nodes. To improve write performance and avoid erasure penalties, storage systems widely employ append-only mechanisms. These mechanisms use log recording and batch aggregation to ensure that the data pool only adds new internal objects without modification write operations; invalid data (such as data deleted or overwritten by users) is released through background garbage collection. Distributed storage systems typically achieve data redundancy through replication or erasure coding mechanisms, but failures can reduce data redundancy, and subsequent failures may lead to data loss. Therefore, a data reconstruction mechanism is needed to restore redundancy: copying data from failed nodes / disks to healthy nodes. Traditional data reconstruction schemes rely on modification logs in placement groups, which can only support incremental recovery after a short period of recovery from a failed node. If a node fails for an extended period or cannot be recovered, the modification log records in the placement group are insufficient to cover all missing data. In such cases, a full recovery method is required, which involves scanning all objects on the primary object storage device (OSD) and the object storage device to be recovered, and completing a full data copy by scanning and comparing object version numbers. Because garbage collection is performed only after sufficient storage capacity has been available, a large number of garbage objects accumulate continuously in the data pool. Traditional full recovery mechanisms perform a unified reconstruction operation on all objects, resulting in a massive amount of reconstructed data and a significant increase in reconstruction time. During this process, the system remains in a low-redundancy state for an extended period, severely impacting the performance of upper-layer business operations and significantly increasing the risk of data loss due to secondary failures. Furthermore, this type of solution relies on modification logs and object version numbers, increasing the overhead of metadata commits during normal write processes, further restricting system write performance and overall reliability. To address these issues, this invention provides a data reconstruction method.

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0012] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] The specific application environment architecture or specific hardware architecture on which the execution of the data reconstruction method depends is described here.

[0015] The embodiments of the present invention provide a data reconstruction method, and the method is described in detail in conjunction with the execution flow of the data reconstruction method. Figure 1 A flowchart of the data reconstruction method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the method includes: S101. Based on the internal object identifiers written at the end of the main placement group and the placement group to be restored, divide the missing range and residual range of the placement group to be restored.

[0016] It should be noted that a placement group is a logical unit in a distributed storage system used to organize, manage, and locate data storage locations. The primary placement group refers to the placement group that serves as the data benchmark and provides complete and valid data during the placement group peering process. The placement group to be recovered refers to the placement group that requires data repair and synchronization during the placement group peering process. To accurately define the scope of the reconstruction object, this invention records the identifier of the most recently written internal object (last_write) in the placement group. This identifier is persisted to disk synchronously with each internal object write transaction, ensuring that data is not lost after a power outage. Combining the characteristics of the system's append-write mechanism—that internal objects are only created or deleted, with no modification operations, and that the internal object identifier monotonically increases—the reconstruction scope of the placement group to be recovered can be divided into two categories: missing range and residual range.

[0017] The missing range refers to the range of objects in the recovery placement group that need to be supplemented with data. The missing range extends from the last_write+1 (last write flag incremented by one) of the recovery placement group to the last_write (last write flag) of the main placement group. Objects within this range are new objects not received by the recovery placement group during the failure period and need to be supplemented through data reconstruction. The residual range refers to the range of objects in the recovery placement group that need to have invalid data cleaned up. The residual range extends from the minimum (MIN) object flag to the last_write of the recovery placement group. Objects within this range may have been deleted by garbage collection during the failure period and need to be cleaned up through scanning and comparison. Internal objects refer to storage units in a distributed storage system that aggregate and batch persist fragmented data written by multiple clients to the data pool when using an append-only write mechanism. An internal object flag is a unique identifier used to uniquely identify each internal object in the data pool and serves as the basis for dividing the placement group range and data reconstruction.

[0018] The execution entity of the data reconstruction method provided by this invention can be the control node where the main reset group is located.

[0019] In implementation, step S101 obtains the identifiers of the internal objects written at the end of the main placement group and the placement group to be restored. Using these identifiers as a basis, the data range is precisely divided, enabling rapid location of the missing and residual ranges that need to be processed in the placement group to be restored, thus achieving fine-grained initialization of the reconstruction range. Compared to the traditional method of scanning all objects in full, this method can complete the range definition based solely on the identifiers of the last objects, without traversing all internal objects. This effectively reduces the computational overhead and I / O operations during the range division process, laying the foundation for subsequent targeted cleaning of invalid data, filtering of valid objects, and execution of precise reconstruction. It avoids redundant data processing problems caused by full reconstruction from the source, improving the overall execution efficiency of the reconstruction process.

[0020] S102. Clean up residual objects in the missing range.

[0021] It should be noted that residual objects refer to invalid objects that exist in the pending restoration group but no longer exist in the main restoration group.

[0022] In implementation, step S102 cleans up residual objects within the missing range, specifically removing invalid objects that exist in the recovery placement group but no longer exist in the main placement group. This prevents redundant objects from participating in subsequent reconstruction processes, thereby reducing unnecessary data read / write and computational overhead. This operation, together with the range division step, forms a collaborative processing logic. First, it cleans up abnormally existing objects, then reconstructs the missing data, making the reconstruction process more hierarchical and targeted. This further reduces system resource consumption and provides a clean processing environment for subsequent effective data screening and reconstruction operations within the residual range, ensuring the accuracy and efficiency of data reconstruction.

[0023] S103. Filter the objects to be reconstructed and filter the garbage data in the residual range, and perform data reconstruction on the valid objects obtained after filtering.

[0024] It should be noted that the object to be reconstructed refers to a valid object that exists in the main placement group but is missing in the placement group to be restored, and which needs to be recovered.

[0025] In implementation, step S103, by screening objects to be reconstructed and filtering junk data within the residual range, can accurately distinguish between valid and invalid junk data. Only valid objects with recovery value are retained to participate in the subsequent reconstruction process, reducing the scale of data involved in reconstruction from the source and avoiding redundant read / write and resource waste caused by invalid data in traditional full reconstruction. Targeted data reconstruction based on valid objects reduces reconstruction overhead and shortens reconstruction time, while also preventing the system from being in a low-redundancy state for extended periods. This improves data reconstruction efficiency while effectively ensuring the reliability and business continuity of the distributed storage system.

[0026] S104. When both the missing range and the residual range are empty, update the internal object identifier written at the end of the recovery placement group to be consistent with the internal object identifier written at the end of the main placement group.

[0027] During implementation, when both the missing and residual ranges are empty, it indicates that the invalid object cleanup and valid data reconstruction of the restoring group have been completed. At this point, updating the internal object identifier written at the end of the restoring group to be consistent with the main restoring group logically synchronizes the data states of the two restoring groups, ensuring that they maintain unified alignment in the object identifier dimension. This operation not only marks the normal end of this data reconstruction process but also provides an accurate baseline state for subsequent restoring group equivalence and fault recovery, avoiding repeated reconstruction or range division errors caused by inconsistent identifiers, and ensuring the accuracy and continuity of subsequent data management and fault handling in the system.

[0028] In the data reconstruction method provided by this invention, the missing and residual ranges of the recovery placement group are divided based on the internal object identifiers at the end of the main placement group and the placement group to be recovered, which can accurately locate the data area to be processed; residual object cleanup is performed on the missing range to remove invalid and redundant data; the remaining range is sequentially screened for objects to be reconstructed and filtered for garbage data to remove invalid objects and retain valid data, significantly reducing the scale of data involved in reconstruction; data reconstruction is performed on the filtered valid objects to reduce reconstruction overhead and improve reconstruction execution efficiency; the internal object identifiers at the end of the placement group to be recovered are updated synchronously when both the missing and residual ranges are empty, which can ensure the eventual consistency of data between the placement group to be recovered and the main placement group. The entire solution does not rely on the modification logs and object modification version numbers of the placement group, which can eliminate redundant metadata commit operations during normal writing, effectively improve system write performance, and optimize the overall operating efficiency and data reliability of the distributed storage system.

[0029] It's important to note that in distributed storage systems employing append-only log mechanisms, the actual storage location of user data can be recorded using a logical-to-physical (L2P) address mapping. L2P is organized using a key-value pair structure. The key consists of the user object name and the logical offset (i.e., logical location) of the data, uniquely identifying the logical address of the user data. The value consists of the internal object name and the physical offset (i.e., actual location) of the data within that internal object, pointing to the data's physical storage address. The L2P mapping table and other metadata (such as internal object summaries) are managed through a separate metadata storage pool. This metadata storage pool is bound to the data pool storing internal objects to ensure consistency between metadata and business data. When user data is written, the client's write request is first submitted to the metadata storage pool. After the metadata storage pool records the data log, it immediately returns a write success response to the client to ensure the client's response speed. Then, the metadata storage pool requests a new internal object in the background, aggregates the random write data from multiple clients into batch data, and then flushes it to the data pool. After the data flushing is completed, the metadata storage pool updates the L2P and internal object summary and other metadata information.

[0030] Furthermore, in a specific implementation, in the data reconstruction method provided in the embodiments of the present invention, step S101 divides the missing range and residual range of the to-be-restored placement group according to the internal object identifiers written at the end of the main placement group and the placement group to be restored. Specifically, it may include: during the peering process of placement groups, reading the internal object identifiers written at the end of the main placement group and the placement group to be restored; and determining the missing range of data to be supplemented and the residual range of invalid data to be cleaned in the placement group to be restored based on the internal object identifiers, so as to complete the initialization of the reconstruction range.

[0031] Figure 2 This is a flowchart illustrating the data reconstruction method provided in an embodiment of the present invention. In implementation, as... Figure 2 As shown, during the peering process of the placement group, by reading and comparing the most recently written identifier of the main placement group with the most recently written identifier of the placement group to be restored, the missing range and the residual range can be quickly calculated and accurately divided. The reconstruction range initialization can be completed without traversing all objects, effectively reducing system IO and computational overhead, providing an accurate basis for subsequent targeted data supplementation and invalid object cleanup, and improving the efficiency and accuracy of the overall reconstruction process.

[0032] Furthermore, in a specific implementation, in the data reconstruction method provided in the embodiments of the present invention, step S102 cleans up residual objects in the missing range, which may specifically include: starting from the current starting position of the missing range, acquiring objects of the to-be-restored placement group in batches according to a preset processing quantity; comparing the batch-acquired objects of the to-be-restored placement group with the objects in the main placement group; if there is no corresponding object of the to-be-restored placement group in the main placement group, deleting the corresponding object of the to-be-restored placement group; after deletion, updating the starting position of the missing range to the current processing completed position.

[0033] In implementation, such as Figure 2 As shown, starting from the current starting position of the missing range, objects in the recovery placement group can be retrieved in batches according to a preset threshold (e.g., processing 100 objects at a time), and compared with objects in the main placement group. If no corresponding object exists in the main placement group, it indicates that the object has been garbage collected, and the residual object in the recovery placement group is deleted. After the object deletion is completed, the starting position of the missing range is updated to the current processing position. This method adopts a batch processing strategy, which can avoid system performance fluctuations caused by processing too much data at once. By accurately comparing and deleting invalid residual objects, redundant data occupation can be reduced. Furthermore, the processing progress is advanced in stages and the range boundary is dynamically updated to ensure that the cleanup process is orderly and controllable, thereby improving the stability and execution efficiency of missing range processing.

[0034] Furthermore, in a specific implementation, in the data reconstruction method provided in the embodiments of the present invention, step S103 of filtering the objects to be reconstructed in the residual range may specifically include: starting from the current starting position of the residual range, obtaining the objects of the main placement group in batches according to a preset processing quantity; and generating a list of objects to be reconstructed based on the objects of the main placement group obtained in batches.

[0035] In implementation, such as Figure 2 As shown, starting from the current starting position of the residual range, objects of the main placement group can be obtained in batches according to a preset threshold, and a list of objects to be reconstructed can be generated accordingly. This can control the data scale of a single process, avoid excessive system resource consumption due to excessive traversal, and effectively improve the execution efficiency and stability of the reconstructing process.

[0036] Furthermore, in a specific implementation, in the data reconstruction method provided in the embodiments of the present invention, step S103 filters garbage data in the residual range, which may specifically include: reading the effective data length of each object to be reconstructed in the list of objects to be reconstructed from the metadata storage pool; determining whether the effective data ratio corresponding to the effective data length is less than a preset threshold; if so, performing garbage collection processing on the objects to be reconstructed whose effective data ratio is less than the preset threshold; if not, retaining the objects to be reconstructed whose effective data ratio is not less than the preset threshold, and taking the retained objects to be reconstructed as valid objects.

[0037] In implementation, such as Figure 2 As shown, the effective data length of each object to be reconstructed in the list of objects to be reconstructed is read from the metadata storage pool. Based on a preset threshold (e.g., effective data percentage < 10%), objects with an effective data percentage below the threshold are garbage collected. Their effective data is aggregated into new objects, and the old objects are deleted. Objects with a high effective data percentage are retained, and a list of objects requiring actual reconstruction is generated. This precise identification of objects with low effective data percentages based on effective data length and subsequent garbage collection significantly reduces the total amount of data involved in data reconstruction, lowers read / write overhead and system resource consumption during the reconstruction process, thereby improving data reconstruction efficiency and system performance, while also enhancing the reliability and stability of the distributed storage system.

[0038] It's important to note that the metadata storage pool additionally stores internal object summary information. Its core field includes the "valid data length of the internal object," which is updated in real-time during data flushing operations. When user data references are switched from an old internal object to a new one due to data overwriting, garbage collection aggregation, or other operations, the metadata storage pool synchronously deducts the valid length of the transferred data from the old internal object, ensuring that the summary information accurately reflects the valid data ratio of the internal object in real time. The system can scan the internal object summary information in the background, filter objects with low valid data volume, aggregate their remaining valid data into a new object, flush it to the data pool, and then delete the old object to free up storage space. This mechanism can maintain the valid data status of internal objects in real time, accurately filter and reclaim objects with low valid data ratios, thereby reducing the amount of data to be reconstructed and improving reconstruction efficiency and system reliability.

[0039] Furthermore, in a specific implementation, in the data reconstruction method provided in the embodiments of the present invention, step S103 performs data reconstruction on the filtered valid objects, which may specifically include: when the data pool does not use erasure coding storage, reading the valid data of the valid objects; when the data pool uses erasure coding storage, reading a set number of fragment data in the valid objects and calculating the missing fragments of the valid objects; transmitting the valid data of the valid objects or the missing fragments of the valid objects to the recovery and placement group; after the transmission is completed, updating the starting position of the residual range to the currently processed position.

[0040] In practice, the valid data of the object to be reconstructed is read. If it is an erasure coding data pool, sufficient fragment data is read and missing fragments are calculated. The valid data or missing distribution is transmitted to the recovery and placement group. After the data reconstruction is completed, the starting position of the residual range is updated to the current processing position.

[0041] When a data pool experiences data loss in its placement group due to object storage device failure, a local list of missing objects can be obtained by comparing it with the authoritative object storage device. Then, using erasure coding or replication mechanisms, the missing data can be recovered from other healthy object storage device members. This method can accurately recover missing data and dynamically update the processing progress, ensuring the orderly progress of the reconstruction process. Simultaneously, combined with erasure coding or replication redundancy strategies, it guarantees the integrity and reliability of data recovery, effectively improving the efficiency of placement group failure recovery, shortening the time the system is in a low-redundancy state, and reducing the risk of data loss.

[0042] Furthermore, in a specific implementation, in the data reconstruction method provided in the embodiments of the present invention, when both the missing range and the residual range are empty in step S104, the internal object identifier written at the end of the placement group to be restored is updated to be consistent with the internal object identifier written at the end of the main placement group. Specifically, this may include: repeatedly performing operations such as cleaning up residual objects in the missing range, filtering objects to be reconstructed in the residual range and filtering garbage data, and reconstructing valid object data until both the missing range and the residual range are empty; after determining that both the missing range and the residual range are empty, the internal object identifier written at the end of the placement group to be restored is synchronously updated to the same value as the internal object identifier written at the end of the main placement group.

[0043] In implementation, such as Figure 2 As shown, the process iteratively executes the above-mentioned operations: cleaning up residual objects within the missing range, filtering objects to be reconstructed within the residual range and filtering garbage data, and reconstructing data for valid objects, until both the missing and residual ranges are empty, marking the completion of the data reconstruction process. Finally, the most recently written flag of the placement group to be restored is updated to a value consistent with that of the main placement group. This allows for gradual convergence of the range and dynamic control of the progress, ensuring that the cleaning of invalid objects and the restoration of valid data are fully executed. While guaranteeing the accuracy and reliability of the reconstruction results, it also ensures that the states of the two placement groups are completely aligned.

[0044] Furthermore, in a specific implementation, in the data reconstruction method provided in the embodiments of the present invention, before performing step S101 to divide the missing range and residual range of the group to be restored and placed, it may further include: creating internal objects and generating internal object identifiers in integer form; wherein, the internal object identifiers are generated according to a monotonically increasing rule, and the creation order of internal objects is determined based on the internal object identifiers.

[0045] In implementation, internal object names adopt a naming format combining a fixed prefix and a monotonically increasing object identifier. The object identifier can be an integer and is monotonically increasing, allowing direct identification of the creation order of internal objects through the identifier value. Write operations on internal objects are triggered by business logic writes or garbage collection processes to persistently store data in the data pool. Upon completion of the write, the corresponding metadata information is updated synchronously. This naming rule and write mechanism ensures a clear and identifiable object order, providing a reliable basis for scope division, batch processing, and progress updates during the relocation group peering process, further effectively improving the accuracy and execution efficiency of the refactoring process.

[0046] Furthermore, in a specific implementation, the data reconstruction method provided in the above-mentioned embodiments of the present invention may include, while performing data reconstruction on the filtered valid objects, the following: if a newly written internal object is generated, the newly written internal object is written into the recovery placement group, and no reconstruction processing of the missing range and residual range is performed on the newly written internal object.

[0047] During implementation, if new internal objects are written to the data pool during the reconstruction process, since the identifier of these new objects is greater than the most recently written identifier of the main placement group, this invention can directly write them to the placement group to be restored, without needing to perform reconstruction processing on the missing and residual ranges of these newly added objects. This approach avoids interference between the reconstruction process and newly written objects, ensuring that new data is written normally without including newly added objects in the reconstruction scope, effectively reducing reconstruction workload and system overhead, improving reconstruction execution efficiency, and ensuring the continuity of business writes.

[0048] Figure 3 This is a schematic diagram illustrating the association between the metadata pool placement group and the data pool placement group provided in an embodiment of the present invention. In implementation, as... Figure 3As shown, after a client initiates a business write request, the data first enters the metadata pool placement group. Data flushing aggregates scattered writes from multiple clients into internal objects and triggers garbage collection to clean up older objects with low effective data percentages. After aggregation, the internal objects are persisted to the data pool placement group via the internal object write process. The primary object storage device receives and synchronizes the data to the backup object storage device to ensure data redundancy. When the data pool placement group triggers a data reconstruction request due to device failure or data loss, the primary object storage device can act as the authoritative data source, providing missing object information to the data reconstruction operation. Relying on erasure coding or replication mechanisms, the missing data is recovered from the backup object storage device, completing data integrity verification and state synchronization. This layered and decoupled design of the metadata pool and data pool ensures low-latency response for client writes and optimizes storage utilization through internal object aggregation and garbage collection mechanisms. Meanwhile, the redundant architecture of the primary and backup object storage devices and the data reconstruction process work together to quickly recover data in the event of device failure, improving system reliability and business continuity. Furthermore, the design that newly added internal objects during reconstruction can be directly written to the recovery placement group further avoids mutual interference between reconstruction and new writes, reducing system overhead.

[0049] From the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0050] Embodiments of the present invention also provide a data reconstruction apparatus. Figure 4 This is a schematic diagram of the data reconstruction device provided in an embodiment of the present invention. This embodiment is based on functional modules, such as… Figure 4 As shown, the device includes: The range initialization module 10 is used to divide the missing range and residual range of the placement group to be restored based on the internal object identifiers written at the end of the main placement group and the placement group to be restored. The residual object cleanup module 11 is used to clean up residual objects in the missing range; The waste processing module 12 is used to screen the objects to be reconstructed and filter the waste data within the residual range, and to perform data reconstruction on the valid objects obtained after filtering. The object identifier update module 13 is used to update the internal object identifier written at the end of the recovery placement group to be consistent with the internal object identifier written at the end of the main placement group when both the missing range and the residual range are empty.

[0051] In the data reconstruction apparatus provided in this embodiment of the invention, the missing range and residual range of the recovery placement group can be divided based on the internal object identifiers at the end of the main placement group and the placement group to be recovered, enabling precise location of the data area to be processed; residual object cleanup of the missing range can eliminate invalid and redundant data; sequential screening of objects to be reconstructed and filtering of garbage data in the residual range can remove invalid objects and retain valid data, significantly reducing the scale of data involved in reconstruction; data reconstruction is performed on the filtered valid objects, reducing reconstruction overhead and improving reconstruction execution efficiency; when both the missing range and the residual range are empty, the internal object identifiers at the end of the placement group to be recovered are updated synchronously, ensuring the eventual consistency of data between the placement group to be recovered and the main placement group. The entire apparatus does not rely on the modification logs and object modification version numbers of the placement group, eliminating redundant metadata commit operations during normal writing, effectively improving system write performance, and optimizing the overall operating efficiency and data reliability of the distributed storage system.

[0052] Since the embodiments of the data reconstruction apparatus and the data reconstruction method correspond to each other, the descriptions of the features in the embodiments corresponding to the data reconstruction apparatus can be found in the relevant descriptions of the embodiments corresponding to the data reconstruction method, and will not be repeated here. Furthermore, it has the same beneficial effects as the data reconstruction method mentioned above.

[0053] Furthermore, in a specific implementation, in the data reconstruction device provided in the embodiments of the present invention, the range initialization module 10 can be used to read the internal object identifiers written at the end of the main placement group and the placement group to be restored during the placement group matching process; based on the internal object identifiers, determine the missing range of data to be supplemented and the residual range of invalid data to be cleaned in the placement group to be restored, so as to complete the initialization of the reconstruction range.

[0054] Furthermore, in a specific implementation, in the data reconstruction device provided in the embodiments of the present invention, the residual object cleaning module 11 can be specifically used to obtain objects of the to-be-restored placement group in batches according to a preset processing quantity, starting from the current starting position of the missing range; compare the batch-obtained objects of the to-be-restored placement group with the objects in the main placement group; if there is no corresponding object of the to-be-restored placement group in the main placement group, delete the corresponding object of the to-be-restored placement group; after deletion, update the starting position of the missing range to the current processing completed position.

[0055] Furthermore, in a specific implementation, in the data reconstruction apparatus provided in the embodiments of the present invention, the waste processing module 12 may specifically include: The list generation unit is used to retrieve objects from the main placement group in batches according to a preset processing quantity, starting from the current starting position of the residual range; and to generate a list of objects to be refactored based on the batch-retrieved objects from the main placement group. The garbage filtering unit is used to read the effective data length of each object to be reconstructed in the list of objects to be reconstructed from the metadata storage pool; determine whether the effective data ratio corresponding to the effective data length is less than a preset threshold; if so, garbage collection is performed on the objects to be reconstructed whose effective data ratio is less than the preset threshold; if not, the objects to be reconstructed whose effective data ratio is not less than the preset threshold are retained, and the retained objects to be reconstructed are regarded as effective objects. The data reconstruction unit is used to read the valid data of valid objects when the data pool does not use erasure coding storage; when the data pool uses erasure coding storage, it reads a set number of fragment data in the valid objects and calculates the missing fragments of the valid objects; it transmits the valid data of the valid objects or the missing fragments of the valid objects to the recovery and placement group; after the transmission is completed, it updates the starting position of the residual range to the position where the processing is completed.

[0056] Furthermore, in a specific implementation, in the data reconstruction device provided in the embodiments of the present invention, the object identifier update module 13 can be used to cyclically perform operations such as cleaning up residual objects in the missing range, filtering objects to be reconstructed in the residual range and filtering garbage data, and reconstructing valid object data until both the missing range and the residual range are empty; after determining that both the missing range and the residual range are empty, the internal object identifier written at the end of the placement group to be restored is synchronously updated to the same value as the internal object identifier written at the end of the main placement group.

[0057] Furthermore, in a specific implementation, the data reconstruction device provided in the embodiments of the present invention may further include: a data brushing module, used to create internal objects and generate internal object identifiers in integer form; wherein, the internal object identifiers are generated according to a monotonically increasing rule, and the creation order of internal objects is determined based on the internal object identifiers.

[0058] Furthermore, in a specific implementation, the data reconstruction device provided in the embodiments of the present invention may further include: an object writing module, which is used to write the newly written internal object into the recovery and placement group if a new internal object is generated, and not to perform reconstruction processing of the missing range and residual range on the newly written internal object.

[0059] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data reconstruction method embodiments.

[0060] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data reconstruction method embodiments at runtime.

[0061] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0062] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data reconstruction method embodiments.

[0063] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data reconstruction method embodiments.

[0064] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0065] The data reconstruction method and electronic device provided by this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A data reconstruction method, characterized in that, include: Based on the internal object identifiers written at the end of the main placement group and the placement group to be restored, the missing range and residual range of the placement group to be restored are divided. Perform residual object cleanup on the missing range; The residual range is subjected to object screening and garbage data filtering, and data reconstruction is performed on the valid objects obtained after filtering. When both the missing range and the residual range are empty, the internal object identifier written at the end of the restoring group is updated to match the internal object identifier written at the end of the main restoring group.

2. The data reconstruction method according to claim 1, characterized in that, Based on the internal object identifiers written at the end of the main placement group and the placement group to be restored, the missing range and residual range of the placement group to be restored are divided, including: During the group alignment process, read the internal object identifiers written at the end of the main group and the group to be restored. Based on the internal object identifier, the missing range of data to be supplemented and the residual range of invalid data to be cleaned in the group to be restored are determined to complete the initialization of the reconstruction range.

3. The data reconstruction method according to claim 1, characterized in that, Residual object cleanup is performed on the missing range, including: Starting from the current starting position of the missing range, obtain the objects of the group to be restored and placed in batches according to the preset processing quantity; The objects in the batch-acquired restoring group are compared with the objects in the main restoring group; If the corresponding object of the placement group to be restored does not exist in the main placement group, then the corresponding object of the placement group to be restored is deleted; After deletion is complete, update the starting position of the missing range to the position where the processing is currently complete.

4. The data reconstruction method according to claim 1, characterized in that, The residual range is used to filter objects to be reconstructed, including: Starting from the current starting position of the residual range, obtain the objects of the main placement group in batches according to the preset processing quantity; Based on the objects of the main placement group obtained in batches, a list of objects to be refactored is generated.

5. The data reconstruction method according to claim 4, characterized in that, The residual range is filtered for junk data, including: Read the effective data length of each object to be reconstructed from the list of objects to be reconstructed from the metadata storage pool; Determine whether the percentage of valid data corresponding to the valid data length is less than a preset threshold; If so, garbage collection will be performed on the objects to be reconstructed where the proportion of valid data is less than a preset threshold; If not, retain the objects to be reconstructed with a valid data ratio of not less than a preset threshold, and treat the retained objects to be reconstructed as valid objects.

6. The data reconstruction method according to claim 1, characterized in that, Perform data reconstruction on the filtered valid objects, including: When the data pool does not use erasure coding storage, read the valid data of the valid object; When the data pool uses erasure coding storage, a set number of fragment data are read from the valid object, and the missing fragments of the valid object are calculated. Transmit the valid data of the valid object or the missing fragments of the valid object to the group to be restored and placed. After the transmission is completed, the starting position of the residual range is updated to the position where the current processing is completed.

7. The data reconstruction method according to claim 1, characterized in that, When both the missing range and the residual range are empty, the internal object identifier written at the end of the restoring group is updated to match the internal object identifier written at the end of the main restoring group, including: The process of repeatedly cleaning up residual objects in the missing range, filtering objects to be reconstructed and filtering garbage data in the residual range, and reconstructing valid object data continues until both the missing range and the residual range are empty. After determining that both the missing range and the residual range are empty, the internal object identifier written at the end of the restoring group is synchronously updated to the same value as the internal object identifier written at the end of the main restoring group.

8. The data reconstruction method according to claim 1, characterized in that, Before dividing the missing and residual ranges of the group to be restored and repositioned, the process also includes: Create an internal object and generate an internal object identifier in integer form; The internal object identifiers are generated according to a monotonically increasing rule, and the creation order of internal objects is determined based on the internal object identifiers.

9. The data reconstruction method according to claim 1, characterized in that, While performing data reconstruction on the filtered valid objects, it also includes: If a new internal object is created, the new internal object is written to the recovery placement group, and no reconstruction processing of the missing range and residual range is performed on the new internal object.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data reconstruction method as described in any one of claims 1 to 9 when executing the computer program.

Images