Metadata management method and device, data reading method and device, equipment and storage medium

Abstract

The invention discloses a metadata management method and device, a data reading method and device, equipment and a storage medium, and relates to the technical field of data processing.The metadata management method comprises the steps that when it is detected that a fault hard disk exists, metadata of a to-be-recovered data block on the fault hard disk is obtained; according to the metadata, a to-be-recovered data structure is determined, metadata of other data blocks are extracted from the data structure, and the to-be-recovered data blocks are reconstructed; configuring the reconstructed data blocks to other hard disks without faults, creating new strips according to all the reconstructed data blocks, and generating verification blocks and metadata of the verification blocks; establishing a mapping relationship between the metadata of the to-be-recovered data block and the metadata of the reconstructed data block; and storing the mapping relationship in the data structure, and updating the metadata of the reconstructed data block and the metadata of the check block to a new data structure. According to the method, only the changed blocks are subjected to check calculation, the data are directly recovered by utilizing the mapping relation in the metadata, and additional check calculation is not needed.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to metadata management, data reading methods, devices, equipment and storage media. Background Art

[0002] With the rapid growth of data volumes, distributed storage systems are becoming increasingly important in enterprise storage, cloud computing, and big data applications. Redundant Array of Independent Disks (RAID) is a common data redundancy and fault tolerance technology in distributed storage systems. However, traditional RAID technology faces challenges in storage system design, particularly when there are no hot spare drives. How to quickly restore redundancy and minimize the impact on system performance when faced with a disk failure remains a challenge in storage system design.

[0003] Column shrinking and reconstruction technology aims to maintain RAID redundancy by reducing stripe width in the absence of hot spare drives. However, current column shrinking and reconstruction solutions have many practical issues, such as the time-consuming recalculation of full parity blocks, the complex reconstruction process, and the increased system coupling caused by mapping changes. These issues seriously affect system stability and data security. Summary of the Invention

[0004] The present application provides metadata management, data reading methods, devices, equipment and storage media to at least solve the many problems in the relevant technology in the actual application of the current column reduction and reconstruction scheme in the absence of a hot spare disk, thereby affecting the stability of the system and data security.

[0005] The present application provides a metadata management method, comprising: when a faulty hard disk is detected, obtaining metadata of a data block to be recovered on the faulty hard disk;

[0006] Determine the data structure corresponding to the stripe to which the data block to be recovered belongs based on the metadata of the data block to be recovered;

[0007] Extract metadata of other data blocks except the data block to be recovered from the data structure;

[0008] Reconstruct the data block to be recovered based on the metadata of other data blocks, and obtain the reconstructed data block and the metadata of the reconstructed data block;

[0009] After allocating the reconstructed data blocks to other surviving hard disks, a new stripe and a new data structure corresponding to the new stripe are created based on all the reconstructed data blocks. The parity blocks and metadata for the parity blocks are also generated on the new stripe.

[0010] Establishing a mapping relationship between the metadata of the data block to be restored and the metadata of the reconstructed data block corresponding to the data block to be restored;

[0011] The mapping relationship is stored in the data structure to which the stripe to which the data block to be recovered belongs belongs, and the metadata of the reconstructed data block and the metadata of the check block are updated in the new data structure.

[0012] This application also provides a data reading method, including:

[0013] Receiving a data block read request sent by a data block request end, wherein the data block read request includes metadata related to the data block to be read;

[0014] Determine, based on the metadata of the data block to be read, the data structure to which the metadata belongs, as described in the metadata management method of the first aspect;

[0015] According to the data structure, obtain the target data block corresponding to the metadata;

[0016] Feedback the target data block to the data block request end.

[0017] This application also provides a metadata management device, including:

[0018] A detection module is used to detect whether there is a faulty hard disk;

[0019] An acquisition module, used for acquiring metadata of data blocks to be recovered on a faulty hard disk when a faulty hard disk is detected;

[0020] A determination module, configured to determine a data structure corresponding to a stripe to which the data block to be recovered belongs based on metadata of the data block to be recovered;

[0021] An extraction module, configured to extract metadata of other data blocks except the data block to be recovered from the data structure;

[0022] A reconstruction module is used to reconstruct the data block to be recovered based on the metadata of other data blocks, and obtain the reconstructed data block and the metadata of the reconstructed data block;

[0023] A processing module configured to allocate the reconstructed data blocks to other surviving hard disks, create new stripes and new data structures corresponding to the new stripes based on all the reconstructed data blocks, and generate parity blocks on the new stripes and metadata for the parity blocks;

[0024] An establishing module, configured to establish a mapping relationship between metadata of a data block to be recovered and metadata of a reconstructed data block corresponding to the data block to be recovered;

[0025] The update module is used to store the mapping relationship in the data structure to which the stripe to which the data block to be recovered belongs belongs, and to update the metadata of the reconstructed data block and the metadata of the check block into the new data structure.

[0026] The present application also provides a data reading device, comprising: a receiving module, configured to obtain a data block reading request sent by a data block requesting end, wherein the data block reading request includes metadata related to a data block to be read;

[0027] A processing module, configured to determine, based on the metadata of the data block to be read, a data structure in the metadata management method of the first aspect to which the metadata belongs; and obtain, based on the data structure, a target data block corresponding to the metadata;

[0028] The sending module is used to send the target data blocks to the data block request end.

[0029] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned metadata management methods when executing the computer program.

[0030] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned metadata management methods are implemented.

[0031] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned metadata management methods when executed by a processor.

[0032] Through the present application, when a faulty hard disk is detected, the metadata of the data blocks to be recovered on the faulty hard disk is obtained, and the corresponding data structure of the stripe to which the data blocks to be recovered belong is determined based on the metadata, and the metadata of other data blocks are extracted from the data structure. The data blocks to be recovered are reconstructed based on the metadata of other data blocks, and the reconstructed data blocks and the metadata of the reconstructed data blocks are obtained. Then, all the reconstructed data blocks are configured on other hard disks that have not failed, and new stripes and new data structures corresponding to the new stripes are created based on all the reconstructed data blocks, and check blocks on each stripe and metadata corresponding to the check blocks are generated. A mapping relationship between the metadata of the data blocks to be recovered and the metadata of the reconstructed data blocks corresponding to the data blocks to be recovered is established, and then the mapping relationship is stored in the data structure to which the stripe to which the data blocks to be recovered belong belongs, and the metadata of the reconstructed data blocks and the metadata of the check blocks are updated to the new data structure. Using this method, if the data blocks to be recovered are subsequently queried, the metadata of the reconstructed data blocks can be obtained based on the mapping relationship between the metadata of the reconstructed data blocks and the metadata of the data to be recovered, thereby obtaining the reconstructed data. The reconstructed data is the data to be recovered. This method can recover data from a failed hard drive even without a hot spare disk, and the original stripe width does not need to be changed in this solution to maintain redundancy. Furthermore, only the changed data blocks are verified, reducing the amount of verification calculations. Compared to the current column-shrinking reconstruction solution that recalculates the verification blocks for all stripe data, this method is time-efficient and only the changed data is reconstructed. The reconstruction process is simple, the mapping relationship is also relatively simple, and changes in the mapping relationship will not cause increased system coupling. New verification information is recorded in the metadata to ensure the accuracy of data redundancy, the reliability and high availability of metadata, prevent data loss, and simplify the data recovery process. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 A flowchart of a metadata management method provided in an embodiment of the present application;

[0035] Figure 2 A simplified structural diagram of a RAID 5 system provided in an embodiment of the present application;

[0036] Figure 3A schematic diagram of a simplified principle structure for reconstructing blocks of data to be recovered in a stripe included in a RAID5 system provided in an embodiment of the present application;

[0037] Figure 4 A flowchart of another metadata management method provided in an embodiment of the present application;

[0038] Figure 5 A flowchart of another metadata management method provided in an embodiment of the present application;

[0039] Figure 6 A flowchart of a data reading method provided in an embodiment of the present application;

[0040] Figure 7 A simplified schematic diagram of the principle of writing back reconstructed data blocks in a stripe to a new hard disk included in the RAID5 system provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of the structure of a metadata management device provided in an embodiment of the present application;

[0042] Figure 9 A schematic diagram of the structure of a data reading device provided in an embodiment of the present application;

[0043] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0046] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0047] With the rapid growth of data volumes, distributed storage systems are becoming increasingly important in enterprise storage, cloud computing, and big data applications. RAID, a common data redundancy and fault tolerance technology in distributed storage systems, combines multiple physical disks into a single logical unit, providing highly reliable and high-performance storage solutions. However, traditional RAID technology has always been challenging in storage system design, especially when there is no hot spare drive. How to quickly restore redundancy and minimize the impact on system performance when faced with a disk failure remains a challenge.

[0048] Column-shrinking reconstruction is an important technology designed to maintain RAID redundancy by reducing stripe width in the absence of hot spare drives. However, existing column-shrinking reconstruction solutions suffer from numerous practical issues, such as time-consuming full parity block recalculation, complex reconstruction processes, and increased system coupling due to mapping changes. These issues severely impact system stability and data security, necessitating a more efficient and reliable solution.

[0049] Specifically, in RAID, there is usually a check block to store additional information so that data can be restored when it is damaged. However, after performing the column reduction reconstruction, the original check block may no longer be applicable, and a new check block needs to be recalculated. The process of recalculating the check block is relatively time-consuming, especially when the amount of data is large, this defect is particularly obvious. Furthermore, when using existing technologies to perform column reduction reconstruction, it involves the redistribution of data between multiple hard disks and the updating of the check block, resulting in a more complicated reconstruction process. Furthermore, when performing operations through existing column reduction reconstruction technology, because the storage location of the data on the hard disk will be changed, the mapping relationship in the system will change. If these changes are not properly handled, the coupling between systems will increase, making the system more difficult to maintain and expand.

[0050] To solve the above problems, the present invention provides a metadata management method. Figure 1 As shown, the method includes the following steps:

[0051] Step S101 : when a faulty hard disk is detected, metadata of the data blocks to be recovered on the faulty hard disk are obtained.

[0052] For details, see Figure 2 As shown, taking the RAID5 system as an example, assuming that the system includes 4 hard disks, hard disk 1, hard disk 2, hard disk 3 and hard disk 4, there are 3 stripes in the RAID5 system, namely stripe 1 composed of A1 to P1, stripe 2 composed of B2 to A2, and stripe 3 composed of C3 to B3.

[0053] Assume that when a failure of hard disk 3 is detected, it is first necessary to obtain metadata of the to-be-recovered data blocks C1 , P2 , and A3 on the failed hard disk 3 .

[0054] In an optional example, assuming that the metadata of the data block to be recovered is the identification information corresponding to the data block, in the embodiment of the present application, the identification information can be the ID information of the data block. Wherein, the data block ID is a globally unique identifier used to identify each data block.

[0055] Step S102: determining the data structure corresponding to the stripe to which the data block to be recovered belongs according to the metadata of the data block to be recovered.

[0056] Step S103: extract metadata of other data blocks except the data block to be restored from the data structure.

[0057] Specifically, as previously described, each data block belongs to a stripe in a RAID 5 system. Data blocks and stripes inherently have a specific mapping relationship. Therefore, the stripe information of the stripe to which the data block to be recovered belongs can be determined based on the metadata of the data block to be recovered. Furthermore, the data structure to which the stripe belongs can be determined based on the stripe information.

[0058] See also Figure 2 As shown, taking the data block C1 to be recovered as an example, the stripe to which it belongs is stripe 1. In a specific example, the stripe information corresponding to stripe 1 is, for example, identification information of stripe 1, preferably stripe ID information. Stripe ID: a unique identifier of a stripe.

[0059] In an alternative example, assume that the metadata of a stripe and the metadata of each data block in the stripe are stored in a pre-built data structure. In a specific example, the data structure is a B+ tree, but other data structures are also possible, such as a skip list, a hash table, or a distributed hash table (DHT).

[0060] Then, it is also possible to directly filter multiple pre-built data structures according to the metadata of the data block to be recovered, and obtain the data structure to which the metadata of the data block to be recovered belongs.

[0061] Specifically, as described above, each stripe has a data structure that contains metadata corresponding to all data blocks included in the stripe. Therefore, based on the metadata of the data block to be recovered, you can filter from multiple pre-built data structures to obtain the data structure containing the metadata of the data block to be recovered.

[0062] Thereafter, metadata of other data blocks except the data block to be restored may be directly extracted from the data structure.

[0063] Step S104: reconstruct the data block to be recovered according to the metadata of other data blocks, and obtain the reconstructed data block and the metadata of the reconstructed data block.

[0064] Specifically, taking the previously mentioned data block C1 to be recovered as an example, metadata for data blocks A1, B1, and P1 in stripe 1 can be obtained from the data structure, and then data blocks A1, B1, and P1 can be obtained. Then, based on these three data blocks, a new C1' is generated. This is a reconstructed data block equivalent to C1, and metadata corresponding to C1' is generated, which is the identification information (ID) corresponding to the reconstructed data block C1'.

[0065] Step S105: After allocating the reconstructed data blocks to other surviving hard disks, a new stripe and a new data structure corresponding to the new stripe are created based on all the reconstructed data blocks, and a check block and metadata of the check block are generated on the new stripe.

[0066] Specifically, after all the data blocks to be recovered in hard disk 3 have completed data reconstruction, that is, after the corresponding reconstructed data blocks and metadata of the reconstructed data blocks have been obtained, all the reconstructed data are configured on other hard disks that have not failed.

[0067] See Figure 3 As shown in , assume that C1' and A3' are allocated to hard disk 1, and P2' is allocated to hard disk 2. Therefore, two new stripes are created, stripe 4 and stripe 5, and parity blocks are generated on the new stripes. Figure 3 The check block P1" corresponding to stripe 4 and the check block P2" on stripe 5. At the same time, metadata of the check blocks also needs to be generated.

[0068] Step S106: establishing a mapping relationship between the metadata of the data block to be restored and the metadata of the reconstructed data block corresponding to the data block to be restored.

[0069] Step S107 : storing the mapping relationship in the data structure to which the stripe to which the data block to be recovered belongs, and updating the metadata of the reconstructed data block and the metadata of the check block into the new data structure.

[0070] Specifically, one stripe corresponds to a data structure, that is, a B+ tree. Therefore, the mapping relationship between the metadata of the data block to be recovered and the metadata of the reconstructed data block is stored in the B+ tree to which the data block to be recovered belongs, such as a mapping address pointer, which is used to indicate the storage address of the metadata of the reconstructed data block (for example, the pointer is the address of the B+ tree for storing the metadata of the reconstructed data block). Of course, the mapping relationship can also be the ID of the stripe to which the metadata of the reconstructed data block belongs. In another optional example, the mapping relationship can also be the metadata of the reconstructed data block itself. There is no limitation here as to what it is specifically. Then, the metadata of the reconstructed data block and the metadata of the check block are updated to the new data structure to which they belong.

[0071] The above methods can flexibly obtain the metadata of the reconstructed data blocks, which is more convenient and quick.

[0072] Specifically, see Figure 3 As shown, stripe 4 corresponds to a data structure, so the metadata of the data blocks included in stripe 4, such as C1' and P2', and the metadata corresponding to P1" will be stored in the B+ tree corresponding to stripe 4. The metadata corresponding to A3' and P2" will be stored in the B+ tree corresponding to stripe 5.

[0073] The reason why the metadata of the newly generated check block is also stored in the B+ tree is to prevent subsequent hard disk failures so that the final data block can be generated through hierarchical backtracking.

[0074] Specifically, the solution of the present application is that in the absence of a hot spare disk, the data of a failed hard disk can be restored in the above-mentioned manner, ensuring that the current RIAD5 system can still have redundancy even when one hard disk is damaged. That is to say, in this case, even if another hard disk is damaged, for example, if hard disk 1 is damaged, all data in hard disk 1 can be restored. Specifically, reconstructed data block C1' can be generated through P1" and P2. Reconstructed data A3' of A3' can be generated based on P2". Reconstructed data C3' of C3 can be generated based on B3, A3", and P3, and B2' can be reconstructed based on A2, P2' and C2, and reconstructed data A1 can be generated based on P1, C1" and B1.

[0075] The embodiment of the present application provides a metadata management method, which obtains the metadata of the data block to be recovered on the faulty hard disk when a faulty hard disk is detected, and determines the corresponding data structure of the stripe to which the data block to be recovered belongs based on the metadata, and extracts the metadata of other data blocks from the data structure. The data block to be recovered is reconstructed based on the metadata of other data blocks, and the reconstructed data block and the metadata of the reconstructed data block are obtained. Then, all the reconstructed data blocks are configured on other hard disks that have not failed, and new stripes and new data structures corresponding to the new stripes are created based on all the reconstructed data blocks, and check blocks on each stripe and metadata corresponding to the check blocks are generated. A mapping relationship between the metadata of the data block to be recovered and the metadata of the reconstructed data block corresponding to the data block to be recovered is established, and then the mapping relationship is stored in the data structure to which the stripe to which the data block to be recovered belongs belongs, and the metadata of the reconstructed data block and the metadata of the check block are updated to the new data structure. Using this method, if the data blocks to be recovered are subsequently queried, the metadata of the reconstructed data blocks can be obtained based on the mapping relationship between the metadata of the reconstructed data blocks and the metadata of the data to be recovered, thereby obtaining the reconstructed data. The reconstructed data is the data to be recovered. This method can recover data from a failed hard drive even without a hot spare disk, and the original stripe width does not need to be changed in this solution to maintain redundancy. Furthermore, only the changed data blocks are verified, reducing the amount of verification calculations. Compared to the current column-shrinking reconstruction solution that recalculates the verification blocks for all stripe data, this method is time-efficient and only the changed data is reconstructed. The reconstruction process is simple, the mapping relationship is also relatively simple, and changes in the mapping relationship will not cause increased system coupling. New verification information is recorded in the metadata to ensure the accuracy of data redundancy, the reliability and high availability of metadata, prevent data loss, and simplify the data recovery process.

[0076] In an optional example, the data structure includes block structure information corresponding to each data block in the stripe; the fields in the block structure information include metadata fields of the data block, mapping relationship fields, metadata of the stripe to which it belongs, and block status fields corresponding to each data block; Based on the above embodiment, the method may further include the following method steps, see for details Figure 4 Shown, including:

[0077] Step S401: before obtaining the reconstructed data block and metadata of the reconstructed data block, modify the field value of the block status field from the normal state to the reconstructed state.

[0078] Step S402: After obtaining the reconstructed data block and metadata of the reconstructed data block, the field value of the block status field is modified from the reconstructed state to the reconstructed completed state.

[0079] Specifically, the block structure information is as follows, including but not limited to the following fields:

[0080] Metadata field (chunk ID): A globally unique identifier used to identify each chunk.

[0081] Mapping relationship field (eg, ID of reconstructed data): the target block ID of the block after reconstruction.

[0082] Block status field: indicates the current status of the block, such as normal, rebuilding, rebuild completed, write back, etc.

[0083] Metadata of the stripe (stripe ID): a unique identifier for the stripe.

[0084] In a specific example, before obtaining the reconstructed data blocks and the metadata of the reconstructed data blocks, the field value of the block status field is changed from the normal state to the reconstructed state to indicate that the current data to be recovered is in the reconstruction process. If you need to obtain the reconstructed data of the data to be recovered, you need to wait for a while.

[0085] After obtaining the reconstructed data blocks and metadata of the reconstructed data blocks, the field value of the block status field is modified from the reconstructed state to the reconstructed completed state to indicate that the reconstruction is now completed and the reconstructed data can be obtained.

[0086] The field value of the block status field can include not only "reconstructing" and "reconstruction completed", but also "normal" and "write back" states.

[0087] The write-back state is to write the reconstructed data back to the new hard disk in blocks to reduce the storage space occupied by the original hard disk of the current system.

[0088] On the basis of any of the above embodiments, the method may further include the following steps, see Figure 5 Shown, including:

[0089] Step S501: periodically detect the mapping relationship field in the block structure information.

[0090] Step S502: When it is determined according to the data structure that the mapping relationship is invalid, after deleting the mapping relationship in the data structure, the field value of the mapping relationship field in the block structure is updated to the initial value.

[0091] Specifically, in order to ensure that the mapping relationship fields and other fields in the data structure are accurate, periodic testing is required. This includes testing whether the mapping relationship fields in the block structure information are accurate. For example, if the data block has been written back, the mapping relationship has become invalid, and the mapping relationship should be deleted from the data structure, and the field value of the mapping relationship field in the data structure should be updated to the initial value. In a specific example, the initial value of the field value is, for example, the ID information of the data block.

[0092] In an optional example, the data structure is described below, that is, the structure of each node of the B+ tree in the embodiment of the present application. Each node of the B+ tree contains the following fields:

[0093] Keyset: A globally unique identifier (key) used to store data chunks.

[0094] Value set: block structure information (value) of data blocks.

[0095] Child node pointer collection: pointers to child nodes, used for the tree hierarchy.

[0096] Parent node pointer: A pointer to the parent node, used for tree backtracking operations.

[0097] Additional fields: such as data block status, verification information and other auxiliary information.

[0098] Root node (strip ID): As the top node of the B+ tree, the root node contains the entire tree structure and points to its child nodes. The location of the root node is cached in memory to speed up access.

[0099] When the system starts, the B+ tree structure is initialized to ensure that the initial mapping relationship of all blocks is accurate. In the initial state, the mapping relationship of all blocks is itself, that is, the field value in the mapping relationship field is the block ID. When a new mapping relationship is established, first find the appropriate position in the B+ tree based on the reconstructed data block ID; then insert the reconstructed data block ID into that position. If necessary, the node can be split to keep the tree balanced; finally, the pointer of the parent node is updated to ensure that the tree hierarchy is correct. In addition, when the mapping relationship becomes invalid or is updated, it is also necessary to find the corresponding node in the B+ tree (the position where the reconstructed data block ID is inserted) based on the block ID, and delete the mapping relationship from the B+ tree. At the same time, if the corresponding node needs to be deleted, delete the node and update the B+ tree.

[0100] Further optionally, based on any of the aforementioned embodiments, in addition to considering how to increase the redundancy of the system after a hard drive failure, it is actually possible to take preventive measures in the early stages. For example, relevant parameters of the hard drive can be collected in advance, such as the hard drive's health status, temperature, read error rate, hard drive age, usage time, number of reads and writes, and other parameters. A predictive model is trained based on these parameters to enable it to identify potential failure risks and take preventive measures in advance. Of course, some preventive measures can also be included, such as adjusting cache strategies, reducing congestion, performing disk scans, etc.

[0101] The embodiment of the present application provides a data reading method, which includes the following method steps, see Figure 6 Shown, including:

[0102] Step S601: receiving a data block read request sent by a data block request end.

[0103] Specifically, the data block read request includes metadata of the data block to be read.

[0104] Step S602: Determine, based on the metadata of the data block to be read, the data structure to which the metadata belongs, as included in any of the aforementioned embodiments.

[0105] Specifically, the process of determining the data structure to which the metadata belongs based on the metadata of the data block to be read in this embodiment is similar to the process of determining the data structure corresponding to the stripe to which the data block to be recovered belongs based on the metadata of the data block to be recovered in any of the aforementioned embodiments, and will not be elaborated here.

[0106] Step S603: Obtain target data blocks corresponding to the metadata according to the data structure.

[0107] Specifically, in an optional example, when the mapping relationship field value corresponding to the metadata in the block structure information of the data structure is a preset threshold, such as an initial value, and the status value in the block status field is normal, the target data block can be obtained directly based on the metadata in the data structure.

[0108] In another optional example, when it is determined that the field of the mapping relationship field corresponding to the metadata in the block structure information in the data structure is not the preset threshold, for example, it is the address pointer for indicating the metadata of the reconstructed data block or the ID of the stripe to which the reconstructed data block belongs, and the field value of the separation status field in the block structure information is identified as reconstruction completed, then according to the mapping relationship field, the block structure information to which the mapping relationship field belongs is searched, and it is further determined whether the field value of the mapping relationship field corresponding to the mapping relationship field in the block structure information is the preset threshold, until it is determined that the field value of the mapping relationship field of the reconstructed data block field included in the block structure information finally found and obtained is the preset threshold, the target data block is obtained according to the metadata field in the block structure information finally found and obtained.

[0109] Step S604: Feedback the target data block to the data block requesting end.

[0110] Optionally, the above method can provide multiple control modes to meet performance and reliability requirements:

[0111] For example, when reading metadata, a shared lock is acquired to allow multiple read operations to proceed simultaneously, but write operations are blocked; when performing insert, delete, or update operations, an exclusive lock is acquired to ensure that only one write operation is performed at the same time.

[0112] To ensure the reliability of metadata, the following data persistence solution is adopted:

[0113] Storing part of the B+ tree's root node in memory improves access speed, while storing other nodes on disk ensures data persistence. Redundant storage of key B+ tree nodes, distributed across different storage nodes, improves the system's fault tolerance.

[0114] The above data reading method can also be used to write data back to a new hard disk. Figure 7 As shown, Figure 7 The diagram shows that the reconstructed data blocks C1', A3' and P2' are added to the new hard disk 3'. The specific reading process has been described in detail above and will not be repeated here.

[0115] An embodiment of the present application provides a data reading method that receives a data block read request sent by a data block requesting end, wherein the read request includes metadata corresponding to the data block to be read. Based on the metadata, the data structure to which the metadata belongs, such as that described in any of the aforementioned metadata management method embodiments, is first determined. Then, based on the data structure, the target data block corresponding to the metadata is obtained. The target data block is then fed back to the data block requesting end. In this method, because a mapping relationship between reconstructed data and data to be recovered is established in the data structure in the aforementioned method, in this embodiment, when querying the target data block, the metadata of the data block to be read can be used to search the data structure for metadata corresponding to the metadata, or metadata of the reconstructed data block that is mapped to the metadata. The target data block is then obtained based on the metadata corresponding to the data block to be read or the metadata of the reconstructed data block. This method can more conveniently and concisely obtain the target data block, reduce the verification calculation process, and also ensure the accuracy of data redundancy, the reliability and high availability of metadata, improve the fault tolerance of the system, prevent data loss, and simplify the data recovery process.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0117] The embodiment of the present application also provides a metadata management device, see Figure 8 As shown, the apparatus includes: a detection module 801 , an acquisition module 802 , an extraction module 803 , a reconstruction module 804 , a processing module 805 , and an update module 806 .

[0118] Detection module 801, used to detect whether there is a faulty hard disk;

[0119] The acquisition module 802 is used to acquire metadata of the data blocks to be recovered on the faulty hard disk when a faulty hard disk is detected;

[0120] A determination module, configured to determine a data structure corresponding to a stripe to which the data block to be recovered belongs based on metadata of the data block to be recovered;

[0121] Extraction module 803, used to extract metadata of other data blocks except the data block to be restored from the data structure;

[0122] A reconstruction module 804 is configured to reconstruct the data block to be recovered based on metadata of other data blocks, and obtain the reconstructed data block and metadata of the reconstructed data block;

[0123] Processing module 805 is configured to create a new stripe and a new data structure corresponding to the new stripe based on all the reconstructed data blocks after allocating the reconstructed data blocks to other surviving hard disks, and generate a parity block on the new stripe and metadata for the parity block;

[0124] An establishing module, configured to establish a mapping relationship between metadata of a data block to be recovered and metadata of a reconstructed data block corresponding to the data block to be recovered;

[0125] The updating module 806 is configured to store the mapping relationship in the data structure to which the stripe to which the data block to be recovered belongs belongs, and update the metadata of the reconstructed data block and the metadata of the check block into the new data structure.

[0126] In an optional example, the determination module is specifically configured to filter multiple pre-built data structures according to the metadata of the data block to be recovered, and obtain the data structure to which the metadata of the data block to be recovered belongs.

[0127] In an optional example, the data structure includes block structure information corresponding to each data block in the stripe; the fields in the block structure information include a metadata field of the data block, a mapping relationship field, metadata of the stripe to which it belongs, and a block status field corresponding to each data block; the processing module 805 is further configured to modify the field value of the block status field from a normal state to a reconstructed state before obtaining the reconstructed data block and the metadata of the reconstructed data block;

[0128] And, after obtaining the reconstructed data block and the metadata of the reconstructed data block, the field value of the block status field is modified from the reconstructed state to the reconstructed completed state.

[0129] In an optional example, the detection module 801 is further configured to periodically detect a mapping relationship field in the block structure information;

[0130] The updating module 806 is further configured to update the field value of the mapping relationship field in the block structure to an initial value after deleting the mapping relationship in the data structure when the mapping relationship is determined to be invalid according to the data structure.

[0131] For the description of the features in the embodiment corresponding to the metadata management device provided in the embodiment of the present application, please refer to the relevant description of the embodiment corresponding to the metadata management method, and no further details will be given here.

[0132] The metadata management device provided by the embodiment of the present application obtains the metadata of the data block to be recovered on the faulty hard disk when a faulty hard disk is detected, and determines the corresponding data structure of the stripe to which the data block to be recovered belongs based on the metadata, and extracts the metadata of other data blocks from the data structure. The data block to be recovered is reconstructed based on the metadata of other data blocks, and the reconstructed data block and the metadata of the reconstructed data block are obtained. Then, all the reconstructed data blocks are configured on other hard disks that have not failed, and new stripes and new data structures corresponding to the new stripes are created based on all the reconstructed data blocks, and check blocks on each stripe and metadata corresponding to the check blocks are generated. A mapping relationship between the metadata of the data block to be recovered and the metadata of the reconstructed data block corresponding to the data block to be recovered is established, and then the mapping relationship is stored in the data structure to which the stripe to which the data block to be recovered belongs belongs, and the metadata of the reconstructed data block and the metadata of the check block are updated to the new data structure. Using this method, if the data blocks to be recovered are subsequently queried, the metadata of the reconstructed data blocks can be obtained based on the mapping relationship between the metadata of the reconstructed data blocks and the metadata of the data to be recovered, thereby obtaining the reconstructed data. The reconstructed data is the data to be recovered. This method can recover data from a failed hard drive even without a hot spare disk, and the original stripe width does not need to be changed in this solution to maintain redundancy. Furthermore, only the changed data blocks are verified, reducing the amount of verification calculations. Compared to the current column-shrinking reconstruction solution that recalculates the verification blocks for all stripe data, this method is time-efficient and only the changed data is reconstructed. The reconstruction process is simple, the mapping relationship is also relatively simple, and changes in the mapping relationship will not cause increased system coupling. New verification information is recorded in the metadata to ensure the accuracy of data redundancy, the reliability and high availability of metadata, prevent data loss, and simplify the data recovery process.

[0133] The embodiment of the present application also provides a data reading device, see Figure 9 As shown, the device includes: a receiving module 901, a processing module 902, and a sending module 903.

[0134] A receiving module 901 is configured to obtain a data block read request sent by a data block request terminal, wherein the data block read request includes metadata related to the data block to be read;

[0135] Processing module 902 is configured to determine, based on the metadata of the data block to be read, a data structure to which the metadata belongs, as described in any of the embodiments corresponding to the aforementioned metadata management method; and obtain, based on the data structure, a target data block corresponding to the metadata;

[0136] The sending module 903 is used to send the target data block to the data block request end.

[0137] In an optional example, the processing module 902 is specifically configured to identify whether a field value of a mapping relationship field corresponding to metadata in the block structure information in the data structure is a preset threshold;

[0138] When it is determined that the field value of the mapping relationship field is not the preset threshold, identifying the field value of the block status field in the block structure information;

[0139] When the field value of the identification block status field is reconstruction completed, the block structure information to which the mapping relationship field belongs is searched according to the mapping relationship field;

[0140] and further determining whether a field value of a mapping relationship field corresponding to the mapping relationship field in the block structure information to which the mapping relationship field belongs is a preset threshold;

[0141] When it is determined that the field value of the mapping relationship field included in the finally found and obtained block structure information is the preset threshold, the target data block is obtained according to the metadata field in the finally found and obtained block structure information.

[0142] For the description of the features in the embodiment corresponding to a data reading device provided in an embodiment of the present application, reference can be made to the relevant description of the embodiment corresponding to the data reading method, which will not be repeated here.

[0143] An embodiment of the present application provides a data reading device that receives a data block read request sent by a data block requesting end, wherein the read request includes metadata corresponding to the data block to be read. Based on the metadata, the device first determines the data structure to which the metadata belongs, as described in any of the aforementioned metadata management method embodiments. Then, based on the data structure, the device obtains the target data block corresponding to the metadata. The target data block is then fed back to the data block requesting end. In this method, because a mapping relationship between reconstructed data and data to be recovered is established in the data structure in the aforementioned method, in this embodiment, when querying the target data block, the device can search the data structure for metadata corresponding to the metadata of the data block to be read, or metadata of the reconstructed data block that has a mapping relationship with the metadata. The target data block is then obtained based on the metadata corresponding to the data block to be read or the metadata of the reconstructed data block. This method can more conveniently and concisely obtain the target data block, reduce the verification calculation process, and also ensure the accuracy of data redundancy, the reliability and high availability of metadata, improve the fault tolerance of the system, prevent data loss, and simplify the data recovery process.

[0144] The embodiment of the present application also provides an electronic device, such as Figure 10As shown, it includes a memory 10 and a processor 20, the memory 10 stores a computer program, and the processor 20 is configured to run the computer program to execute the steps in any of the above-mentioned metadata management method embodiments, or execute the steps in any of the above-mentioned data reading method embodiments.

[0145] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned metadata management method embodiments, or execute the steps of any of the above-mentioned data reading method embodiments when running.

[0146] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0147] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned metadata management method embodiments are implemented.

[0148] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data reading method embodiments are implemented.

[0149] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] The above is a detailed introduction to the metadata management and data reading method, device, equipment, and storage medium provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A metadata management method, characterized in that: The method comprises: When a faulty hard disk is detected, metadata of the data blocks to be recovered on the faulty hard disk is obtained; Determining, based on the metadata of the data block to be recovered, a data structure corresponding to the stripe to which the data block to be recovered belongs; Extracting metadata of other data blocks except the data block to be restored from the data structure; Reconstructing the data block to be recovered according to the metadata of the other data blocks, and obtaining the reconstructed data block and the metadata of the reconstructed data block; After allocating the reconstructed data blocks to other surviving hard disks, creating new stripes and new data structures corresponding to the new stripes based on all the reconstructed data blocks, and generating parity blocks on the new stripes and metadata for the parity blocks; Establishing a mapping relationship between the metadata of the data block to be restored and the metadata of the reconstructed data block corresponding to the data block to be restored; The mapping relationship is stored in the data structure to which the stripe to which the data block to be recovered belongs belongs, and the metadata of the reconstructed data block and the metadata of the check block are updated in the new data structure.

2. The method according to claim 1, characterized in that The step of determining, based on the metadata of the data block to be recovered, a data structure corresponding to a stripe to which the data block to be recovered belongs includes: According to the metadata of the data block to be restored, screening is performed among the multiple pre-constructed data structures to obtain the data structure to which the metadata of the data block to be restored belongs.

3. The method according to claim 1 or 2, characterized in that The data structure includes block structure information corresponding to each data block in the stripe; the fields in the block structure information include metadata fields of the data block, mapping relationship fields, metadata of the stripe to which it belongs, and block status fields corresponding to each data block; The method further comprises: Before obtaining the reconstructed data block and metadata of the reconstructed data block, modifying the field value of the block status field from a normal state to a reconstructed state; And, after obtaining the reconstructed data block and the metadata of the reconstructed data block, the field value of the block status field is modified from the reconstructed state to the reconstruction completed state.

4. The method according to claim 3, characterized in that The method further comprises: Periodically detecting the mapping relationship field in the block structure information; When it is determined according to the data structure that the mapping relationship is invalid, after deleting the mapping relationship in the data structure, the field value of the mapping relationship field in the block structure is updated to an initial value.

5. A data reading method, characterized in that: The method comprises: Receiving a data block read request sent by a data block request end, wherein the data block read request includes metadata related to the data block to be read; Determining, according to the metadata of the data block to be read, the data structure according to any one of claims 1 to 4 to which the metadata belongs; According to the data structure, obtaining target data blocks corresponding to the metadata; Feedback the target data block to the data block request end.

6. The method according to claim 5, characterized in that The acquiring, according to the data structure, target data blocks corresponding to the metadata specifically includes: Identifying whether a field value of a mapping relationship field corresponding to the metadata in the block structure information in the data structure is a preset threshold; When it is determined that the field value of the mapping relationship field is not the preset threshold, identifying the field value of the block status field in the block structure information; When it is identified that the field value of the block status field is reconstruction completed, searching for the block structure information to which the mapping relationship field belongs according to the mapping relationship field; and further determining whether a field value of a mapping relationship field corresponding to the mapping relationship field in the block structure information to which the mapping relationship field belongs is a preset threshold; When it is determined that the field value of the mapping relationship field included in the finally found and obtained block structure information is the preset threshold, the target data block is obtained according to the metadata field in the finally found and obtained block structure information.

7. A metadata management device, characterized in that: include: A detection module is used to detect whether there is a faulty hard disk; An acquisition module, used for acquiring metadata of data blocks to be recovered on a faulty hard disk when a faulty hard disk is detected; A determination module, configured to determine, based on metadata of the data block to be recovered, a data structure corresponding to a stripe to which the data block to be recovered belongs; An extraction module, configured to extract metadata of other data blocks except the data block to be restored from the data structure; A reconstruction module, configured to reconstruct the data block to be recovered based on the metadata of the other data blocks, and obtain the reconstructed data block and the metadata of the reconstructed data block; a processing module configured to, after allocating the reconstructed data blocks to other surviving hard disks, create new stripes and new data structures corresponding to the new stripes based on all the reconstructed data blocks, and generate parity blocks on the new stripes and metadata for the parity blocks; An establishing module, configured to establish a mapping relationship between the metadata of the data block to be restored and the metadata of the reconstructed data block corresponding to the data block to be restored; An updating module is configured to store the mapping relationship in a data structure to which the stripe to which the data block to be recovered belongs belongs, and to update the metadata of the reconstructed data block and the metadata of the check block into the new data structure.

8. A data reading device, characterized in that: include: A receiving module is configured to obtain a data block reading request sent by a data block request terminal, wherein the data block reading request includes metadata related to the data block to be read; a processing module, configured to determine, based on the metadata of the data block to be read, the data structure according to any one of claims 1 to 4 to which the metadata belongs; According to the data structure, obtaining target data blocks corresponding to the metadata; The sending module is used to send the target data block to the data block request end.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the metadata management method according to any one of claims 1 to 4 when executing the computer program, or implement the steps of the data reading method according to any one of claims 5 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the metadata management method according to any one of claims 1 to 4 are implemented, or when the computer program is executed by the processor, the steps of the data reading method according to any one of claims 5 to 6 are implemented.

Images