Metadata storage method and device, electronic equipment and storage medium

Abstract

The invention discloses a metadata storage method and device, electronic equipment and a storage medium, and relates to the technical field of storage, and the method comprises the steps that when a first processing node obtains fault information corresponding to a second processing node, address indication information can be extracted from the fault information, and based on the address indication information and the data stored in the data block included in the storage volume managed by the second processing node, determining whether metadata which is not stored in the disk exists, if so, determining candidate data blocks in the storage volume according to the address indication information and the preset number of data blocks, and storing the candidate data blocks in the storage volume. And extracting metadata which is not in the disk from the candidate data blocks. And finally, determining a first initial address according to the address indication information and a preset data block length, and taking the first initial address as an initial address to disk the unloaded metadata into the storage volume. Under the fault processing mechanism, the second processing node does not need to synchronize the metadata to the first processing node in real time, so that the problem of resource waste can be avoided.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a metadata storage method, device, electronic device, and storage medium. Background Art

[0002] In the field of storage technology, a storage system generally includes multiple processing nodes and a storage pool. The storage pool includes multiple volumes. Each processing node is used to manage one or more volumes in the storage pool.

[0003] Processing nodes can fail, and if metadata on the processing node isn't flushed to disk in a timely manner, this metadata can be lost, making operations on the data corresponding to this metadata impossible. Therefore, each processing node is typically assigned a corresponding backup node. The processing node can synchronize the generated metadata to the backup node in real time to prevent metadata loss. However, this real-time metadata synchronization mechanism wastes a significant amount of resources. Summary of the Invention

[0004] The present application provides a metadata storage method, device, electronic device, storage medium, and program product to solve the problem of wasting a lot of resources in synchronizing metadata.

[0005] The present application provides a metadata storage method, which is applied to a storage system and executed by a first processing node in the storage system. The method includes:

[0006] When fault information corresponding to the second processing node is obtained, address indication information is extracted from the fault information, wherein the first processing node is a standby node corresponding to the second processing node;

[0007] determining whether the second processing node has metadata that has not been written to disk, based on the address indication information and the data stored in the data blocks included in the storage volume managed by the second processing node;

[0008] When it is determined that the second processing node has metadata that has not been written to the disk, the candidate data blocks where the entity data corresponding to the metadata that has not been written to the disk is located are determined from the data blocks included in the storage volume based on the address indication information and the preset number of data blocks;

[0009] Extract metadata that has not been written to disk from the entity data stored in the candidate data block;

[0010] According to the address indication information and the preset data block length, the metadata that has not been written to the disk is stored in the storage volume.

[0011] The present application also provides a metadata storage device, which is applied to a first processing node in a storage system, and includes:

[0012] an extraction module configured to extract address indication information from the fault information when fault information corresponding to the second processing node is acquired, wherein the first processing node is a standby node corresponding to the second processing node;

[0013] a determination module configured to determine, based on the address indication information and data stored in data blocks included in the storage volume managed by the second processing node, whether the second processing node has metadata that has not been written to disk; and, when it is determined that the second processing node has metadata that has not been written to disk, determine, based on the address indication information and a preset number of data blocks, candidate data blocks containing entity data corresponding to the metadata that has not been written to disk, from the data blocks included in the storage volume;

[0014] The extraction module is further used to extract metadata that has not been written to the disk from the data stored in the candidate data blocks;

[0015] The storage module is further configured to store metadata that has not been written to the disk into the storage volume according to the address indication information and the preset data block length.

[0016] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned metadata storage methods when executing the computer program.

[0017] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned metadata storage methods are implemented.

[0018] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned metadata storage methods when executed by a processor.

[0019] Through the present application, each processing node can be provided with a corresponding backup node, and metadata can be added to a designated field of the entity data. In this way, when a processing node fails (the failed node, also known as the second processing node), the backup node (also known as the first processing node) can determine whether the failed node has completed the metadata write to disk at the time of the failure based on the address indication information carried in the failure information and the data stored in the data blocks included in the storage volume managed by the failed node, that is, determine whether there is metadata that has not been written to disk. If so, the backup node can select a candidate data block from the storage volume managed by the failed node. The candidate data block is the data block where the entity data corresponding to the metadata that has not been written to disk is located. Since the designated field of the entity data has been added with the corresponding metadata, the metadata that has not been written to disk can be extracted from the data stored in the candidate data block. Finally, based on the address indication information and the preset data block length, the starting address can be determined, and based on this, the write operation for the metadata that has not been written to disk is performed. Under such a mechanism, the problem of metadata loss after a processing node failure can be avoided without real-time metadata synchronization, which can also avoid the problem of excessive resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A schematic diagram of the architecture of a storage system provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a metadata storage method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a direction for writing a data block provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the storage location relationship between entity data and metadata provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the data structure of entity data provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of a metadata storage device provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0030] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0031] The metadata storage method provided in this application can be implemented by a storage system, which can include multiple processing nodes and storage pools. Each processing node establishes a communication connection with the storage pool, for example, Figure 1 The first processing node, the second processing node, ... in the storage pool may include multiple storage volumes, for example, Figure 1Storage volume 1, storage volume 2, ..., storage volume N in the storage pool. Each processing node is used to manage one or more storage volumes in the storage pool. Each storage volume can include at least one data block (Block), which can be used to store entity data, metadata (Meta), pointers or indexes to other data blocks (the data block storing this data can be called Master Index Block), segment index information (the data block storing this data can be called Segment Index Block), etc. Among them, the entity data contains business-related data, which can also be called user data (User Data), master data, etc. Metadata provides data that describes the entity data, such as the structure, source, size, physical location, logical location, etc. of the data. Each storage volume can include at least one data block (Block), which can be used to store entity data, metadata, pointers or indexes to other data blocks (the data block storing this data can be called Master Index Block), segment index information (the data block storing this data can be called Segment Index Block), etc. In addition, each processing node can serve as a backup node corresponding to one or more other processing nodes in the storage system. For example, in Figure 1 In the example, the first processing node can serve as a backup node for the second processing node, and is configured to take over the services of the storage volume managed by the second processing node when the second processing node fails.

[0032] The above storage system may be a multi-controller storage system (MCSS), wherein each processing node may be a controller in the multi-controller storage system.

[0033] The embodiment of the present application provides a metadata storage method, which can be executed by a first processing node in a storage system, such as Figure 2 As shown, the specific processing steps of the metadata storage method may include:

[0034] Step S201: When fault information corresponding to the second processing node is obtained, address indication information is extracted from the fault information.

[0035] Among them, the fault information may include the identification information of the second processing node, the identification information of the storage volume managed by the second processing node, and the address indication information. The address indication information may include the address indication information corresponding to the data block of the latest stored data in each storage volume managed by the second processing node, for example, it may include the starting address of the data block, the data block number, etc. For example, the storage volume can be divided into one or more data blocks according to the preset data block size, and a number is assigned to each data block in an incremental manner. Physical data and metadata can be written to high-numbered areas in an append-only manner to avoid performance loss problems caused by random I / O. Figure 3 As shown, the direction of writing data is from low-numbered data blocks to high-numbered data blocks.

[0036] Specifically, the first processing node, acting as a backup node for the second processing node, can periodically monitor whether the second processing node has experienced a failure. While performing related business operations on the storage volume it manages, the second processing node can update the target pointer (i.e., the address indication information corresponding to the data block where the last data was stored) each time it completes a data write operation. Furthermore, when sending failure information to the first processing node, the updated address indication information is sent to the second processing node. In this way, after a failure of the second processing node occurs, the first processing node can determine the location of the most recently stored data in the storage volume managed by the second processing node for subsequent processing.

[0037] Step S202 : determining whether the second processing node has metadata that has not been written to disk based on the address indication information and the data stored in the data blocks included in the storage volume managed by the second processing node.

[0038] Specifically, the second processing node may fail after storing the physical data, or fail after storing the metadata. If it is the former, it means that the metadata has not yet been written to the storage volume it manages and needs to be restored before it can be written to the disk. If it is the latter, it means that the metadata has been successfully written to the storage volume it manages and does not need to be restored. Therefore, after determining that the second processing node has failed, the first processing node can take over the storage volume managed by the second processing node and perform corresponding operations under different circumstances. Accordingly, after the first processing node obtains the address indication information, it can perform the following specific steps, including:

[0039] Step 1: According to the address indication information, a data block corresponding to the address indication information is determined from the data blocks included in the storage volume.

[0040] Step 2: Read the target data type from the data block corresponding to the address indication information.

[0041] Step three: when it is determined that the target data type is an entity data type, it is determined that the second processing node has metadata that has not been written to the disk.

[0042] or,

[0043] Step 4: When it is determined that the target data type is not an entity data type, it is determined that the second processing node does not have any metadata that has not been written to the disk.

[0044] Specifically, when the second processing node writes metadata to the storage volume it manages, it can do so in batches. That is, each time the second processing node generates metadata, it first caches it in memory. Whenever it detects that the number of data blocks corresponding to the metadata cached in memory equals a preset number of data blocks, it stores the metadata cached in memory as a batch of data in the storage volume. Furthermore, during data storage, a data type can be added to a designated field in the data to indicate the type of data stored in the data block.

[0045] Under such a disk-storage mechanism, the first processing node can determine the storage volume managed by the second processing node in the storage pool based on the identification information of the storage volume in the fault information, or can obtain the identification information of the storage volume managed by the second processing node based on the identification information of the second processing node in the fault information, and determine the storage volume managed by the second processing node in the storage pool based on the identification information of the storage volume managed by the second processing node. Then, the first processing node can select the data block corresponding to the address indication information from the data blocks included in the storage volume managed by the second processing node based on the address indication information, that is, the data block with the latest stored data. Then, the first processing node can read the target data type from the data block. If the target data type is an entity data type, it means that the second processing node failed after the entity data was stored on the disk, that is, the second processing node has metadata that has not been stored on the disk. If the target data type is not an entity data type, it means that the second processing node failed after the metadata was stored on the disk, that is, there is no metadata that has not been stored on the disk. Alternatively, it is also possible to determine whether the target data type is a metadata type to determine whether there is metadata that has not been written to the disk, that is, when it is determined that the data type is a metadata type, it is determined that there is metadata that has not been written to the disk on the second processing node, or when it is determined that the data type is not a metadata type, it is determined that there is metadata that has not been written to the disk on the second processing node.

[0046] By adding a data type to a designated field within each data block (physical data or metadata), the data type can be retrieved from the most recently stored data block when determining whether any metadata has been flushed to disk. This allows the location of the most recently stored data block to be determined by simply extracting the data type stored in that block, eliminating the need to traverse all data blocks. This saves resources and improves metadata recovery efficiency.

[0047] Step S203: When it is determined that the second processing node has metadata that has not been written to disk, candidate data blocks containing physical data corresponding to the metadata that has not been written to disk are determined from the data blocks included in the storage volume based on the address indication information and the preset number of data blocks.

[0048] Among them, the range of the preset number of data blocks can be between 50 and 150. The preset number of data blocks can be fixed, or it can be updated in real time according to the load of the second processing node, that is, the second processing node can update the preset number of data blocks corresponding to itself according to the real-time load such as the utilization of its own central processing unit and the network bandwidth. Accordingly, after each adjustment is completed, the adjusted preset number of data blocks can be synchronized to the first processing node, so that the first processing node can use the accurate preset number of data blocks to determine the candidate data blocks when a failure occurs in the second processing node. For example, determine the utilization range where the utilization rate is located and the network bandwidth range where the network bandwidth is located, and obtain the first number corresponding to the utilization range and the second number corresponding to the network bandwidth range, and then use the average of the first number and the second number as the preset number of data blocks. In this way, different preset numbers of data blocks can be used for business operations according to the real-time load, which is more flexible.

[0049] Specifically, before executing the operation of writing entity data to disk, the second processing node can add metadata to a specified field of the entity data and then write it to disk. In this way, if there is metadata that has not been written to disk at the second processing node, the metadata can be recovered from the written entity data and then written to disk. Accordingly, the first processing node can first select candidate data blocks from the data blocks included in the storage volume managed by the second processing node in the following multiple ways.

[0050] Method 1: the address indication information includes the data block number.

[0051] Step 1: Determine candidate data block numbers that are equal in number to the preset number of data blocks based on the address indication information and the preset number of data blocks.

[0052] The preset number of data blocks may be used to indicate the maximum number of data blocks corresponding to metadata written to disk in a batch.

[0053] Step 2: Determine a data block corresponding to each candidate data block number among the data blocks included in the storage volume as a candidate data block.

[0054] Specifically, when the address indication information includes a data block number, the first processing node may use the data block number as a starting point and count forward a preset number of data blocks to obtain a number of candidate data block numbers (including address indication information) equal to the preset number of data blocks. In this way, the first processing node may determine, from the data blocks included in the storage volume, a data block corresponding to each candidate data block number as a candidate data block.

[0055] Because the number of metadata blocks flushed to disk in a batch is fixed, address indication information, simply indicating the block number of the most recently stored data on the second processing node, can efficiently determine the candidate blocks containing the physical data corresponding to all metadata not yet flushed to disk. Furthermore, address indication information only includes a single block number, which reduces communication resource usage and minimizes the impact on other read and write services.

[0056] In the second method, the address indication information includes the starting address of the data block.

[0057] Step 1: Determine candidate data block numbers that are equal in number to the preset number of data blocks based on the address indication information and the preset number of data blocks.

[0058] Step 2: Determine a data block corresponding to each candidate data block number among the data blocks included in the storage volume as a candidate data block.

[0059] Specifically, when the address indication information includes the starting address of a data block, the first processing node can use the starting address of the data block as a starting point and a preset data block length as a step length to determine a number of candidate starting addresses (including address indication information) equal to the preset number of data blocks. For example, the starting address in the address indication information is subtracted from the preset data block length to obtain a new starting address. The new starting address is then subtracted from the preset data block length to obtain another new starting address, and so on. After obtaining candidate starting addresses equal to the preset number of data blocks, the operation is stopped. In this way, the first processing node can determine, from the data blocks included in the storage volume, the data blocks corresponding to each candidate starting address as candidate data blocks.

[0060] Because the number of metadata blocks flushed to disk in a batch is fixed, address indication information, indicating only the starting address of the block of the most recently stored data on the second processing node, can efficiently determine the candidate data blocks containing the physical data corresponding to all metadata not yet flushed to disk. Furthermore, address indication information only includes a single starting address, which reduces communication resource usage and reduces the impact on other read and write services.

[0061] Step S204: extract metadata that has not been written to disk from the physical data stored in the candidate data block.

[0062] Among them, the entity data stored in each candidate data block may include total header data and at least one data group, each data group may include sub-header data corresponding to the data group and at least one sub-entity data, as well as log information corresponding to at least one sub-entity data.

[0063] Specifically, when the second processing node writes entity data to the disk, it can nest metadata into the entity data. Accordingly, when extracting metadata, it can perform the following steps:

[0064] Step 1: extract the first data block number and the first segment identifier from the total header data of the first candidate data block.

[0065] The first candidate data block is any candidate data block.

[0066] Step 2: extracting the total length of the sub-entity data included in each data group from the sub-header data corresponding to each data group included in the first candidate data block.

[0067] Step three: extracting the sub-length corresponding to each sub-entity data and the position information of each sub-entity data in the data segment corresponding to the first segment identifier from the log information corresponding to each sub-entity data included in the first candidate data block.

[0068] Among them, the extracted first data block number, the first segment identifier, the total length of the sub-entity data included in each data group, the sub-length corresponding to each sub-entity data, and the position information of each sub-entity data in the data segment corresponding to the first segment identifier constitute the metadata corresponding to the first candidate data block that has not been written to the disk.

[0069] Step 4: After the extraction operations corresponding to all candidate data blocks are completed, the metadata corresponding to all candidate data blocks that have not been written to the disk are obtained.

[0070] By nesting metadata within entity data and recording detailed log information (such as sub-entity length and location information) at the sub-entity data level, we ensure that metadata that has not been written to disk can be accurately recovered even after a processing node failure. Furthermore, through structured storage (total header data, sub-header data, sub-entity data, and their corresponding log information), metadata management and retrieval processes are simplified. This organizational approach facilitates rapid location and retrieval of required information, improving operational efficiency. Furthermore, by centrally managing metadata and closely integrating it with entity data, additional storage overhead is reduced. This approach also avoids the inconvenience of searching and potential data inconsistencies caused by decentralized metadata storage.

[0071] Step S205 : storing the metadata that has not been written to the disk into the storage volume according to the address indication information and the preset data block length.

[0072] Step 1: Determine a first starting address according to the address indication information and a preset data block length.

[0073] For example, the preset data block length may be 64 MB.

[0074] Step 2: Using the first starting address as the starting address, the metadata that has not been written to the disk is stored in the storage volume.

[0075] Specifically, when the address indication information includes a data block number, the first processing node adds one to the data block number and multiplies the result by the preset data block length to obtain the first starting address. Alternatively, when the address indication information includes a starting address, the first starting address may be obtained by adding the preset data block length to the starting address included in the address indication information. The first processing node may use the first starting address as the starting address to store the metadata that has not been flushed to the disk in the storage volume.

[0076] In some optional implementations, when it is determined that the second processing node does not have any metadata that has not been flushed to disk, the first processing node may extract all the metadata that has been flushed to disk from the storage volume managed by the second processing node for use in subsequent related business operations. Accordingly, the specific steps of extracting the metadata that has been flushed to disk may include:

[0077] Step 1: read a second starting address from a first data block among the data blocks included in the storage volume.

[0078] Step 2: Read the metadata that has been written to the disk from the data blocks included in the storage volume according to the second starting address and the preset data block length.

[0079] Step 1: According to the second starting address and the preset data block length, determine a data block whose stored data type is a metadata type from among the data blocks included in the storage volume.

[0080] Step 2: Read the metadata that has been written to the disk from the metadata type data block.

[0081] Step 3: Load the metadata that has been written to the disk into the memory of the first processing node.

[0082] Specifically, the first data block in the storage volume managed by the second processing node can be used to store the starting address of the first metadata data block, that is, the second starting address. The starting address of the next metadata data block can also be stored in each metadata type data block. In this way, the first processing node can first read the second starting address from the first data block, and read the metadata that has been written to the disk and the starting address of the next metadata data block from the data stored in the data block corresponding to the second starting address. Furthermore, based on the starting address of the next metadata data block, the next metadata is read, and so on. All metadata that has been written to the disk can be read and loaded into the memory of the first processing node. In this way, the first processing node can perform relevant business operations on the storage volume managed by the second processing node based on the metadata loaded into its own memory.

[0083] In some optional embodiments, when the number of preset data blocks is fixed, the first data block in the storage volume managed by the second processing node can be used to store the starting address of the data block containing the first metadata, i.e., the second starting address. Furthermore, the data block containing the last metadata block in each batch can store the data block address of the first metadata block in the next batch. Accordingly, when loading the metadata that has been flushed to disk, the first processing node can first read the second starting address from the first data block and, starting from the second starting address, read the metadata stored in a preset number of data blocks, thereby reading the metadata flushed to disk in the first batch. Furthermore, the first processing node can read the third starting address from the last data block in the first batch. Then, starting from the third starting address, it can read metadata for a preset number of data blocks, thereby reading the metadata flushed to disk in the second batch. Similarly, if the starting address is not read from the last data block in any batch, it is determined that all flushed metadata has been read. In this way, the first processing node can load all flushed metadata into its own memory and, based on this metadata, perform relevant business operations on the storage volume managed by the second processing node.

[0084] Because the number of data blocks corresponding to each batch of metadata is fixed, after each batch is flushed, the second processing node can store the starting address of the first data block of the current batch into the last data block of the previous batch. This design saves storage resources and increases efficiency by requiring only one start address to load each batch of metadata.

[0085] In some optional implementations, whenever it is monitored that the number of data blocks corresponding to the newly generated metadata corresponding to the second processing node in the memory is equal to the preset number of data blocks, the newly generated metadata is stored as a batch of data in the storage volume.

[0086] Specifically, for a storage volume managed by a second processing node, after generating entity data and its corresponding metadata corresponding to the second processing node, the first processing node can first write the entity data to disk and cache the metadata in its own memory. In this way, whenever the first processing node detects that the number of newly generated metadata blocks corresponding to the second processing node in its memory equals a preset number of data blocks, it stores the newly generated metadata cached in its memory as a batch of data in the storage volume.

[0087] If each metadata is generated by first generating a pre-write log and then flushing it to disk, frequent disk I / O operations will significantly increase latency and sacrifice throughput, for example, by over 30%. These drawbacks are particularly pronounced in high-concurrency scenarios. This solution, however, avoids these issues by flushing metadata in batches.

[0088] In some optional implementations, after the first processing node receives the second processing node's fault recovery information, it can first store the currently generated metadata in the storage volume managed by the second processing node, and then transfer management rights to the second processing node. In this way, the second processing node can load all the metadata stored in disk into its own memory in a manner similar to the first processing node's loading of metadata stored in disk, so that it can subsequently perform relevant business operations on the storage volume managed by it based on this metadata.

[0089] In the metadata storage method of an embodiment of the present application, each processing node can be provided with a corresponding backup node, and metadata can be added to a designated field of the entity data. Thus, when a processing node fails (the failed node, also known as the second processing node), the backup node (also known as the first processing node) can determine whether the failed node has completed metadata flushing at the time of the failure based on the address indication information carried in the failure information and the data stored in the data blocks included in the storage volume managed by the failed node, that is, determine whether there is metadata that has not been flushed to disk. If so, the backup node can select a candidate data block from the storage volume managed by the failed node. The candidate data block is the data block containing the entity data corresponding to the metadata that has not been flushed to disk. Since the designated field of the entity data has been added with the corresponding metadata, the metadata that has not been flushed to disk can be extracted from the data stored in the candidate data block. Finally, based on the address indication information and the preset data block length, the starting address can be determined, and based on this, the flushing operation is performed on the metadata that has not been flushed to disk. Under such a mechanism, the problem of metadata loss after a processing node failure can be avoided without real-time metadata synchronization, thus avoiding the problem of excessive resource waste.

[0090] Furthermore, related technologies typically divide storage space into different areas, including one for physical data and one for metadata. This requires pre-reserving physical space or dynamically balancing the two, making the design complex and prone to design omissions. However, this solution stores physical data and metadata together, eliminating the need for separate storage areas. Furthermore, the corresponding metadata is stored in a preset number of data blocks following the physical data, making management easier.

[0091] The data structure of the above entity data is described in detail below using a specific example.

[0092] like Figure 4 As shown in the figure, in the logical space, business data can be divided into multiple entity data. In the physical space, data blocks are used to store entity data or metadata.

[0093] The data structure of the entity number can be as follows Figure 5 As shown in the figure, the entity data includes the total header data (BlockHeader) and three data groups (Group). The total header data includes the data type (also known as the data block type, BlockType), address indication information (Block ID), and segment identification (Segment ID). For example, the length of the total header data can be 13B, of which the length of the data type can be 1B, the length of the address indication information can be 8B, and the length of the segment identification can be 4B.

[0094] Taking the first data group as an example, the data group includes sub-header data (Padding Header), three sub-entity data and padding fields (Padding Data), as well as log information (LogHeader) corresponding to each sub-entity data. For example, the sub-header data can be 20 bytes long, the log information can be 14 bytes long, and the padding fields are used to align the data group length to 512 bytes.

[0095] The sub-header data space includes the Compressed Length and Reserved fields. The Compressed Length is the total length of the sub-entity data included in each of the three data groups.

[0096] Taking the third log information as an example, the log information may include the log type (Log Type), log version (Version), the length of the sub-entity data when it is not compressed, the compression mode (Compressed Mode), the position information (Position) of the sub-entity data in the data segment corresponding to the segment identifier, the length of the sub-entity data after compression, verification information, etc. Among them, the log type can be used to indicate the type of sub-entity data, including one of normal storage, storage of all-zero data, invalid data, etc. The verification information can be a hash fingerprint or cyclic redundancy check (CRC) information, for example, its length can be 32 bits. The length of the sub-entity data when it is not compressed and the length of the sub-entity data after compression constitute the sub-length corresponding to the sub-entity data.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0098] The embodiment of the present application also provides a metadata storage device, such as Figure 6 Shown, including:

[0099] an extraction module 610 for extracting address indication information from the fault information when fault information corresponding to the second processing node is obtained, wherein the first processing node is a backup node corresponding to the second processing node;

[0100] Determining module 620 is configured to determine whether the second processing node has metadata that has not been flushed to disk based on the address indication information and the data stored in the data blocks included in the storage volume managed by the second processing node; and when it is determined that the second processing node has metadata that has not been flushed to disk, determine, based on the address indication information and a preset number of data blocks, candidate data blocks containing entity data corresponding to the metadata that has not been flushed to disk from the data blocks included in the storage volume;

[0101] The extraction module 610 is further configured to extract metadata that has not been written to disk from the data stored in the candidate data block;

[0102] The storage module 630 is further configured to store metadata that has not been written to the disk into the storage volume according to the address indication information and the preset data block length.

[0103] In some optional embodiments, the apparatus further includes a reading module 640 and a loading module 650;

[0104] The reading module 640 is configured to, when determining that no metadata that has not been flushed to disk exists on the second processing node, read a second starting address from a first data block in the data blocks included in the storage volume; and read the flushed metadata from the data blocks included in the storage volume based on the second starting address and a preset data block length.

[0105] The loading module 650 is configured to load the metadata written to the disk into the memory of the first processing node.

[0106] In some optional implementations, the reading module 640 is specifically configured to:

[0107] According to the second starting address and the preset data block length, determining a data block whose stored data type is a metadata type from the data blocks included in the storage volume;

[0108] Read the metadata that has been flushed to disk from the metadata type data block.

[0109] In some optional implementations, the determination module 620 is specifically configured to:

[0110] According to the address indication information, determining a data block corresponding to the address indication information among the data blocks included in the storage volume;

[0111] Reading a target data type from a data block corresponding to the address indication information;

[0112] When it is determined that the target data type is an entity data type, determining that the second processing node has metadata that has not been written to the disk;

[0113] or,

[0114] When it is determined that the target data type is not the entity data type, it is determined that the second processing node does not have metadata that has not been written to the disk.

[0115] In some optional implementations, the address indication information includes a data block number;

[0116] The determination module 620 is specifically configured to:

[0117] Determine, based on the address indication information and the preset number of data blocks, a number of candidate data block numbers that is equal to the preset number of data blocks;

[0118] A data block corresponding to each candidate data block number among the data blocks included in the storage volume is determined as a candidate data block.

[0119] In some optional implementations, the entity data stored in each candidate data block includes total header data and at least one data group, wherein each data group includes sub-header data corresponding to the data group and at least one sub-entity data, and log information corresponding to the at least one sub-entity data;

[0120] The extraction module 610 is specifically configured to:

[0121] extracting a first data block number and a first segment identifier from total header data of a first candidate data block, wherein the first candidate data block is any candidate data block;

[0122] Extracting the total length of the sub-entity data included in each data group from the sub-header data corresponding to each data group included in the first candidate data block;

[0123] Extracting, from the log information corresponding to each sub-entity data included in the first candidate data block, the sub-length corresponding to each sub-entity data and the position information of each sub-entity data in the data segment corresponding to the first segment identifier;

[0124] The extracted first data block number, the first segment identifier, the total length of the sub-entity data included in each data group, the sub-length corresponding to each sub-entity data, and the position information of each sub-entity data in the data segment corresponding to the first segment identifier constitute the metadata corresponding to the first candidate data block that has not been written to the disk;

[0125] After the extraction operations corresponding to all candidate data blocks are completed, metadata corresponding to all candidate data blocks that have not been written to the disk are obtained.

[0126] In some optional implementations, the storage module 630 is further configured to:

[0127] Whenever it is detected that the number of data blocks of the newly generated metadata corresponding to the second processing node in the memory is equal to the preset number of data blocks, the newly generated metadata is stored as a batch of data in the storage volume.

[0128] For the description of the features in the embodiment corresponding to the metadata storage device, reference can be made to the relevant description of the embodiment corresponding to the metadata storage method, which will not be repeated here.

[0129] The embodiment of the present application also provides an electronic device, such as Figure 7 As shown, it includes a memory 10 and a processor 20. The memory 10 stores a computer program, and the processor 20 is configured to run the computer program to perform the steps in any of the above metadata storage method embodiments. The electronic device can be a processing node in the above storage system.

[0130] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned metadata storage method embodiments when running.

[0131] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0132] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned metadata storage method embodiments are implemented.

[0133] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned metadata storage method embodiments are implemented.

[0134] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The above describes in detail the metadata storage method, device, electronic device, storage medium, and program product provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. It should be noted that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A metadata storage method, characterized in that: The method is applied to a storage system and is executed by a first processing node in the storage system. The method includes: When fault information corresponding to a second processing node is obtained, extracting address indication information from the fault information, wherein the first processing node is a backup node corresponding to the second processing node; determining whether the second processing node has metadata that has not been written to disk, based on the address indication information and data stored in a data block included in the storage volume managed by the second processing node; When it is determined that the second processing node has the metadata that has not been written to the disk, determining, from the data blocks included in the storage volume, candidate data blocks where the entity data corresponding to the metadata that has not been written to the disk is located, based on the address indication information and the preset number of data blocks; Extracting the metadata that has not been written to the disk from the entity data stored in the candidate data block; According to the address indication information and the preset data block length, the metadata that has not been written to the disk is stored in the storage volume.

2. The metadata storage method according to claim 1, wherein: The method further comprises: When it is determined that the second processing node does not have the metadata that has not been written to the disk, read a second starting address from a first data block among the data blocks included in the storage volume; Reading the metadata that has been written to the disk from the data blocks included in the storage volume according to the second starting address and the preset data block length; The metadata that has been written to the disk is loaded into the memory of the first processing node.

3. The metadata storage method according to claim 2, wherein: The step of reading the metadata written to the disk from the data blocks included in the storage volume according to the second starting address and the preset data block length includes: Determining, according to the second starting address and the preset data block length, a data block whose stored data type is a metadata type from among the data blocks included in the storage volume; The metadata that has been written to the disk is read from a data block of the metadata type.

4. The metadata storage method according to any one of claims 1 to 3, characterized in that: The determining, based on the address indication information and the data stored in the data blocks included in the storage volume managed by the second processing node, whether the second processing node has metadata that has not been written to the disk includes: determining, according to the address indication information, a data block corresponding to the address indication information among the data blocks included in the storage volume; Reading a target data type from a data block corresponding to the address indication information; When it is determined that the target data type is an entity data type, determining that the second processing node has metadata that has not been written to the disk; or, When it is determined that the target data type is not the entity data type, it is determined that the second processing node does not have any metadata that has not been written to the disk.

5. The metadata storage method according to any one of claims 1 to 3, characterized in that: The address indication information includes a data block number; when it is determined that the second processing node has the metadata that has not been written to the disk, determining, based on the address indication information and a preset number of data blocks, a candidate data block containing the physical data corresponding to the metadata that has not been written to the disk from the data blocks included in the storage volume, including: Determining, according to the address indication information and the preset number of data blocks, a number of candidate data block numbers equal to the preset number of data blocks; A data block corresponding to each candidate data block number among the data blocks included in the storage volume is determined as the candidate data block.

6. The metadata storage method according to any one of claims 1 to 3, characterized in that: The entity data stored in each candidate data block includes total header data and at least one data group, wherein each data group includes sub-header data and at least one sub-entity data corresponding to the data group, and log information corresponding to at least one sub-entity data; extracting the metadata not written to disk from the entity data stored in the candidate data block includes: Extracting a first data block number and a first segment identifier from total header data of a first candidate data block, wherein the first candidate data block is any candidate data block; Extracting the total length of the sub-entity data included in each of the data groups from the sub-header data corresponding to each of the data groups included in the first candidate data block; Extracting, from the log information corresponding to each sub-entity data included in the first candidate data block, the sub-length corresponding to each sub-entity data and the position information of each sub-entity data in the data segment corresponding to the first segment identifier; The extracted first data block number, the first segment identifier, the total length of the sub-entity data included in each data group, the sub-length corresponding to each sub-entity data, and the position information of each sub-entity data in the data segment corresponding to the first segment identifier constitute the metadata corresponding to the first candidate data block that has not been written to the disk; After the extraction operations corresponding to all candidate data blocks are completed, metadata corresponding to all candidate data blocks that have not been written to the disk are obtained.

7. The metadata storage method according to claim 2 or 3, characterized in that: The method further comprises: Whenever it is monitored that the number of data blocks of the newly generated metadata corresponding to the second processing node in the memory is equal to the preset number of data blocks, the newly generated metadata is stored as a batch of data in the storage volume.

8. A metadata storage device, characterized in that: The device is applied to a first processing node in a storage system, and includes: an extraction module configured to extract address indication information from fault information corresponding to a second processing node when the fault information corresponding to the second processing node is acquired, wherein the first processing node is a standby node corresponding to the second processing node; a determination module configured to determine, based on the address indication information and data stored in data blocks included in the storage volume managed by the second processing node, whether the second processing node has metadata that has not been written to disk; and, when it is determined that the second processing node has metadata that has not been written to disk, determine, based on the address indication information and a preset number of data blocks, candidate data blocks in the data blocks included in the storage volume where entity data corresponding to the metadata that has not been written to disk is located; The extraction module is further configured to extract the metadata not yet written to the disk from the data stored in the candidate data block; The storage module is further configured to store the metadata that has not been written to the disk into the storage volume according to the address indication information and a preset data block length.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the metadata storage method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the metadata storage method according to any one of claims 1 to 7 are implemented.

Images