Metadata management method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a metadata management method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a target object, and generating target metadata of the target object; performing field splitting on the target metadata to obtain each metadata field; determining a target compression algorithm corresponding to the metadata field according to the field feature of the metadata field; compressing the corresponding metadata field by using the target compression algorithm to obtain a field compression result; and generating a metadata compression result of the target metadata according to the field compression result of each metadata field. By applying the technical scheme provided by the invention, the compression rate can be effectively improved and the CPU occupation can be reduced in the metadata compression process of the data object, and the IO delay of a storage system is further reduced.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a metadata management method, as well as a metadata management device, electronic device, and computer-readable storage medium. Background Technology

[0002] Distributed storage systems are distributed hierarchical architectures that provide various storage services, including block, file, and object storage. To accelerate access performance for small files / objects, they support writing small files / objects to a high-performance cache layer and then returning the data via I / O; when the cache layer reaches a threshold, the data is flushed back to the persistent storage layer.

[0003] When data is flushed, it passes through the intelligent append layer (AppendOnly), which sorts and aggregates small I / O operations. During this process, the metadata manifest mapping from the data objects generated by AppendOnly to the data storage layer needs to be stored in the distributed key-value engine. However, due to the large size of the metadata manifest, it is usually compressed before being stored in the distributed key-value engine. However, currently used metadata manifest compression methods suffer from low compression ratios and high CPU usage, both of which increase the I / O latency of the distributed storage system.

[0004] Therefore, how to effectively improve the compression ratio and reduce CPU usage during the data object metadata compression process, thereby reducing the IO latency of the storage system, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a metadata management method that can effectively improve the compression ratio and reduce CPU usage during the data object metadata compression process, and further reduce the IO latency of the storage system. Another purpose of this application is to provide a metadata management device, electronic device, and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0006] Firstly, this application provides a metadata management method, including:

[0007] Obtain the target object and generate the target metadata of the target object;

[0008] The target metadata is split into fields to obtain the metadata fields;

[0009] The target compression algorithm corresponding to the metadata field is determined based on the field characteristics of the metadata field;

[0010] The target compression algorithm is used to compress the corresponding metadata fields to obtain the field compression results;

[0011] The metadata compression result of the target metadata is generated based on the field compression result of each of the metadata fields.

[0012] Preferably, determining the target compression algorithm corresponding to the metadata field based on the field characteristics of the metadata field includes:

[0013] When the field feature is an integer type field, the target compression algorithm corresponding to the metadata field is determined to be the Varint compression algorithm;

[0014] When the field feature is a completely duplicated field, the target compression algorithm corresponding to the metadata field is determined to be a fixed compression algorithm;

[0015] When the field feature is a prefix repetition type field, the target compression algorithm corresponding to the metadata field is determined to be a prefix compression algorithm;

[0016] When the field feature is a linearly increasing field value, the target compression algorithm corresponding to the metadata field is determined to be a linear compression algorithm.

[0017] Preferably, after generating the metadata compression result of the target metadata based on the field compression results of each of the metadata fields, the method further includes:

[0018] Write the target metadata into the metadata cache layer;

[0019] Write the compressed metadata result into the metadata storage layer;

[0020] The target object is read using the metadata caching layer and / or the metadata storage layer.

[0021] Preferably, the metadata caching layer includes a first cache and a second cache; writing the target metadata into the metadata caching layer includes:

[0022] When there is available space in the first cache, the target metadata is written into the first cache;

[0023] When there is no available space in the first cache, determine the evictable metadata in the first cache, write the metadata compression result of the evictable metadata into the second cache, and write the target metadata into the first cache.

[0024] Preferably, writing the metadata compression result of the obsolete metadata into the second cache includes:

[0025] When there is available space in the second cache, the metadata compression result of the evictable metadata is written into the second cache;

[0026] When there is no available space in the second cache, determine the obsolete metadata compression result in the second cache, delete the obsolete metadata compression result, and write the obsolete metadata compression result into the second cache.

[0027] Preferably, reading the target object using the metadata caching layer and / or the metadata storage layer includes:

[0028] Query the metadata cache layer to determine whether the metadata cache layer contains the target metadata or metadata compression result corresponding to the target object;

[0029] If the metadata cache layer contains target metadata or metadata compression results corresponding to the target object, then the target object is read in the data storage layer using the target metadata or the metadata compression results;

[0030] If the metadata cache layer does not contain the target metadata or metadata compression result corresponding to the target object, the metadata storage layer is queried to obtain the metadata compression result corresponding to the target object, and the target object is read from the data storage layer using the metadata compression result.

[0031] Preferably, acquiring the target object includes:

[0032] Obtain the target object sent by the data caching layer;

[0033] Accordingly, after obtaining the target object, the following is also included:

[0034] The target object is written to the data storage layer to achieve the write processing of the target object.

[0035] Secondly, this application also discloses a metadata management device, comprising:

[0036] The acquisition module is used to acquire the target object and generate the target metadata of the target object;

[0037] The splitting module is used to split the target metadata into fields to obtain each metadata field;

[0038] The determination module is used to determine the target compression algorithm corresponding to the metadata field based on the field characteristics of the metadata field;

[0039] The compression module is used to compress the corresponding metadata fields using the target compression algorithm to obtain the field compression result;

[0040] The generation module is used to generate the metadata compression result of the target metadata based on the field compression result of each of the metadata fields.

[0041] Thirdly, this application also discloses an electronic device, comprising:

[0042] Memory, used to store computer programs;

[0043] A processor for implementing any of the metadata management methods described above when executing the computer program.

[0044] Fourthly, this application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the metadata management methods described above.

[0045] This application provides a metadata management method, comprising: acquiring a target object and generating target metadata of the target object; splitting the target metadata into fields to obtain each metadata field; determining a target compression algorithm corresponding to each metadata field based on the field characteristics of the metadata fields; compressing the corresponding metadata fields using the target compression algorithm to obtain field compression results; and generating a metadata compression result of the target metadata based on the field compression results of each metadata field.

[0046] As can be seen, the metadata management method provided in this application, for the target metadata of the target object, can first split the target metadata into fields to obtain individual metadata fields, then extract the field features of each metadata field, so as to select an appropriate target compression algorithm to perform compression processing on the corresponding metadata field based on the field features, and finally combine the field compression results of all metadata fields to generate the target metadata compression result. Therefore, this technical solution uses different compression algorithms for different metadata fields during the metadata compression process to ensure that each metadata field can be compressed using the most suitable compression algorithm, effectively improving the compression ratio, reducing CPU usage, and thus achieving low IO latency in the storage system.

[0047] The metadata management device, electronic device, and computer-readable storage medium provided in this application all have the aforementioned beneficial effects, which will not be elaborated further here. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings used in the description of the prior art and the embodiments of this application will be briefly introduced below. Of course, the accompanying drawings described below with respect to the embodiments of this application are only a part of the embodiments in this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and such other drawings also fall within the protection scope of this application.

[0049] Figure 1 A flowchart illustrating a metadata management method provided in this application;

[0050] Figure 2 A flowchart illustrating another metadata management method provided in this application;

[0051] Figure 3 An architecture diagram of a distributed storage system provided in this application;

[0052] Figure 4 An architecture diagram of a two-level metadata cache provided in this application;

[0053] Figure 5 This application provides a metadata compression / decompression processing architecture diagram;

[0054] Figure 6 A flowchart illustrating a data reading method provided in this application;

[0055] Figure 7 A flowchart illustrating a data writing method provided in this application;

[0056] Figure 8 A schematic diagram of the structure of a metadata management device provided in this application;

[0057] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0058] The core of this application is to provide a metadata management method that can effectively improve the compression ratio and reduce CPU usage during the data object metadata compression process, and further reduce the IO latency of the storage system. Another core aspect of this application is to provide a metadata management device, system, and computer-readable storage medium that also have the above-mentioned beneficial effects.

[0059] To provide a clearer and more complete description of the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0060] This application provides a metadata management method.

[0061] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a metadata management method provided in this application, which may include the following steps S101 to S105.

[0062] S101: Obtain the target object and generate the target metadata of the target object.

[0063] This step aims to acquire the target object and generate its metadata. It should be noted that the metadata management method provided in this application can be applied to the intelligent append-only layer in a distributed storage system. It can interact with the data caching layer, data storage layer (persistent storage layer), and metadata storage layer, receiving the target object from the data caching layer, generating the target metadata for that object, and then saving the target metadata to the metadata storage layer and the target object to the data storage layer. To ensure efficient space utilization in the metadata storage layer, the target metadata can be compressed before being saved to the metadata storage layer. For details, please refer to the description of the target metadata compression process in subsequent steps.

[0064] Furthermore, based on the aforementioned distributed storage system architecture, obtaining the target object can include: obtaining the target object issued by the data caching layer; correspondingly, after obtaining the target object, it can also include: writing the target object into the data storage layer to realize the write processing of the target object. Thus, the data storage (data writing) process of the distributed storage system is realized.

[0065] S102: Split the target metadata into fields to obtain the metadata fields.

[0066] This step aims to split the target metadata into fields to obtain the various metadata fields within the target metadata. It is understood that metadata fields can generally include, but are not limited to, the object ID field (object_id), object data block offset field (offset), object length field (length), data block index (data_block_index), timestamp (timestamp), object compression type (compress_type), etc.

[0067] S103: Determine the target compression algorithm corresponding to the metadata field based on the field characteristics of the metadata field.

[0068] This step aims to determine the appropriate target compression algorithm based on the field characteristics of the metadata fields. Specifically, after splitting the target metadata into its individual metadata fields, the field characteristics of each metadata field can be extracted. This allows for the selection of a suitable target compression algorithm for each metadata field based on its characteristics; that is, different metadata fields in the target metadata can be compressed using different target compression algorithms.

[0069] In one embodiment of this application, determining the target compression algorithm corresponding to the metadata field based on the field characteristics of the metadata field may include: when the field characteristic is an integer field, determining the target compression algorithm corresponding to the metadata field as the Varint compression algorithm; when the field characteristic is a completely repeated field, determining the target compression algorithm corresponding to the metadata field as a fixed compression algorithm; when the field characteristic is a prefix repeated field, determining the target compression algorithm corresponding to the metadata field as a prefix compression algorithm; and when the field characteristic is a field with linearly increasing field values, determining the target compression algorithm corresponding to the metadata field as a linear compression algorithm.

[0070] This application provides different target compression algorithms for different metadata fields. For integer fields, such as the data block index and timestamp, since their values ​​are purely integers, generally without repetition and with large value ranges, the Varint compression algorithm can be used, offering advantages such as high compression ratio and fast decoding. For completely repetitive fields, such as the object compression type, since their values ​​generally do not change, a fixed compression algorithm can be used to avoid low compression ratios caused by repeated compression of identical field data. For prefix-repeating fields (prefix unchanged, suffix changes regularly), such as the object ID field (object_id), which exhibits patterns like obj_202405_001 and obj_202405_002, a prefix compression algorithm can be used to avoid low compression ratios caused by repeated compression of identical prefixes. For fields with linearly increasing numerical values, such as the object data block offset field mentioned above, which exhibit a pattern of increasing offset by 4KB per data block, a linear compression algorithm can be used to effectively improve compression ratio and fault tolerance.

[0071] S104: Compress the corresponding metadata fields using the target compression algorithm to obtain the field compression results.

[0072] This step aims to compress metadata fields, that is, to use the most suitable target compression algorithm for each metadata field to compress it, and obtain the field compression results for each metadata field.

[0073] S105: Generate the metadata compression result of the target metadata based on the field compression results of each metadata field.

[0074] This step aims to generate the metadata compression result of the target metadata. Obviously, by concatenating the compression results of each metadata field (in the order of the metadata fields in the target metadata), the metadata compression result of the target metadata can be obtained.

[0075] It's important to note that traditional metadata manifest compression methods typically employ a single, general-purpose compression algorithm. During the compression process, this algorithm usually needs to repeatedly traverse the strings within the metadata to find patterns. Obviously, the more times it's traversed, the more CPU resources are consumed. Furthermore, when inputting / outputting metadata, the algorithm repeatedly copies data to memory. Since memory copying is slower than CPU processing speed, the CPU has to repeatedly wait for memory copying operations, further increasing CPU usage. Therefore, general-purpose compression algorithms typically consume a significant amount of CPU resources, affecting both metadata decompression throughput and data I / O in the storage system.

[0076] However, this technical solution employs different compression algorithms for different metadata fields during the metadata compression process. This ensures that each metadata field can be compressed using the most suitable algorithm, facilitating more efficient data pattern finding, reducing traversal times, and consequently reducing CPU usage. Simultaneously, the algorithm can directly modify or streamline the original data, reducing memory copying and effectively minimizing CPU usage. Therefore, this technical solution effectively reduces CPU usage, improves decompression throughput, and lowers storage system I / O latency.

[0077] In summary, the metadata management method provided in this application, for the target metadata of a target object, can first split the target metadata into fields to obtain individual metadata fields, then extract the field features of each metadata field, so as to select an appropriate target compression algorithm to compress the corresponding metadata field based on the field features, and finally combine the field compression results of all metadata fields to generate the target metadata compression result. Therefore, in the metadata compression process, this technical solution uses different compression algorithms for different metadata fields to achieve metadata compression, so as to ensure that each metadata field can be compressed using the most suitable compression algorithm, effectively improving the compression ratio, reducing CPU usage, and thus achieving low IO latency in the storage system.

[0078] This application provides another method for metadata management.

[0079] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a metadata management method provided in this application, which may include the following steps S201 to S208.

[0080] S201: Obtain the target object and generate the target metadata of the target object;

[0081] S202: Split the target metadata into fields to obtain the metadata fields;

[0082] S203: Determine the target compression algorithm corresponding to the metadata field based on the field characteristics of the metadata field;

[0083] S204: Compress the corresponding metadata fields using the target compression algorithm to obtain the field compression results;

[0084] S205: Generate the metadata compression result of the target metadata based on the field compression results of each metadata field;

[0085] S206: Write the target metadata to the metadata cache layer;

[0086] S207: Write the metadata compression result to the metadata storage layer;

[0087] S208: Use the metadata caching layer and / or metadata storage layer to read the target object.

[0088] To further reduce IO latency, this application proposes adding a metadata caching layer to the distributed storage system to achieve efficient storage of target metadata. Specifically, after generating the metadata compression result of the target metadata, the target metadata can be saved to the metadata caching layer, and the metadata compression result of the target metadata can be saved to the metadata storage layer. The metadata storage layer stores the full metadata compression result (the metadata compression result of the target metadata of all target objects), while the metadata caching layer stores a small amount of frequently used target metadata (target metadata regardless of frequently used target objects). Thus, during subsequent data reading in the distributed storage system, the data can be retrieved by querying the metadata caching layer and / or the metadata storage layer. In the implementation process, for the target object to be read, the system first checks whether the target object's metadata exists in the metadata caching layer. If it exists, the target object's metadata is directly obtained from the metadata caching layer for reading the target object in the data storage layer; if it does not exist, the system continues to query the metadata storage layer to obtain the target object's metadata compression result for reading the target object in the data storage layer.

[0089] In one embodiment of this application, the metadata caching layer includes a first cache and a second cache; writing the target metadata into the metadata caching layer may include: when there is available space in the first cache, writing the target metadata into the first cache; when there is no available space in the first cache, determining the metadata that can be evicted in the first cache, writing the metadata compression result of the evicted metadata into the second cache, and writing the target metadata into the first cache.

[0090] This application further proposes a metadata caching layer with two levels of caching, aiming to effectively improve the metadata cache hit rate and further reduce IO latency. Specifically, the metadata caching layer may include a first cache and a second cache. The first cache is used to store metadata, and the second cache is used to store metadata compression results. Of course, the metadata in the first cache and the metadata compression results in the second cache correspond to different data objects. Both data objects are frequently used data objects, but the usage frequency of the data objects corresponding to the metadata in the first cache is higher than that of the data objects corresponding to the metadata compression results in the second cache. The determination of evictable metadata in the first cache can be implemented using a preset cache replacement strategy, such as the LRU algorithm, Clock algorithm, LFU algorithm, etc.

[0091] The process of writing the compressed metadata results of evictable metadata into the second cache can include: when there is available space in the second cache, writing the compressed metadata results of evictable metadata into the second cache; when there is no available space in the second cache, determining the compressed metadata results of evictable metadata in the second cache, deleting the compressed metadata results of evictable metadata, and writing the compressed metadata results of evictable metadata into the second cache. Of course, determining the compressed metadata results of evictable metadata in the second cache can also be achieved using the aforementioned preset cache replacement strategy.

[0092] Based on this, the above-mentioned reading processing of the target object using the metadata caching layer and / or metadata storage layer can include: querying the metadata caching layer to determine whether the metadata caching layer contains the target metadata or metadata compression result corresponding to the target object; if the metadata caching layer contains the target metadata or metadata compression result corresponding to the target object, then using the target metadata or metadata compression result to read the target object in the data storage layer; if the metadata caching layer does not contain the target metadata or metadata compression result corresponding to the target object, then querying the metadata storage layer to obtain the metadata compression result corresponding to the target object, and using the metadata compression result to read the target object in the data storage layer.

[0093] In the specific implementation process, the first cache of the metadata cache layer can be queried to see if the target metadata corresponding to the target object exists. If it exists, the target object is read from the data storage layer using that target metadata. If it does not exist, the second cache of the metadata cache layer is queried to determine if the compressed metadata result corresponding to the target object exists there. If it exists, the compressed metadata result is used to read the target object from the data storage layer. If it does not exist, the metadata storage layer is queried to obtain the compressed metadata result corresponding to the target object, and then the target object is read from the data storage layer using that compressed metadata result. This completes the data reading process in the distributed storage system.

[0094] As can be seen, the metadata management method provided in this application proposes a metadata cache layer with two levels of cache, which is used to realize the effective storage of metadata / metadata compression results of commonly used data objects. It can effectively improve the metadata cache hit rate during data reading and further reduce the IO latency of the distributed storage system.

[0095] This application provides yet another method for metadata management.

[0096] First, please refer to Figure 3 , Figure 3 The architecture diagram of a distributed storage system provided in this application mainly includes a distributed caching layer (data caching layer), an append-only layer (including two-level caching, namely a metadata caching layer), a distributed DKV (metadata storage layer), and a data persistence layer (data storage layer).

[0097] The distributed caching layer is responsible for caching all files written by the business logic. Specifically, it aggregates and splits files into objects, and flushes objects when the file size or number of objects reaches a threshold. The ManifestMgr is responsible for managing the object's metadata. The AppendOnly layer manages the SLog (fixed-length data segment), merging the objects flushed from the distributed caching layer into the SLog. Each object is split into data blocks, and the Manifest contains the object's metadata, mainly recording the object ID, length, offset, and metadata of each internal data block. All IO operations through AppendOnly require reading the Manifest. AppendOnly can improve IO performance by using metadata compression and a two-level caching system to store the Manifest. The distributed key-value pair is responsible for persistently storing the Manifest, and its local engine can include, but is not limited to, RocksDB, LevelDB, and FoundationDB. The data persistence layer is responsible for the persistent and reliable storage of the SLog, supporting EC pools and replica pools, and can manage disks.

[0098] Based on this, refer to Figure 3 The distributed storage system architecture shown below has the following data read / write process.

[0099] 1. Data writing process:

[0100] (1) The distributed cache layer flushes objects to the AppendOnly layer;

[0101] (2) The AppendOlny layer requests and allocates SLog, splits the object into data blocks and generates the object metadata Manifest, then writes the object into SLog, and writes SLog into the data persistence layer.

[0102] (3) After the data is successfully written, the AppendOlny layer writes the Manifest to DKV and writes the Manifest to ManifestCache (two-level cache), including a first-level uncompressed cache and a second-level compressed cache. The compression algorithm can be a semantically aware metadata compression algorithm.

[0103] 2. Data Reading Process:

[0104] (1) The file read operation of the distributed cache layer reads the object from the Appendix layer;

[0105] (2) The AppendOnly layer determines the object's Manifest: First, it queries the ManifestCache. If the ManifestCache is found, it directly returns the Manifest corresponding to the object; if the ManifestCache is not found, it loads the Manifest from the DKV.

[0106] (3) The AppendOlny layer reads the object from the data persistence layer based on the Manifest and assembles it into a file for return.

[0107] For further information, please refer to [link / reference]. Figure 4 , Figure 4 The architecture diagram of a two-level metadata cache provided in this application mainly includes a semantically aware metadata compression module and a two-level cache module. It aims to store more Manifests with limited memory configuration, reduce the latency of reading Manifests, and thus improve the read, write, and delete performance (IO performance) of the Appendix layer. Specifically, the first-level cache has a smaller memory configuration and is used to store uncompressed Manifests; the second-level cache has a larger memory configuration and is used to store compressed Manifests.

[0108] Based on this, refer to Figure 4 The two-level metadata caching architecture shown below has the following metadata read / write process.

[0109] 1. Manifest writing process:

[0110] (1) The appendOnly layer updates the object metadata and writes it to the Manifest. After passing through the semantically aware metadata compression module, the compressed Manifest is persisted to the RocksDB engine of DKV. After DKV write is successful, the Manifest is inserted into the first-level cache.

[0111] (2) If the first-level cache is full, one or more existing Manifests are evicted according to the cache replacement strategy (such as LRU algorithm). The evicted Manifests are then inserted into the second-level compressed cache after passing through the semantically aware metadata compression module. If the second-level compressed cache is full, it can be discarded directly, or one or more existing compressed Manifests can be evicted according to the cache replacement strategy.

[0112] 2. Manifest reading process:

[0113] (1) The Appendix layer reads the object and queries the Manifest in the two-level cache;

[0114] (2) First, check the first-level cache. If a match is found, return the cache directly.

[0115] (3) If the first-level cache is not hit, then query the second-level compressed cache;

[0116] (4) If the secondary compressed cache is hit, the compressed Manifest is returned and decompressed by the semantically aware metadata compression module. The decompressed Manifest is returned. At the same time, it can be recorded and judged whether the number of times the Manifest is hit in the secondary compressed cache exceeds the threshold T. If it exceeds the threshold T, the Manifest is inserted into the primary cache.

[0117] (5) If the secondary compressed cache is not hit, then query the DKV;

[0118] (6) If the DKV is found, the compressed manifest is loaded and decompressed by the semantically aware metadata compression module. The decompressed manifest is returned. At the same time, the manifest can be directly inserted into the first-level cache.

[0119] (7) If DKV is not found, the query result will be returned directly.

[0120] For further information, please refer to [link / reference]. Figure 5 , Figure 5The metadata compression / decompression processing architecture diagram provided in this application mainly includes input / output buffers and a compression / decompression engine with an independent compression / decompression thread pool (to avoid affecting other IO). The input / output buffers can split the various fields of the Manifest according to the business type, and then use different compression algorithms (such as Varint / fixed compression / prefix compression / linear compression, etc.) to compress the fields. Finally, the compressed bytes are organized into a string type in sequence. The compression engine has an independent compression / decompression thread pool. After processing by the compression engine, a compression algorithm tag (fixed length) can be added to the header, and the compressed fields are sequentially output and written to the Compressed buffer (output buffer, which can automatically expand).

[0121] Based on this, refer to Figure 5 The metadata compression / decompression processing architecture shown is described below, and its metadata compression / decompression processing flow is as follows.

[0122] 1. Compression Algorithm Flow (Encode):

[0123] (1) Select a semantically aware compression algorithm for each field of the metadata Manifest; pass in the Manifest to be compressed and the compressed string compress buf; add the compression algorithm tag to compress buf;

[0124] (2) Iterate through each field of the Manifest and call the corresponding compression algorithm to perform the compression process; write the compressed data into the compress buf in sequence;

[0125] (3) Determine whether all fields have been processed. If not, proceed to step (2); otherwise, proceed to step (4).

[0126] (4) Returns the compressed buf and the compressed length.

[0127] 2. Decompression Algorithm Flow (Decode):

[0128] (1) Read fixed-length tags to obtain semantically aware compression algorithms; pass in the string to be decompressed, to_decode, and Manifest;

[0129] (2) Determine if the end of to_decode has been reached;

[0130] (3) If the end has not been reached, the corresponding decompression algorithm is called for each field, the decompressed string is assigned to the Manifest, and then step (2) is entered.

[0131] (4) If the end is reached, return the Manifest and the length after decompression.

[0132] Finally, combining Figures 3 to 5 Please refer to the architecture diagram shown. Figure 6 and Figure 7 , Figure 6 This is a flowchart illustrating a data reading method provided in this application. Figure 7 The flowchart of a data writing method provided in this application is shown below, and the specific implementation process of both is as follows.

[0133] 1. Data reading process:

[0134] (1) The ManifestMgr in the distributed cache layer issues a read request to read the object corresponding to obj_id; at the same time, it loads a read-write lock;

[0135] (2) The Appendix layer first queries the ManifestCache first-level cache. If it is found, it proceeds to step (5).

[0136] (3) If the first-level cache is not hit, the second-level cache is queried; if the second-level cache is hit, the SlimManifest is decompressed into a Manifest using the semantically aware metadata compression algorithm and then proceeds to step (5).

[0137] (4) If the second-level cache is not found, query the RocksDB in DKV; if found, decompress the SlimManifest into a Manifest using the semantically aware metadata compression algorithm and proceed to step (5); if not found, set the Manifest to empty.

[0138] (5) Release the read-write lock and return the query result Manifest, so that the object can be read from the data persistence layer based on the Manifest and assembled into a file for return.

[0139] 2. Data writing process:

[0140] (1) The ManifestMgr in the distributed cache layer issues a write request and acquires a read-write lock on the object;

[0141] (2) The Appendix layer first queries the ManifestCache based on obj_id. If the ManifestCache matches the Manifest, it returns the result; if the ManifestCache does not match the Manifest, it reads the result from DKV.

[0142] (3) If the DKV hit occurs, insert the hit Manifest into the ManifestCache and return the Manifest; if the DKV miss occurs, return null.

[0143] (4) If the Manifest is not empty, then AppendixOnly copies the old Manifest, writes the new content, and generates a new Manifest; if the Manifest is empty, then generates a new Manifest directly.

[0144] (5) Compress the Manifest using a semantically aware metadata compression algorithm to generate a SlimManifest, and write the SlimManifest to DKV's RocksDB; if the DKV write fails, clean up the Manifest of obj_id in the ManifestCache and proceed to step (7).

[0145] (6) If the DKV write is successful, the Manifest is inserted into the first-level cache; if the first-level cache is full, the existing Manifest is evicted according to LRU; the evicted Manifest is processed by the semantically aware metadata compression algorithm and inserted into the second-level cache; if the second-level cache is full, it is discarded directly; at the same time, the read-write lock is removed and the write is returned as successful.

[0146] (7) Remove the read-write lock and return write failure.

[0147] As can be seen, the metadata management method provided in this application, for the target metadata of a target object, can first split the target metadata into fields to obtain individual metadata fields, then extract the field features of each metadata field, so as to select an appropriate target compression algorithm to compress the corresponding metadata field based on the field features, and finally combine the field compression results of all metadata fields to generate the metadata compression result of the target metadata. Thus, this technical solution uses different compression algorithms for different metadata fields during the metadata compression process to ensure that each metadata field can be compressed using the most suitable compression algorithm, effectively improving the compression ratio, reducing CPU usage, and thereby achieving low IO latency in the storage system. Furthermore, a metadata caching layer with two levels of caching is proposed to achieve effective storage of metadata / metadata compression results of frequently used data objects, which can effectively improve the metadata cache hit rate during data reading and further reduce the IO latency of the distributed storage system.

[0148] This application provides a metadata management device.

[0149] Please refer to Figure 8 , Figure 8 This application provides a schematic diagram of the structure of a metadata management device, which may include:

[0150] Module 1 is used to acquire the target object and generate the target metadata of the target object;

[0151] Splitting module 2 is used to split the target metadata into fields to obtain the metadata fields;

[0152] Module 3 is used to determine the target compression algorithm corresponding to the metadata field based on the field characteristics of the metadata field;

[0153] Compression module 4 is used to compress the corresponding metadata fields using the target compression algorithm to obtain the field compression result;

[0154] Module 5 is used to generate the metadata compression result of the target metadata based on the field compression result of each metadata field.

[0155] As can be seen, the metadata management device provided in this application, for the target metadata of a target object, can first split the target metadata into fields to obtain individual metadata fields, then extract the field features of each metadata field, so as to select an appropriate target compression algorithm to perform compression processing on the corresponding metadata field based on the field features, and finally combine the field compression results of all metadata fields to generate the metadata compression result of the target metadata. Therefore, this technical solution uses different compression algorithms for different metadata fields during the metadata compression process to ensure that each metadata field can be compressed using the most suitable compression algorithm, effectively improving the compression ratio, reducing CPU usage, and thus achieving low IO latency in the storage system.

[0156] In one embodiment of this application, the determination module 3 can be specifically used to determine the target compression algorithm corresponding to the metadata field as Varint compression algorithm when the field feature is an integer field; to determine the target compression algorithm corresponding to the metadata field as a fixed compression algorithm when the field feature is a completely repeated field; to determine the target compression algorithm corresponding to the metadata field as a prefix compression algorithm when the field feature is a prefix repeated field; and to determine the target compression algorithm corresponding to the metadata field as a linear compression algorithm when the field feature is a field with linearly increasing field values.

[0157] In one embodiment of this application, the metadata management device may further include a metadata storage module, which is used to write the target metadata into a metadata cache layer after generating the metadata compression result of the target metadata based on the field compression result of each metadata field; write the metadata compression result into the metadata storage layer; and perform read processing on the target object using the metadata cache layer and / or the metadata storage layer.

[0158] In one embodiment of this application, the metadata storage module can be specifically used to write the target metadata into the first cache when there is available space in the first cache; when there is no available space in the first cache, determine the metadata that can be eliminated in the first cache, write the metadata compression result of the metadata that can be eliminated into the second cache, and write the target metadata into the first cache.

[0159] In one embodiment of this application, the metadata storage module can be specifically used to write the metadata compression result of the evictable metadata into the second cache when there is available space in the second cache; when there is no available space in the second cache, determine the evictable metadata compression result in the second cache, delete the evictable metadata compression result, and write the evictable metadata compression result into the second cache.

[0160] In one embodiment of this application, the metadata storage module can be specifically used to query the metadata cache layer to determine whether the metadata cache layer contains target metadata or metadata compression results corresponding to the target object; if the metadata cache layer contains target metadata or metadata compression results corresponding to the target object, the target object is read from the data storage layer using the target metadata or metadata compression results; if the metadata cache layer does not contain target metadata or metadata compression results corresponding to the target object, the metadata storage layer is queried to obtain the metadata compression results corresponding to the target object, and the target object is read from the data storage layer using the metadata compression results.

[0161] In one embodiment of this application, the acquisition module 1 can be specifically used to acquire the target object issued by the data cache layer; correspondingly, the metadata management device can also include a data storage module, which is used to write the target object into the data storage layer after acquiring the target object, so as to realize the write processing of the target object.

[0162] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.

[0163] This application provides an electronic device.

[0164] Please refer to Figure 9 , Figure 9 This application provides a schematic diagram of the structure of an electronic device, which may include:

[0165] Memory 11 is used to store computer programs;

[0166] The processor 10 is configured to execute a computer program and implement the steps of any of the metadata management methods described above.

[0167] like Figure 9The diagram shows the structural composition of an electronic device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0168] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0169] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the metadata management method.

[0170] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0171] Obtain the target object and generate target metadata for the target object; split the target metadata into fields to obtain each metadata field; determine the target compression algorithm corresponding to the metadata field based on the field characteristics of the metadata field; compress the corresponding metadata field using the target compression algorithm to obtain the field compression result; generate the metadata compression result of the target metadata based on the field compression result of each metadata field.

[0172] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0173] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0174] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0175] Of course, it should be noted that, Figure 9 The structure shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 9 More or fewer components as shown, or combinations of certain components.

[0176] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of any of the metadata management methods described above.

[0177] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0179] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0180] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0182] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A metadata management method characterized by comprising: The method comprises: acquiring a target object and generating target metadata of the target object; performing field splitting on the target metadata to obtain each metadata field; determining a target compression algorithm corresponding to the metadata field according to a field characteristic of the metadata field; compressing the corresponding metadata field by using the target compression algorithm to obtain a field compression result; generating a metadata compression result of the target metadata according to the field compression result of each metadata field.

2. The metadata management method according to claim 1, characterized by, The method comprises: when the field characteristic is an integer type field, determining that the target compression algorithm corresponding to the metadata field is a Varint compression algorithm; when the field characteristic is a completely repeated field, determining that the target compression algorithm corresponding to the metadata field is a fixed compression algorithm; when the field characteristic is a prefix repeated field, determining that the target compression algorithm corresponding to the metadata field is a prefix compression algorithm; when the field characteristic is a field value linear growth field, determining that the target compression algorithm corresponding to the metadata field is a linear compression algorithm.

3. The metadata management method according to claim 1 or 2, characterized by, After generating the metadata compression result of the target metadata according to the field compression result of each metadata field, the method further comprises: writing the target metadata into a metadata cache layer; writing the metadata compression result into a metadata storage layer; performing read processing on the target object by using the metadata cache layer and / or the metadata storage layer.

4. The metadata management method according to claim 3, characterized by, The metadata cache layer comprises a first cache and a second cache. Writing the target metadata into the metadata cache layer comprises: when the first cache has available space, writing the target metadata into the first cache; when the first cache does not have available space, determining eliminable metadata in the first cache, writing the metadata compression result of the eliminable metadata into the second cache, and writing the target metadata into the first cache.

5. The metadata management method of claim 4, wherein, Writing the metadata compression result of the eliminable metadata into the second cache comprises: when the second cache has available space, writing the metadata compression result of the eliminable metadata into the second cache; when the second cache does not have available space, determining eliminable metadata compression result in the second cache, deleting the eliminable metadata compression result, and writing the metadata compression result of the eliminable metadata into the second cache.

6. The metadata management method of claim 4, wherein, Performing read processing on the target object by using the metadata cache layer and / or the metadata storage layer comprises: querying the metadata cache layer to determine whether the metadata cache layer has target metadata or a metadata compression result corresponding to the target object; if the metadata cache layer has target metadata or a metadata compression result corresponding to the target object, reading the target object in the data storage layer by using the target metadata or the metadata compression result. If the metadata cache layer does not exist target metadata or metadata compression result corresponding to the target object, the metadata storage layer is queried to obtain the metadata compression result corresponding to the target object, and the metadata compression result is used to read the target object from the data storage layer.

7. The metadata management method of claim 6, wherein, Obtaining a target object includes: Obtaining the target object issued by the data cache layer; Correspondingly, after obtaining the target object, it further includes: Writing the target object into the data storage layer to realize the write processing of the target object.

8. A metadata management apparatus characterized by comprising: It includes: An obtaining module is configured to obtain a target object and generate target metadata of the target object; A splitting module is configured to split the target metadata by fields to obtain each metadata field; A determining module is configured to determine a target compression algorithm corresponding to each metadata field according to field characteristics of the metadata field; A compression module is configured to compress the corresponding metadata field by using the target compression algorithm to obtain a field compression result; A generating module is configured to generate a metadata compression result of the target metadata according to the field compression result of each metadata field.

9. An electronic device, comprising: It includes: A memory is configured to store a computer program; A processor is configured to execute the computer program to realize the steps of the metadata management method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the metadata management method in any one of claims 1 to 7.

Images