Metadata index processing method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for processing metadata index, and relates to the field of distributed storage technology. The method includes: obtaining at least one metadata of a distributed storage service; storing hotspot data in at least one metadata in a metadata cache module, wherein the calling frequency of the hotspot data is higher than the preset calling frequency, and the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module; using an array storage method or a hash table storage method, the target index information of the metadata is recorded in the memory index in the metadata storage module, wherein the target index information includes the key-value pair of the metadata, the data pointing information of the key-value pair, and the object name and data offset information corresponding to the value in the key-value pair. The present application solves the technical problem of low read and write efficiency of the database in the prior art.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a metadata index processing method, device, electronic device and storage medium. Background Art

[0002] With the advent of the big data era, business growth has led to increasing user demands for high IOPS (Input / Output Operations Per Second, the number of read / write operations per second) and low-latency request access. Distributed all-flash storage was born for the "cloud" and can provide massive, highly reliable, highly scalable, and decentralized storage services. In the distributed block scenario, since low-latency request access is required and the block metadata information also needs to record the conversion information from the logical address of the storage volume to the underlying physical address, each IO (Input / Output, write / read) request must access and query the mapping relationship, so the block storage performance is highly dependent on the metadata query performance.

[0003] The existing technical implementation scheme is implemented through the LSM tree (Log-Structured Merge-tree). The core idea is to use sequential writing to improve write performance, but the layered design will reduce read performance. Because LSM has multiple layers, data may exist in each layer, and read requests need to search for the requested key from top to bottom, which may cause read amplification. LSM tree sacrifices some read performance in exchange for write performance, so LSM is more suitable for scenarios with more writes and less reads.

[0004] It can be seen from the above content that in the prior art, a high-performance database solution is still needed to support high-performance read and write requests.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The present application provides a metadata index processing method, device, electronic device and storage medium to at least solve the technical problem of low database reading and writing efficiency in the prior art.

[0007] According to one aspect of the present application, a method for processing metadata indexes is provided, which is applied to a distributed storage service, wherein the distributed storage service includes at least a metadata cache module and a metadata storage module, and the method for processing metadata indexes includes: obtaining at least one metadata of the distributed storage service; storing hot data in the at least one metadata in the metadata cache module, wherein the call frequency of the hot data is higher than the preset call frequency, and the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module; using an array storage method or a hash table storage method to record target index information of the metadata in a memory index in the metadata storage module, wherein the target index information includes a key-value pair of the metadata, data pointing information of the key-value pair, and an object name and data offset information corresponding to the value in the key-value pair; when the target query data does not exist in the metadata cache module, obtaining a memory index, and reading the target query data from the disk according to the memory index.

[0008] Optionally, in the metadata index processing method of the present application, an array storage method or a hash table storage method is used to record the target index information of the metadata in the memory index in the metadata storage module, including: detecting the byte length of the key-value pair of the metadata; when the byte length is less than a preset length, recording the target index information of the metadata in the memory index by using an array storage method; when the byte length is greater than or equal to the preset length, recording the target index information of the metadata in the memory index by using a hash table storage method.

[0009] Optionally, in the metadata index processing method of the present application, an array storage method or a hash table storage method is used to record the target index information of the metadata in the memory index in the metadata storage module, including: detecting whether the metadata is continuous integer metadata for a core path, wherein the core path is an access path with an access frequency greater than a preset frequency; in the case where the metadata is continuous integer metadata for the core path, an array storage method is used to record the target index information of the metadata in the memory index; in the case where the metadata is not continuous integer metadata for the core path, a hash table storage method is used to record the target index information of the metadata in the memory index.

[0010] Optionally, in the metadata index processing method of the present application, in the process of recording the target index information of the metadata in the memory index in the metadata storage module, the metadata index processing method also includes: setting a namespace corresponding to the metadata according to the key-value pair type of the metadata, wherein the namespace corresponding to the metadata is used to reduce the primary key prefix information of the key-value pair of the metadata.

[0011] Optionally, the metadata index processing method also includes: when recording the target index information of the metadata in the memory index using an array storage method, converting the object name of the primary key of the key-value pair into a number as the subscript of the array; when recording the target index information of the metadata in the memory index using a hash table storage method, without performing any conversion on the object name of the primary key of the key-value pair, recording the object name of the primary key of the key-value pair in the memory index, wherein the length of the memory index structure message generated by using the hash table storage method is greater than the length of the memory index structure message generated by using the array storage method.

[0012] Optionally, a batch append write method is used to write metadata in the distributed storage service and update the information in the memory index. The batch append write method of log objects is used to assign a globally unique and monotonically increasing version number to the append write request without accessing the memory index, and append the write request to the data backend of the distributed storage service. The version number is used for verification and judgment during data reconstruction.

[0013] Optionally, in the metadata index processing method of the present application, metadata is written in a distributed storage service by batch appending to log objects, including: serializing the metadata and appending it to the log object, returning an error when the sum of the current log object size and the size of the data to be written is greater than a preset threshold, and generating a new log object to rewrite the data.

[0014] Optionally, in the metadata index processing method of the present application, after setting the namespace corresponding to the metadata according to the key-value pair type of the metadata, it is detected whether the priority of the namespace corresponding to the metadata is higher than the preset priority; when the priority of the namespace corresponding to the metadata is higher than the preset priority, the metadata is stored as hot data in the metadata cache module; when the priority of the namespace corresponding to the metadata is not higher than the preset priority, the time to store the metadata in the metadata storage module is determined according to the memory usage of the distributed storage service.

[0015] Optionally, in the metadata index processing method of the present application, after recording the target index information of the metadata in the memory index in the metadata storage module, it is detected whether the ratio between the total metadata write amount and the actual write amount in the distributed storage service is greater than a preset value; when the ratio is greater than the preset value, the log objects in the distributed storage service are fully scanned to determine the log objects in the distributed storage service that have hole data, wherein the hole data is used to characterize redundant data that has been overwritten or partially deleted; when the Ith log object is a log object with hole data, and it is detected that the hole rate of the Ith log object exceeds the preset hole rate, a data hole recovery operation is performed on the Ith log object, wherein the hole rate characterizes the proportion of the hole data of the Ith log object in the total data of the Ith log object, and the data hole recovery operation is used to append the existing data to the log object and update the memory information, and after completion, the log object corresponding to the hole data in the read-only state is deleted.

[0016] Optionally, in the metadata index processing method of the present application, the log objects in the distributed storage service are fully scanned to determine the log objects with empty data in the distributed storage service, including: detecting whether the metadata is recorded in the memory index; when the metadata is not recorded in the memory index, determining that the metadata is empty data; when the metadata is recorded in the memory index, detecting whether the object version number of the log object corresponding to the metadata is lower than the version number of the metadata in the memory index; when the object version number is lower than the version number of the metadata in the memory index, determining that the metadata is empty data; when the object version number is not lower than the version number of the metadata in the memory index, determining that the metadata is not empty data; and determining the log object corresponding to the empty data as the log object with empty data in the distributed storage service.

[0017] Optionally, in the metadata index processing method of the present application, after the target index information of the metadata is recorded in the memory index in the metadata storage module, when multiple metadata are concurrently written to the same primary key, the multiple metadata are sorted according to the version number corresponding to each metadata to obtain the sorting result; according to the sorting result, the metadata with the largest version number is used as the metadata actually written to the primary key, wherein the metadata with the largest version number is the metadata that is last requested to be written to the primary key among the multiple metadata.

[0018] Optionally, in the metadata index processing method of the present application, after the target index information of the metadata is recorded in the memory index in the metadata storage module, when a node restart occurs in the distributed storage service, the loading order of the metadata is determined according to the priority of the namespace corresponding to the metadata.

[0019] Optionally, in the metadata index processing method of the present application, after the target index information of the metadata is recorded in the memory index in the metadata storage module, when a node restart occurs in the distributed storage service, the metadata in the data backend is obtained by accessing the metadata controller of the distributed storage service for reverse loading, wherein the reverse loading is used to restore the metadata with the same name with the largest version number by comparing the version numbers of the metadata.

[0020] Optionally, in the metadata index processing method of the present application, the distributed storage service includes the following components: a metadata controller, used to manage data information of all log objects written in the distributed storage service; a data backend, used to store the latest data of the distributed storage service; a block device service component, wherein the block device service component includes a block device logic module and a metadata module, and the metadata module includes a metadata cache module and a metadata storage module.

[0021] According to another aspect of the present application, a metadata index processing device is also provided, which is applied to a distributed storage service, wherein the distributed storage service includes at least a metadata cache module and a metadata storage module, and the metadata index processing device includes: an acquisition unit, used to acquire at least one metadata of the distributed storage service; a first storage unit, used to store hot data in the at least one metadata in the metadata cache module, wherein the call frequency of the hot data is higher than the preset call frequency, and the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module; a second storage unit, used to record the target index information of the metadata in a memory index in the metadata storage module by using an array storage method or a hash table storage method, wherein the target index information includes a key-value pair of the metadata, data pointing information of the key-value pair, and an object name and data offset information corresponding to the value in the key-value pair, and when the target query data does not exist in the metadata cache module, the memory index is acquired, and the target query data is read from the disk according to the memory index.

[0022] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute any one of the above-mentioned metadata index processing methods.

[0023] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes any one of the above-mentioned metadata index processing methods.

[0024] In the present application, a method for processing metadata index is provided, which is applied to a distributed storage service, wherein the distributed storage service includes at least a metadata cache module and a metadata storage module. First, at least one metadata of the distributed storage service is obtained; then, the hot data in the at least one metadata is stored in the metadata cache module, wherein the calling frequency of the hot data is higher than the preset calling frequency, and the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module; the target index information of the metadata is recorded in the memory index in the metadata storage module by using an array storage method or a hash table storage method, wherein the target index information includes the key-value pair of the metadata, the data pointing information of the key-value pair, and the object name and data offset information corresponding to the value in the key-value pair. When the target query data does not exist in the metadata cache module, the memory index is obtained, and the target query data is read from the disk according to the memory index.

[0025] From the above content, it can be seen that the present application stores hot spot data in a metadata cache module, and designs the data reading priority of the metadata cache module to be higher than the data reading priority of the metadata storage module, so that when reading metadata, it is read from the metadata cache module first. If the metadata to be read is not found in the metadata cache module, it is read from the metadata storage module, thereby reducing the number of metadata IO times (at most one IO is required for the metadata storage module), thereby improving the data access efficiency of the database.

[0026] In addition, the present application also adopts an array storage method or a hash table storage method to record the target index information of the metadata in the memory index in the metadata storage module. Compared with the hash table storage method commonly used in the prior art to store metadata, the array storage method can reduce the size of data that needs to be cached, thereby improving the utilization of database memory.

[0027] It can be seen that the technical solution of the present application achieves the purpose of realizing high-performance access to block storage metadata in a distributed all-flash storage scenario, thereby achieving the technical effect of supporting high IOPS and low-latency data access, and further solving the technical problem of low database read and write efficiency in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 is a flowchart of an optional metadata index processing method according to an embodiment of the present application;

[0030] Figure 2 is a schematic diagram of an optional distributed storage service architecture according to an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of an optional log object appending writing method according to an embodiment of the present application;

[0032] Figure 4 This is a schematic diagram of an optional IO interaction during data writing according to an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of an optional data reading process according to an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of an optional solution for simultaneously writing multiple metadata into one primary key according to an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of an optional metadata index processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0039] According to an embodiment of the present application, an embodiment of a method for processing a metadata index is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 1 is a flowchart of an optional metadata index processing method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0041] Step S101, obtaining at least one metadata of a distributed storage service.

[0042] In step S101, the distributed storage service includes at least a metadata cache module and a metadata storage module.

[0043] Optionally, a distributed storage service (DS) is a distributed storage network storage service that stores data in multiple independent devices. It is suitable for mass data storage and has the advantages of high scalability, high reliability, high performance, and high cost-effectiveness. The distributed storage service includes a metadata service (MetaService) and a storage service (Storage Service). The metadata service is responsible for managing the configuration and data of the storage service, and the storage service is responsible for managing the reading, writing, reading, and retrieval of data. The data of the distributed storage service is stored in the form of key-value pairs. A key-value pair (KV) includes a key and a value.

[0044] In an optional embodiment, the present application proposes a new block storage architecture for a distributed storage service, wherein under the block storage structure, the distributed storage service includes the following components:

[0045] A metadata controller is used to manage the data information of all log objects written in the distributed storage service; a data backend is used to store the latest data of the distributed storage service; a block device service component, wherein the block device service component includes a block device logic module and a metadata module, and the metadata module includes a metadata cache module and a metadata storage module.

[0046] Optionally, Figure 2 is a schematic diagram of an optional distributed storage service architecture according to an embodiment of the present application, such as Figure 2 As shown in the figure, the distributed storage service consists of a block device service (BlockServer), a metadata controller (Meta Controler), and a data backend (ChunkServer). The block device service consists of a block device logic module (BlockService) and a metadata module (BlockMeta), and the metadata module includes a metadata cache module (MetaCache) and a metadata storage module (XKV).

[0047] Step S102: storing hotspot data in at least one metadata in a metadata cache module.

[0048] In step S102, the calling frequency of the hotspot data is higher than the preset calling frequency, and the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module.

[0049] Optionally, the metadata cache module is used to cache hot data, that is, the metadata cache module is a module used to cache metadata with a call frequency higher than a preset call frequency, wherein the preset call frequency can be customized. In addition, a cache elimination algorithm is also deployed in the metadata cache module to filter out hot data from all metadata in the distributed storage service.

[0050] In addition, it should be noted that in the distributed storage service of the present application, the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module.

[0051] Step S103: Using an array storage method or a hash table storage method, the target index information of the metadata is recorded in a memory index in the metadata storage module.

[0052] In step S103, the target index information includes the key-value pair of metadata, data pointing information of the key-value pair, and the object name and data offset information corresponding to the value in the key-value pair.

[0053] Optionally, the metadata storage module is composed of a memory index, an index persistence module, and a metadata persistence module. The memory index is used to store the key-value pairs of metadata, the data pointing to the key-value pairs, and to record the object name and data offset information corresponding to the Value. When the target query data does not exist in the metadata cache module, the memory index is obtained, and the target query data is read from the disk according to the memory index.

[0054] It should be noted that the memory index records the information of all massive data. In this application, the memory index supports two ways of storing data: hash table and array.

[0055] Based on the contents of the above steps S101 to S103, it can be seen from the above contents that the present application stores hot spot data in a metadata cache module, and designs the data reading priority of the metadata cache module to be higher than the data reading priority of the metadata storage module, so that when reading metadata, it is read from the metadata cache module first. If the metadata to be read is not found in the metadata cache module, it is read from the metadata storage module, thereby reducing the number of IO times of the metadata (at most one IO is required for the metadata storage module), thereby improving the data access efficiency of the database.

[0056] In addition, the present application also adopts an array storage method or a hash table storage method to record the target index information of the metadata in the memory index in the metadata storage module. Compared with the hash table storage method commonly used in the prior art to store metadata, the array storage method can reduce the size of data that needs to be cached, thereby improving the utilization of database memory.

[0057] It can be seen that the technical solution of the present application achieves the purpose of realizing high-performance access to block storage metadata in a distributed all-flash storage scenario, thereby achieving the technical effect of supporting high IOPS and low-latency data access, and further solving the technical problem of low database read and write efficiency in the prior art.

[0058] In an optional embodiment, the distributed storage service can detect the byte length of the key-value pair of the metadata. When the byte length is less than a preset length, the target index information of the metadata is recorded in the memory index using an array storage method; when the byte length is greater than or equal to the preset length, the target index information of the metadata is recorded in the memory index using a hash table storage method.

[0059] Optionally, Table 1 is an example table of detailed information of an optional KEY according to an embodiment of the present application.

[0060] Table 1

[0061]

[0062] Optionally, in view of the characteristics of storage volume metadata, the present application can design a fixed length of key-value pairs, use array storage for metadata with a fixed length of 2 bytes, and use a hash table data structure to store metadata with a length of 3-10 bytes.

[0063] Optionally, in the above example, the preset length is set to 3 bytes. In actual applications, the preset length may also be other byte lengths.

[0064] In an optional embodiment, the distributed storage service may further detect whether the metadata is continuous integer metadata for a core path, wherein the core path is an access path with an access frequency greater than a preset frequency. In the case where the metadata is continuous integer metadata for the core path, an array storage method is used to record the target index information of the metadata in the memory index; in the case where the metadata is not continuous integer metadata for the core path, a hash table storage method is used to record the target index information of the metadata in the memory index.

[0065] Optionally, the memory index records information of all massive data and supports both hash table and array methods. In order to improve data access efficiency, continuous integer metadata of the core path can be preferably cached in an array manner, thereby further improving addressing and access speed.

[0066] It should be noted that, using the technical solution of the present application, when a user accesses a key, the metadata cache module (MetaCache) is first searched. If it does not exist, the memory index is accessed to find the target object name, corresponding offset and length information corresponding to the key.

[0067] In an optional embodiment, the distributed storage service may also set a namespace corresponding to the metadata according to the key-value pair type of the metadata, wherein the namespace corresponding to the metadata is used to reduce the primary key prefix information of the key-value pair of the metadata.

[0068] Optionally, in order to reduce the length of the key and reduce memory usage, this application can design namespaces for different types of key-value pairs to store them, supporting a default upper limit of 32. By reducing the prefix of the key through the namespace, the storage data can be streamlined and the memory usage of the cache can be reduced. Different types of namespaces are shown in Table 1, which contains the type information of 11 namespaces.

[0069] In an optional embodiment, when the target index information of the metadata is recorded in the memory index using an array storage method, the object name of the primary key of the key-value pair is converted into a number as the subscript of the array; when the target index information of the metadata is recorded in the memory index using a hash table storage method, the object name of the primary key of the key-value pair is recorded in the memory index without any conversion of the object name of the primary key of the key-value pair, wherein the length of the memory index structure message generated by the hash table storage method is greater than the length of the memory index structure message generated by the array storage method.

[0070] Optionally, the memory index structure is as follows:

[0071] struct key_entry_memory {

[0072] uint64_t version:57 = 0;

[0073] uint64_t ns:5 = 0;

[0074] uint64_t status:2 = 0;

[0075] uint64_t chunk_index:14 = 0;

[0076] uint64_t data_offset:28 = 0;

[0077] uint64_t value_length:22 = 0;

[0078] uint8_tkey_name[get_key_length(ns)];

[0079] };

[0080] The metadata cache structure is as follows:

[0081] struct data_entry_memroy {

[0082] uint32_t crc = 0;

[0083] uint64_t version:57 = 0;

[0084] uint32_t ns:5 = 0;

[0085] uint32_t status:2 = 0; uint32_t value_length:22 = 0;

[0086] uint8_t key[get_key_length(ns)];

[0087] uint8_t value[value_length];

[0088] };

[0089] It should be noted that the memory index structure key_entry_memory and the metadata cache structure data_entry_memory are memory structures generated based on the return result after the data is written successfully, or they are memory structures rebuilt by reading data after the node is restarted. Among them, key_entry_memory records the key version (version), namespace (namespace, abbreviated as ns), status (where 1 means existence and 2 means delete mark), large object name (chunk index), data offset (data_offset) and data length (value_length) information.

[0090] Optionally, if hash table storage is used, the object name key_name will be additionally occupied; if array storage is used, the key will be converted into a number as the array subscript.

[0091] In addition, since the key length of the same namespace is fixed, the length of the key name can be determined through the namespace. The metadata cache data records the cyclic redundancy check code (crc), version (version), namespace (namespace, referred to as ns), status, and data length (value).

[0092] For array storage, the value size of the memory index is fixed to 16 bytes after the key_entry_memory is streamlined (the value part of the memory index does not include the key_name, and its number is used as the array subscript). For example, in an extreme case, a key with a length of 2 and a value of 16 bytes in a node requires 15G to store 1 billion metadata.

[0093] For the hash table storage method, the memory needs to record the key_name, so the size of key_entry_memory is 16 bytes plus the byte length of key_name. For example, the key in the data backend is 0x12345678ABCDEFFF1234, and its length is 10. It is recorded in the hash table corresponding to the namespace, and its total size (key_entry_memory) is 26 bytes (the sum of 10 bytes of the hash table key and 16 bytes of the hash table value).

[0094] In an optional embodiment, the distributed storage service may adopt a batch append write method for log objects to write metadata in the distributed storage service and update the information in the memory index, wherein the batch append write method for log objects is used to assign a globally unique and monotonically increasing version number to the append write request without the need to access the memory index, and append the write request to the data backend of the distributed storage service, wherein the version number is used for verification and judgment during data reconstruction.

[0095] Optionally, in this application, data is written by batch appending to the log object (Active File), and the memory index information is updated at the same time. Compared with traditional databases, there is no need to write log information, which reduces the number of IO interactions (traditional databases first write logs and then update data storage, which requires at least 2 IOs. This application only needs to write logs and read data from logs). At the same time, this application also compresses the value part of the data before writing it, reducing space usage and improving writing performance.

[0096] Optionally, the structure of the persistent key in this application is as follows:

[0097] struct data_header {

[0098] uint32_t header_crc = 0;

[0099] uint64_t version:57 = 0;

[0100] uint32_t ns:5 = 0;

[0101] uint32_t status:2 = 0;

[0102] uint32_t v_length:22 = 0;

[0103] };

[0104] Optionally, the structure of the persistent value in this application is as follows:

[0105] struct data_body {

[0106] uint32_t body_crc = 0;

[0107] uint8_t key[get_key_length(ns)];

[0108] uint8_t value[v_length];

[0109] };

[0110] As can be seen from the above content, persistent metadata is divided into data_header and data_body structures, where the written part contains the description information of the key and value (data_header) and the key value (data_body), where the data_header contains the header's own crc, version number (version), namespace, status, and value length. In the data recovery process, the data_header is parsed and the key length is obtained through ns, and then the data_body is read; the data_body part records the cyclic redundancy check code (crc), the key name and value data, where the crc is used to verify data consistency. When writing, there is no need to access the memory index. The append write request is assigned a globally unique monotonically increasing version and appended to the chunkserver. The version is used for verification judgment during data reconstruction.

[0111] In an optional embodiment, the distributed storage service may serialize the metadata and append it to the log object, and return an error when the sum of the current log object size and the size of the data to be written is greater than a preset threshold, and generate a new log object to rewrite the data.

[0112] Optionally, Figure 3 is a schematic diagram of an optional log object appending method according to an embodiment of the present application, such as Figure 3 As shown in the figure, when writing data, the key value and data are serialized and directly appended to the log object. If the sum of the data size of the current log object and the size of the data to be written is greater than the preset threshold (for example, 64M), the distributed storage service will return an error and generate a new log object Active File to rewrite the data. At the same time, the current log object is marked as a read-only log object (corresponding to Figure 3 ReadOnlyFile in , referred to as ReafFile). In addition, Figure 3It is also shown that the new log object Active File includes the already written data (data) and the additionally written data (data_header and data_body).

[0113] Optionally, when the ActiveFile is full, it becomes a read-only file, and the metadata information is updated to the metadata controller to generate a new ActiveFile. The metadata controller records all read-only files and active files in the order in which they are written. The metadata controller manages all written log files, and its ActiveFile has only one file. When it is full, it will lock and create a new ActiveFile, and set the historical ActiveFile to a read-only file. When the node restarts and needs to rebuild the memory index, it needs to access the metadata controller.

[0114] Optionally, Figure 4 is an optional IO interaction diagram when writing data according to an embodiment of the present application, such as Figure 4 As shown, the caller first calls the XKV interface individually or in batches (put / batch_put), and then can implement the following operations:

[0115] Interaction with memory index (key_value location information); index persistence; XKV persistence; interaction between KV configuration information and Meta Controller.

[0116] Finally, the data is written to the ChunkServer in a sequential append-write manner, and the memory index and metadata cache module are updated after success. The read service can only be accessed after the writing process is completed. The corresponding prompt information will be returned to the caller if the data is written successfully or failed.

[0117] In an optional embodiment, the distributed storage service can detect whether the priority of the namespace corresponding to the metadata is higher than the preset priority. If the priority of the namespace corresponding to the metadata is higher than the preset priority, the metadata is stored as hot data in the metadata cache module; if the priority of the namespace corresponding to the metadata is not higher than the preset priority, the time to store the metadata in the metadata storage module is determined according to the memory usage of the distributed storage service.

[0118] Optionally, the present application can set different priorities for data in different namespaces. For high-priority data, the metadata value will be updated to the metadata cache module for access acceleration; for low-priority data, the time to store the metadata in the metadata storage module is determined based on the current memory usage, that is, when the memory is sufficient, it will be cached to the metadata storage module, otherwise it will not be cached.

[0119] Optionally, Figure 5 is a schematic diagram of an optional data reading process according to an embodiment of the present application, such as Figure 5 As shown, the following steps are included:

[0120] 1. The caller first calls the XKV interface individually or in batches (get / batch_get);

[0121] 2. Check whether the desired data exists in the metadata cache module (MetaCache, referred to as Cache). During the retrieval phase, the interaction between XKV persistence, KV configuration information and Meta Controller can continue.

[0122] 3. If the data you want to read exists in the cache, return the data directly to the caller;

[0123] 3.1. If the data you want to read does not exist in the cache, read the information about the data in the memory index. The memory index records the location information of the key-value of all data, and the memory index can be persisted. In addition, the memory index also needs to be loaded during the boot (process restart) phase of the distributed storage service.

[0124] 3.2. According to the information about the data to be read recorded in the memory index, the memory data is read in the ChunkServer and the read data is returned to the caller.

[0125] 4. Whether the data is read successfully or failed, the corresponding prompt information will be returned to the caller.

[0126] In an optional embodiment, the distributed storage service can also detect whether the ratio between the total metadata write amount and the actual write amount in the distributed storage service is greater than a preset value. In the case where the ratio is greater than the preset value, the log objects in the distributed storage service are fully scanned to determine the log objects in the distributed storage service that have hole data, wherein the hole data is used to characterize redundant data that has been overwritten or partially deleted; when the first log object is a log object with hole data, and it is detected that the hole rate of the first log object exceeds the preset hole rate, a data hole recovery operation is performed for the first log object, wherein the hole rate characterizes the proportion of the hole data of the first log object in the total data of the first log object, and the data hole recovery operation is used to append the existing data to the log object and update the memory information, and after completion, the log object corresponding to the hole data in the read-only state is deleted.

[0127] Optionally, during the data recovery process, if written data is overwritten or part of the data is deleted, storage space that needs to be recovered will be generated, that is, there will be empty data. In this case, it is necessary to perform data hole recovery operations on these empty data.

[0128] Optionally, in an embodiment of the present application, if the amplification factor satisfies a certain threshold value n (where n>1), for example, n=1.5, then when the ratio between the total metadata write amount and the actual write amount in the distributed storage service is greater than 1.5, a full scan of all log objects is started, and when the hole rate of a single log object exceeds a threshold value (for example, 50%), a data hole recovery operation is performed on the log object.

[0129] In an optional embodiment, the distributed storage service can detect whether the metadata is recorded in the memory index. When the metadata is not recorded in the memory index, the metadata is determined to be empty data; when the metadata is recorded in the memory index, it is detected whether the object version number of the log object corresponding to the metadata is lower than the version number of the metadata in the memory index; when the object version number is lower than the version number of the metadata in the memory index, the metadata is determined to be empty data; when the object version number is not lower than the version number of the metadata in the memory index, the metadata is determined not to be empty data. Finally, the log object corresponding to the empty data is determined as the log object with empty data in the distributed storage service.

[0130] Optionally, when scanning the log object, start scanning from the beginning in the order in which the ChunkServer is written. Taking scanning a single ChunkServer in a read-only state as an example, all log objects are read and parsed, and the existing data is appended to the Active File, and the memory information is updated. After completion, the read-only log is deleted and the metadata controller is updated. In addition, the data recovery process supports both automatic and manual methods, that is, the data recovery operation can be triggered not only automatically according to the set threshold, but also manually.

[0131] In an optional embodiment, when multiple metadata are concurrently written to the same primary key, the multiple metadata are sorted according to the version number corresponding to each metadata to obtain a sorting result. According to the sorting result, the metadata with the largest version number is used as the metadata actually written to the primary key, wherein the metadata with the largest version number is the metadata that is last requested to be written to the primary key among the multiple metadata.

[0132] Optionally, for the competition scenario of concurrent writing to the same key, in order to ensure the sequential execution of the IO callback function, this application solves the competition conflict problem through a monotonically increasing unique version number version. It should be noted that the version is pre-assigned before receiving the request.

[0133] Optionally, Figure 6 is a schematic diagram of an optional solution for simultaneously writing multiple metadata into a primary key according to an embodiment of the present application, such as Figure 6 As shown in the figure, after receiving the request req1 for the first time at time T1, the version number is assigned as version1, and after receiving the second request req2 at time T2, the version number is assigned as version2. Among them, version2 is greater than version1. On this basis, if req2 completes the callback first at time T3, it is judged that the version of the request is version2, which is greater than the version of the current memory index, so the memory index version is updated to version2; then, when req1 completes the callback at time T4, it is judged that the version of the request is version1, which is less than the version of the memory index version2, and the distributed storage service returns the information that the req1 callback failed.

[0134] In an optional embodiment, when a node restart occurs in the distributed storage service, the order in which metadata is loaded is determined according to the priority of the namespace corresponding to the metadata.

[0135] Optionally, since the memory index is a full index, the index needs to be fully loaded when the node is restarted. In order to restore user services as soon as possible, load the priorities of different namespaces, and load the metadata required by the business first to restore the business first.

[0136] In an optional embodiment, when a node restart occurs in a distributed storage service, the metadata in the data backend is obtained by accessing the metadata controller of the distributed storage service for reverse loading, wherein the reverse loading is used to restore the metadata with the same name with the largest version number by comparing the version numbers of the metadata.

[0137] Optionally, during the metadata loading process, you can access the metadata controller to obtain the metadata in the data backend for reverse loading, because the latest ChunkSever stores the latest data, and during the recovery process, you can recover the metadata by comparing the version number. For example, if the memory index has recovered the "0x12a2" data of the namespace l2p, when the previous ChunkServer is recovered, data with the same name appears, and the version number is smaller than the version number of the current memory index, you can skip recovering this data with the same name.

[0138] From the above content, it can be seen that the present application stores hot spot data in a metadata cache module, and designs the data reading priority of the metadata cache module to be higher than the data reading priority of the metadata storage module, so that when reading metadata, it is read from the metadata cache module first. If the metadata to be read is not found in the metadata cache module, it is read from the metadata storage module, thereby reducing the number of metadata IO times (at most one IO is required for the metadata storage module), thereby improving the data access efficiency of the database.

[0139] In addition, the present application also adopts an array storage method or a hash table storage method to record the target index information of the metadata in the memory index in the metadata storage module. Compared with the hash table storage method commonly used in the prior art to store metadata, the array storage method can reduce the size of data that needs to be cached, thereby improving the utilization of database memory.

[0140] According to another aspect of the present application, a metadata index processing device is also provided, which is applied to a distributed storage service, wherein the distributed storage service at least includes a metadata cache module and a metadata storage module, such as Figure 7 As shown, the metadata index processing device includes the following units:

[0141] An acquisition unit 701 is used to acquire at least one metadata of the distributed storage service; a first storage unit 702 is used to store hot data in the at least one metadata in the metadata cache module, wherein the calling frequency of the hot data is higher than the preset calling frequency, and the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module; a second storage unit 703 is used to record the target index information of the metadata in the memory index in the metadata storage module by using an array storage method or a hash table storage method, wherein the target index information includes a key-value pair of the metadata, data pointing information of the key-value pair, and an object name and data offset information corresponding to the value in the key-value pair; when the target query data does not exist in the metadata cache module, the memory index is acquired, and the target query data is read from the disk according to the memory index.

[0142] Optionally, the second storage unit 703 includes: a first detection subunit, a first processing subunit, and a second processing subunit. The first detection subunit is used to detect the byte length of the key-value pair of the metadata; the first processing subunit is used to record the target index information of the metadata in the memory index by using the array storage method when the byte length is less than the preset length; the second processing subunit is used to record the target index information of the metadata in the memory index by using the hash table storage method when the byte length is greater than or equal to the preset length.

[0143] Optionally, the second storage unit 703 includes: a second detection subunit, a third processing subunit, and a fourth processing subunit. The second detection subunit is used to detect whether the metadata is continuous integer metadata for a core path, wherein the core path is an access path with an access frequency greater than a preset frequency; the third processing subunit is used to record the target index information of the metadata in the memory index in the array storage method when the metadata is continuous integer metadata for the core path; and the fourth processing subunit is used to record the target index information of the metadata in the memory index in the hash table storage method when the metadata is not continuous integer metadata for the core path.

[0144] Optionally, the metadata index processing device also includes: a setting unit, used to set the namespace corresponding to the metadata according to the key-value pair type of the metadata, wherein the namespace corresponding to the metadata is used to reduce the primary key prefix information of the key-value pair of the metadata.

[0145] Optionally, the metadata index processing device further includes: a first recording unit and a second recording unit. The first recording unit is used to convert the object name of the primary key of the key-value pair into a number as the subscript of the array when the target index information of the metadata is recorded in the memory index in the array storage mode; the second recording unit is used to record the object name of the primary key of the key-value pair in the memory index without performing any conversion on the object name of the primary key of the key-value pair when the target index information of the metadata is recorded in the memory index in the hash table storage mode, wherein the length of the memory index structure message generated by the hash table storage mode is greater than the length of the memory index structure message generated by the array storage mode.

[0146] Optionally, the metadata index processing device also includes: a data writing unit, which is used to write the metadata in the distributed storage service and update the information in the memory index by adopting a batch append writing method for log objects, wherein the batch append writing method for log objects is used to assign a globally unique and monotonically increasing version number to the append writing request without the need to access the memory index, and append it to the data backend of the distributed storage service, wherein the version number is used for verification and judgment during data reconstruction.

[0147] Optionally, the data writing unit includes: a data rewriting sub-unit, which is used to serialize the metadata and append it to the log object, and return an error when the sum of the current log object size and the size of the data to be written is greater than a preset threshold, and generate a new log object for data rewriting operations.

[0148] Optionally, the metadata index processing device further includes: a first detection unit, a first determination unit, and a second determination unit. The first detection unit is used to detect whether the priority of the namespace corresponding to the metadata is higher than the preset priority; the first determination unit is used to store the metadata as the hot data in the metadata cache module when the priority of the namespace corresponding to the metadata is higher than the preset priority; the second determination unit is used to determine the time to store the metadata in the metadata storage module according to the memory usage of the distributed storage service when the priority of the namespace corresponding to the metadata is not higher than the preset priority.

[0149] Optionally, the metadata index processing device further includes: a second detection unit, a third determination unit, and a fourth determination unit. The second detection unit is used to detect whether the ratio between the total metadata write amount and the actual write amount in the distributed storage service is greater than a preset value; the third determination unit is used to, when the ratio is greater than the preset value, fully scan the log objects in the distributed storage service to determine the log objects with hole data in the distributed storage service, wherein the hole data is used to represent redundant data that has been overwritten or partially deleted; the fourth determination unit is used to, when the first log object is the log object with hole data and it is detected that the hole rate of the first log object exceeds the preset hole rate, perform a data hole recovery operation on the first log object, wherein the hole rate represents the proportion of the hole data of the first log object in the full data of the first log object, and the data hole recovery operation is used to append the existing data to the log object and update the memory information, and after completion, delete the log object corresponding to the hole data in the read-only state.

[0150] Optionally, the third determination unit includes: a third detection subunit, a first determination subunit, a fourth detection subunit, a second determination subunit, a third determination subunit and a fourth determination subunit. The third detection subunit is used to detect whether the metadata is recorded in the memory index; the first determination subunit is used to determine that the metadata is the hole data when the metadata is not recorded in the memory index; the fourth detection subunit is used to detect whether the object version number of the log object corresponding to the metadata is lower than the version number of the metadata in the memory index when the metadata is recorded in the memory index; the second determination subunit is used to determine that the metadata is the hole data when the object version number is lower than the version number of the metadata in the memory index; the third determination subunit is used to determine that the metadata is not the hole data when the object version number is not lower than the version number of the metadata in the memory index; the fourth determination subunit is used to determine that the log object corresponding to the hole data is the log object with hole data in the distributed storage service.

[0151] Optionally, the metadata index processing device further includes: a sorting unit and a metadata processing unit. The sorting unit is used to sort the multiple metadata according to the version number corresponding to each metadata when multiple metadata are concurrently written to the same primary key to obtain a sorting result; the metadata processing unit is used to use the metadata with the largest version number as the metadata actually written to the primary key according to the sorting result, wherein the metadata with the largest version number is the metadata that is last requested to be written to the primary key among the multiple metadata.

[0152] Optionally, the metadata index processing device also includes: a loading order determination unit, which is used to determine the loading order of the metadata according to the priority of the namespace corresponding to the metadata when a node restart occurs in the distributed storage service.

[0153] Optionally, the metadata index processing device also includes: a reverse loading unit, which is used to obtain the metadata in the data backend for reverse loading by accessing the metadata controller of the distributed storage service when a node restart occurs in the distributed storage service, wherein the reverse loading is used to restore the metadata with the same name with the largest version number by comparing the version numbers of the metadata.

[0154] Optionally, it is characterized in that the distributed storage service includes the following components: a metadata controller, used to manage data information of all log objects written in the distributed storage service; a data backend, used to store the latest data of the distributed storage service; a block device service component, wherein the block device service component includes a block device logic module and a metadata module, and the metadata module includes the metadata cache module and the metadata storage module.

[0155] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute any one of the above-mentioned metadata index processing methods.

[0156] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes any one of the above-mentioned metadata index processing methods.

[0157] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0158] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0160] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0161] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0163] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A metadata index processing method, applied to distributed storage services, It is characterized in that The distributed storage service at least includes a metadata cache module and a metadata storage module, and the metadata index processing method includes: Obtain at least one metadata of the distributed storage service; Storing hotspot data in the at least one metadata in the metadata cache module, wherein a calling frequency of the hotspot data is higher than a preset calling frequency, and a data reading priority of the metadata cache module is higher than a data reading priority of the metadata storage module; The target index information of the metadata is recorded in a memory index in the metadata storage module by using an array storage method or a hash table storage method, wherein the target index information includes a key-value pair of the metadata, data pointing information of the key-value pair, and an object name and data offset information corresponding to the value in the key-value pair; when the target query data does not exist in the metadata cache module, the memory index is obtained, and the target query data is read from the disk according to the memory index; Wherein, adopting an array storage method or a hash table storage method to record the target index information of the metadata in the memory index in the metadata storage module includes: detecting the byte length of the key-value pair of the metadata; when the byte length is less than a preset length, adopting the array storage method to record the target index information of the metadata in the memory index; when the byte length is greater than or equal to the preset length, adopting the hash table storage method to record the target index information of the metadata in the memory index; Detecting whether the metadata is continuous integer metadata for a core path, wherein the core path is an access path having an access frequency greater than a preset frequency; in a case where the metadata is continuous integer metadata for the core path, recording target index information of the metadata in the memory index in the array storage manner; and in a case where the metadata is not continuous integer metadata for the core path, recording target index information of the metadata in the memory index in the hash table storage manner.

2. The method for processing metadata index according to claim 1, It is characterized in that In the process of recording the target index information of the metadata in the memory index in the metadata storage module, the metadata index processing method further includes: A namespace corresponding to the metadata is set according to a key-value pair type of the metadata, wherein the namespace corresponding to the metadata is used to reduce primary key prefix information of the key-value pair of the metadata.

3. The method for processing metadata index according to claim 1, It is characterized in that The metadata index processing method further includes: When the target index information of the metadata is recorded in the memory index in the array storage manner, the object name of the primary key of the key-value pair is converted into a number as the subscript of the array; When the target index information of the metadata is recorded in the memory index by adopting the hash table storage method, the object name of the primary key of the key-value pair is recorded in the memory index without performing any conversion on the object name of the primary key of the key-value pair, wherein the length of the memory index structure message generated by adopting the hash table storage method is greater than the length of the memory index structure message generated by adopting the array storage method.

4. The method for processing metadata index according to claim 1, It is characterized in that The metadata index processing method further includes: The metadata is written into the distributed storage service and the information in the memory index is updated by adopting a method of batch append writing of log objects, wherein the method of batch append writing of log objects is used to assign a globally unique and monotonically increasing version number to the append writing request without the need to access the memory index, and append the number to the data backend of the distributed storage service, wherein the version number is used for verification judgment during data reconstruction.

5. The method for processing metadata index according to claim 4, It is characterized in that The metadata is written into the distributed storage service by appending log objects in batches, including: The metadata is serialized and appended to the log object, and an error is returned when the sum of the current log object size and the size of the data to be written is greater than a preset threshold, and a new log object is generated to rewrite the data.

6. The method for processing metadata index according to claim 2, It is characterized in that After setting the namespace corresponding to the metadata according to the key-value pair type of the metadata, the metadata index processing method further includes: Detecting whether the priority of the namespace corresponding to the metadata is higher than a preset priority; When the priority of the namespace corresponding to the metadata is higher than the preset priority, storing the metadata as the hot data in the metadata cache module; When the priority of the namespace corresponding to the metadata is not higher than the preset priority, the time to store the metadata in the metadata storage module is determined according to the memory usage of the distributed storage service.

7. The method for processing metadata index according to claim 1, It is characterized in that After recording the target index information of the metadata in the memory index in the metadata storage module, the metadata index processing method further includes: Detecting whether the ratio between the total metadata write amount and the actual write amount in the distributed storage service is greater than a preset value; When the ratio is greater than the preset value, a full scan is performed on the log objects in the distributed storage service to determine the log objects in the distributed storage service that have hole data, wherein the hole data is used to represent redundant data that has been overwritten or partially deleted; When the Ith log object is the log object with hole data and it is detected that the hole rate of the Ith log object exceeds the preset hole rate, a data hole recovery operation is performed on the Ith log object, wherein the hole rate represents the proportion of the hole data of the Ith log object in the full data of the Ith log object, and the data hole recovery operation is used to append the existing data to the log object and update the memory information, and after completion, delete the log object corresponding to the hole data in the read-only state.

8. The method for processing metadata index according to claim 7, It is characterized in that Fully scanning the log objects in the distributed storage service to determine the log objects in the distributed storage service that have empty data includes: Detecting whether the metadata is recorded in the memory index; When the metadata is not recorded in the memory index, determining that the metadata is the hole data; When the metadata is recorded in the memory index, detecting whether the object version number of the log object corresponding to the metadata is lower than the version number of the metadata in the memory index; When the object version number is lower than the version number of the metadata in the memory index, determining that the metadata is the hole data; When the object version number is not lower than the version number of the metadata in the memory index, determining that the metadata is not the hole data; The log object corresponding to the hole data is determined as the log object with the hole data in the distributed storage service.

9. The method for processing metadata index according to claim 1, It is characterized in that After recording the target index information of the metadata in the memory index in the metadata storage module, the metadata index processing method further includes: When multiple metadata are concurrently written to the same primary key, the multiple metadata are sorted according to the version number corresponding to each metadata to obtain a sorting result; According to the sorting result, the metadata with the largest version number is used as the metadata actually written into the primary key, wherein the metadata with the largest version number is the metadata that is last requested to be written into the primary key among the multiple metadata.

10. The method for processing metadata index according to claim 1, It is characterized in that After recording the target index information of the metadata in the memory index in the metadata storage module, the metadata index processing method further includes: When a node restart occurs in the distributed storage service, the order in which the metadata is loaded is determined according to the priority of the namespace corresponding to the metadata.

11. The method for processing metadata index according to claim 1, It is characterized in that After recording the target index information of the metadata in the memory index in the metadata storage module, the metadata index processing method further includes: When a node restart occurs in the distributed storage service, the metadata in the data backend is obtained by accessing the metadata controller of the distributed storage service and reverse loading is performed, wherein the reverse loading is used to restore the metadata with the largest version number by comparing the version numbers of the metadata.

12. The method for processing metadata index according to claim 1, It is characterized in that The distributed storage service includes the following components: A metadata controller, used to manage data information of all log objects written in the distributed storage service; A data backend, used to store the latest data of the distributed storage service; A block device service component, wherein the block device service component includes a block device logic module and a metadata module, and the metadata module includes the metadata cache module and the metadata storage module.

13. A metadata index processing device, applied to a distributed storage service, It is characterized in that The distributed storage service at least includes a metadata cache module and a metadata storage module, and the metadata index processing device includes: An acquisition unit, configured to acquire at least one metadata of the distributed storage service; A first storage unit, used for storing hotspot data in the at least one metadata in the metadata cache module, wherein the calling frequency of the hotspot data is higher than a preset calling frequency, and the data reading priority of the metadata cache module is higher than the data reading priority of the metadata storage module; a second storage unit, configured to record the target index information of the metadata in a memory index in the metadata storage module by adopting an array storage method or a hash table storage method, wherein the target index information includes a key-value pair of the metadata, data pointing information of the key-value pair, and an object name and data offset information corresponding to the value in the key-value pair; when the target query data does not exist in the metadata cache module, the memory index is obtained, and the target query data is read from the disk according to the memory index; The second storage unit includes: a first detection subunit, used to detect the byte length of the key-value pair of the metadata; a first processing subunit, used to record the target index information of the metadata in the memory index by adopting the array storage method when the byte length is less than the preset length; a second processing subunit, used to record the target index information of the metadata in the memory index by adopting the hash table storage method when the byte length is greater than or equal to the preset length; The second storage unit also includes: a second detection subunit, used to detect whether the metadata is continuous integer metadata for a core path, wherein the core path is an access path with an access frequency greater than a preset frequency; a third processing subunit, used to record the target index information of the metadata in the memory index by adopting the array storage method when the metadata is continuous integer metadata for the core path; and a fourth processing subunit, used to record the target index information of the metadata in the memory index by adopting the hash table storage method when the metadata is not continuous integer metadata for the core path.

14. An electronic device, It is characterized in that It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the metadata index processing method described in any one of claims 1 to 12.

15. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the metadata index processing method according to any one of claims 1 to 12.

Images