Metadata index construction method and device, equipment, storage medium and program product
By building a learning index and hash table index structure in the cloud block storage system, the problem of low metadata index efficiency is solved, efficient metadata query and update performance is achieved, and the low latency requirements of the cloud block storage system are met.
Patent Information
- Application Number
- CN202410987089.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
The inefficiency of metadata indexing in existing cloud block storage systems leads to poor query and update performance, which is difficult to meet the needs of modern low-latency cloud block storage systems.
By dividing the storage space into multiple shard spaces, a learning secondary index structure and a primary index structure in the form of hash tables are constructed, and learning index instances are generated using machine learning models to achieve rapid metadata query and update.
It reduces the memory usage of the auxiliary index structure, improves index query and update performance, and adapts to the efficient storage needs of cloud block storage systems.
Smart Images

Figure CN118939657A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud computing technology, and in particular to a metadata index construction method, device, equipment, storage medium and program product. Background Art
[0002] As an emerging general-purpose block storage service, cloud block storage has the advantages of high performance, high scalability, high reliability, and versatility. Currently, all major cloud providers provide cloud block storage services to tenants, supporting basic data storage in various industries. A typical cloud block storage system consists of the following parts:
[0003] The client server is responsible for the virtualization of cloud block devices and provides a logical access view of cloud hard disks. Input / output (IO) requests from users will be redirected to the backend storage cluster at the block device driver layer of the client server.
[0004] System management server, responsible for key metadata management, storage node management, garbage collection management, and data migration management of the storage cluster;
[0005] The index server stores the metadata of all data blocks within its scope of responsibility, is responsible for the mapping management and conversion of user logical addresses to actual physical storage addresses in the cloud block storage system, and builds block metadata indexes to accelerate metadata query and update;
[0006] The storage server is responsible for the persistent storage of data blocks. The storage server will directly operate the storage device (for example, mechanical hard disk, solid-state drive, etc.) to perform the final data access operation. In addition, the cloud block storage system usually uses a log structure to store block data, that is, the user's update of a certain data block will not overwrite the original old data block, but will be written to the tail of the hard disk log space in a sequential append manner. This method is conducive to the sequential write performance of existing mainstream storage devices. Because of this, the cloud block storage system requires a large number of metadata indexes to record the latest storage location of each data block;
[0007] Network cables and switches, the transmission of data and requests between servers is based on the high-speed network and switches of the data center.
[0008] At present, cloud block storage systems based on advanced network technology can already achieve access delays of hundreds of microseconds for the entire IO request chain (including request sending, index query, data reading and writing). Due to the high access latency of the hard disk, the data reading and writing steps already account for most of the latency overhead. This places extremely high demands on the indexing performance of cloud block storage. Specifically, the cloud block storage system requires its metadata index to meet low-latency index access, query and update performance of less than 10 microseconds in order not to affect the overall performance of the system. If additional hard disk access is introduced during the indexing process, the performance will be halved.
[0009] Metadata indexing is one of the main technologies used in cloud block storage systems to improve data block query and access performance. An index is usually composed of multiple index items, each of which contains an index key and an index value. The index links all index items through a specific auxiliary index structure to achieve query and update functions.
[0010] At present, cloud block storage systems generally use hard disks (e.g., mechanical hard disks or solid-state drives) as persistent storage devices. However, since the IO access latency of hard disks is much higher than that of dynamic random access memory (DRAM), the performance of simply using the hard disk traversal method to query the block data required by users is extremely poor. Therefore, how to quickly find block data in the hard disk has become a key indicator affecting the performance of cloud block storage systems. In this regard, existing cloud block storage systems first extract metadata (e.g., storage physical location, block size, etc.) for all block data in the system, and then use the metadata as an index value and assign it a unique index key, thereby establishing a metadata index to achieve fast query of block data.
[0011] As the amount of data continues to grow, industrialized cloud block storage systems generally have a total storage capacity of hundreds of terabytes (TB) to tens of petabytes (PB). In such large-scale block storage systems, the cost and performance issues of traditional indexes (e.g., B+ trees, red-black trees, etc.) used to store metadata are becoming increasingly prominent, mainly manifested in: excessive memory space usage, multiple indirect searches required for each query, and the efficiency of index query and update based on file or database persistent indexes is also affected by the IO performance of block devices, with high latency and low throughput, which makes it difficult to meet the index performance requirements of modern low-latency cloud block storage systems. Summary of the invention
[0012] The technical solution of the present invention aims to provide a metadata index construction method, device, equipment, storage medium and program product to solve the problem of low efficiency of existing metadata indexes.
[0013] In order to solve the above technical problems, the embodiments of the present invention provide the following technical solutions:
[0014] An embodiment of the present invention provides a metadata index construction method, including:
[0015] Dividing the storage space into a plurality of shard spaces, each shard space including at least one user data block;
[0016] Extracting metadata of the user data block and storing the metadata;
[0017] Constructing a learning-type secondary index structure for the shard space, the secondary index structure comprising: first identification information of the shard space and a learning index instance, the learning index instance comprising: a correspondence between a first index key of the metadata and a storage address of the metadata;
[0018] A primary index structure in the form of a hash table is constructed for the secondary index structure, wherein the primary index structure includes: a second index key of the first identification information, and a storage address of the secondary index structure.
[0019] Optionally, the metadata includes a metadata key and a metadata value:
[0020] The metadata key includes information indicating an offset of a logical address of the user data block in the shard space;
[0021] The metadata value includes at least one of the following:
[0022] Indicative information of the physical storage address of the user data block;
[0023] Size information of the user data block;
[0024] Verification information of the user data block;
[0025] Update information of the user data block;
[0026] Deletion information of the user data block.
[0027] Optionally, the secondary index structure further includes at least one of the following:
[0028] Indication information used to indicate whether the shard space is in a garbage collection state;
[0029] Indication information used to indicate the storage server of the slice space.
[0030] Optionally, the learning index instance is obtained in the following manner:
[0031] Obtaining a first index key of the metadata and a storage address of the metadata;
[0032] Generate learning data according to the first index key and the corresponding storage address;
[0033] Dividing the learning data into training data and supervision data;
[0034] The learning index instance is obtained according to the training data and the supervision data using a machine learning model.
[0035] Optionally, the method further comprises:
[0036] Receive a query request, the query request including a first index key and a second index key;
[0037] According to the second index key and the primary index structure, query the storage address of the secondary index structure;
[0038] When the storage address of the secondary index structure is queried, the storage address of the metadata is obtained according to the first index key and the secondary index structure.
[0039] Optionally, the method further comprises at least one of the following:
[0040] In the case where the metadata corresponding to the metadata storage address is updated, in the metadata storage address, replacing the metadata value in the metadata stored at the storage address with the metadata value in the updated metadata;
[0041] When the shard space in the storage space is added or deleted, metadata of the user data block is added or deleted in the storage address of the metadata corresponding to the user data block in the shard space.
[0042] Optionally, the method further comprises at least one of the following:
[0043] In the case where some user data blocks are added or deleted in the shard space, the learning index instance is retrained according to the storage address of the metadata corresponding to the added or deleted user data blocks;
[0044] When the shard space in the storage space is added or deleted as a whole, the corresponding secondary index structure is added or deleted according to the added or deleted shard space, and the primary index structure is updated.
[0045] The embodiment of the present invention also provides a metadata index construction device, including:
[0046] A first processing module, configured to divide a storage space into a plurality of shard spaces, wherein the shard space includes at least one user data block;
[0047] A second processing module, used for extracting metadata of the user data block and storing the metadata;
[0048] A first construction module is used to construct a learning type secondary index structure for the shard space, wherein the secondary index structure includes: first identification information of the shard space and a learning index instance, wherein the learning index instance includes: a correspondence between a first index key of the metadata and a storage address of the metadata;
[0049] The second construction module is used to construct a primary index structure in the form of a hash table for the secondary index structure, wherein the primary index structure includes: a second index key of the first identification information, and a storage address of the secondary index structure.
[0050] An embodiment of the present invention further provides a metadata index construction device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program implements the metadata index construction method described above when executed by the processor.
[0051] An embodiment of the present invention further provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the steps in the metadata index construction method described in any one of the above are implemented.
[0052] An embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in any one of the metadata index construction methods described above.
[0053] At least one of the above technical solutions of the present invention has the following beneficial effects:
[0054] The metadata index construction method provided by the scheme of the present invention divides the storage space into multiple shard spaces, extracts the metadata of the user data blocks in the shard space, stores the data, and constructs a learning secondary index structure for the shard space to index the metadata, thereby reducing the memory usage of the auxiliary index structure, improving the index query and update performance, and constructing a primary index structure in the form of a hash table for the secondary index structure, that is, using the hash table as the primary index structure to save all secondary index structures, and different secondary index structures can be accessed or updated in parallel, thereby improving index efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 A flowchart of a metadata index construction method provided by an embodiment of the present invention;
[0056] Figure 2 A specific flow chart of constructing a metadata index provided by an embodiment of the present invention;
[0057] Figure 3 A flow chart of a hybrid index query method provided by an embodiment of the present invention;
[0058] Figure 4 A schematic diagram of the structure of a metadata index controller for cloud block storage provided by an embodiment of the present invention;
[0059] Figure 5 A schematic diagram of the structure of a metadata index construction device provided by an embodiment of the present invention;
[0060] Figure 6 A schematic diagram of the structure of a metadata index construction device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0061] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0062] In order to solve the problem of low efficiency of existing metadata indexing, embodiments of the present invention provide a metadata indexing construction method, apparatus, device, storage medium and program product.
[0063] like Figure 1 As shown, an embodiment of the present invention provides a metadata index construction method, including:
[0064] Step 101: Divide a storage space into a plurality of shard spaces, each shard space including at least one user data block.
[0065] It should be noted that the metadata index construction method provided in the embodiment of the present invention is mainly applicable to a cloud block storage system based on a log storage structure.
[0066] In this step, the physical storage space of the cloud block storage system is logically divided into fixed-size slice spaces (Plane). The Plane is a management unit containing a certain number of user data blocks (called Page). The fixed size is determined by the system architecture of the cloud block storage, and its value can be dynamically adjusted during the initialization or operation of the cloud block storage system. For example, the default value is 32GB.
[0067] Afterwards, a plane identification (ID) is assigned to each plane, and a page ID is assigned to each page.
[0068] Step 102: extract metadata of the user data block and store the metadata.
[0069] Specifically, in this step, the metadata of all Pages in the Plane are generated in sequence according to the ascending order of the Plane ID. The metadata is organized into an index item structure in the format of [index key, index value]. The index key is the Page ID of the metadata of the Page, and the index value is a memory pointer, which points to the memory address of a structure composed of the offset field, size field and other fields of the Page. Afterwards, the index item structures of all Pages in the Plane are sequentially filled into a metadata storage array according to the ascending order of the above index keys.
[0070] Step 103: construct a learned secondary index structure for the shard space, the secondary index structure including: first identification information of the shard space and a learned index instance, the learned index instance including: a correspondence between a first index key of the metadata and a storage address of the metadata.
[0071] It should be noted that the learning index uses machine learning technology to learn the distribution characteristics of data and uses a fitting function to establish a correspondence between the index key and the actual storage location of the index item. When querying, the learning index provides a direct search method that directly calculates the memory location of the index value based on the requested index key and the distribution fitting function. This index construction and query method has the following outstanding advantages: (1) Small memory usage: Through machine learning algorithms to learn the distribution law of data, the learning index uses mathematical models (for example, linear models, piecewise linear models, neural networks, etc.) to replace the auxiliary index structure of the traditional index, thereby greatly reducing the memory usage caused by the auxiliary index structure itself, and using very little memory to store information. (2) Higher query efficiency: Compared with the indirect search method of the traditional index, the direct search method of the learning index can directly use a calculation method to locate the memory location of the index value, thereby improving the query performance. In addition, the learning index requires the index items to be sorted in order, so that it can support functions such as sequential query and range query.
[0072] Specifically, in this step, an independent secondary index structure is assigned to each Plane, and the secondary index structure includes: a globally unique first identification information and a learning index instance, wherein the first identification information can be a Plane ID field, which is used to indicate the Plane corresponding to the secondary index structure, and the learning index instance includes the correspondence between the first index key of the metadata and the storage address of the metadata, the first index key of the metadata is the Page ID, and the storage address of the metadata can be the subscript of the metadata in the metadata storage array. Optionally, the correspondence between the first index key of the metadata and the storage address of the metadata is a piecewise linear function, that is, the input value of the piecewise linear function is the Page ID, and the output value is the subscript in the metadata storage array.
[0073] The learning index instance corresponds one-to-one to the metadata storage array of the shard space.
[0074] By using the secondary index structure to index the metadata of the cloud block storage system, the memory usage of the auxiliary index structure is reduced and the index query and update performance is improved.
[0075] Optionally, the secondary index structure further includes at least one of the following:
[0076] Indication information for indicating whether the shard space is in a garbage collection state, wherein the indication information is a garbage collection (GC) state field, which is used to indicate whether the Plane corresponding to the secondary index is in a garbage collection state;
[0077] The indication information is used to indicate the storage server of the slice space, wherein the indication information is an ExtSev ID field, which is used to indicate the storage server corresponding to the Plane.
[0078] Step 104: construct a primary index structure in the form of a hash table for the secondary index structure, wherein the primary index structure includes: a second index key of the first identification information, and a storage address of the secondary index structure.
[0079] That is, in this step, a primary index structure is established to manage all secondary index structures.
[0080] Specifically, a primary index structure of a hash table item is assigned to each secondary index structure. The primary index structure of the hash table item is a binary structure in the format of [index key, index value], wherein the index key (second index key) is the Plane ID field of the secondary index structure, and the index value is a memory address pointer pointing to the secondary index structure (i.e., the storage address of the secondary index structure).
[0081] When constructing the primary index structure of the hash table item, extract the Plane ID field (i.e., the first identification information) in the secondary index structure as the index key (second index key) of the hash table item. Obtain the memory address pointer of the secondary index structure as the index value of the hash table item.
[0082] Afterwards, all hash table items are managed by a global hash table, that is, a primary index structure in the form of a hash table includes the primary index structure of at least one hash table item mentioned above, and the hash table is an existing hash table implementation, such as the unordered_map of the Standard Template Library (STL), Cuckoo Hashing, etc.
[0083] A primary index structure in the form of a hash table is used to save all secondary index structures. Learning instances in different secondary index structures can be accessed or updated in parallel, overcoming the shortcoming that a single learning instance is difficult to access in parallel and improving the overall parallelism of metadata indexing.
[0084] Specifically, the embodiment of the present invention is a hybrid index construction method combining a hash table and a learning index. The hybrid index uses a low-to-high construction method.
[0085] Optionally, the metadata (also referred to as metadata information) includes a metadata key and a metadata value:
[0086] The metadata key includes: indication information of the offset of the logical address of the user data block in the shard space, wherein the indication information is a Page ID field, and the field value is the logical address offset of the Page in the current Plane;
[0087] The metadata value includes at least one of the following:
[0088] The indication information of the physical storage address of the user data block, optionally, the indication information is an offset field, the field value is the actual storage position of the current Page in the storage server;
[0089] The size information of the user data block, wherein the size information is a size field, and the field value is the size of the current Page;
[0090] Verification information of the user data block;
[0091] Update information of the user data block;
[0092] Deletion information of the user data block.
[0093] Among them, verification information, update information, and deletion information are information used to indicate the status of Page data.
[0094] In an optional embodiment, the learning index instance is obtained by:
[0095] Obtaining the first index key of the metadata and the storage address of the metadata. Specifically, according to the filled metadata storage array, extracting the first index key of the metadata and the subscript of the metadata in the metadata storage array, where the subscript is the indication information of the storage address of the metadata;
[0096] Learning data is generated according to the first index key and the corresponding storage address. Specifically, the first index key of the metadata and the subscript of the metadata in the metadata storage array are organized into a generation structure in the format of [index key, array subscript], and the generation structure of all Pages in the shard space is stored in a temporary generation array as learning data.
[0097] The learning data is divided into training data (or training set) and supervision data (or supervision set), wherein the division ratio can be set according to requirements.
[0098] The learning index instance is obtained according to the training data and the supervision data using a machine learning model, wherein the machine learning model includes but is not limited to a linear regression model, a deep neural network and other models. In this embodiment, the linear regression model is used by default, and the training data and the supervision data are input into the linear regression model for training and supervision to obtain a piecewise linear function in the learning index instance, and the piecewise linear relationship indicates the correspondence between the first index key of the metadata and the storage address of the metadata.
[0099] It can also be understood that learning index instances is an important part of secondary indexes, including:
[0100] (1) A core index component, which is a piecewise linear function. The input value of the function is PageID, and the output value is the subscript of the metadata storage array. The generation method is to first extract the first index key of the metadata of the Page and the subscript of the metadata in the metadata storage array according to the filled metadata storage array, and organize them into a generation structure in the format of [index key, array subscript], and then store the generation structure of all Pages of Plane into a temporary generation array. Finally, the generation array is input into the trained index generator, and the generator will calculate and output the core index component corresponding to the learning index instance.
[0101] (2) An index generator for generating core index components. The generator is trained by a machine learning method, and the method is as follows: first, an existing conventional machine learning model is selected for the generator. The machine learning model includes but is not limited to a linear regression model, a deep neural network model, and the like. The linear regression model is used by default, and then a training set and a supervision set are input into the generator. The training set and the supervision set are respectively a set of labeled [index key, array subscript] generation structures. Finally, the existing machine learning training method is used for training, and the conventional supervised learning method is used by default.
[0102] After all secondary index structures are generated, continue to build the primary index structure.
[0103] Combine the following Figure 2 , specifically describes the metadata index construction process:
[0104] Divide the system's physical storage space into multiple Planes; extract the metadata of all user data blocks in the Plane; fill the metadata into the metadata storage array corresponding to the Plane in sequence; assign a secondary index structure to each Plane; call the index generator based on the metadata to generate the core index component; establish a primary index structure based on the Plane ID of the secondary index structure.
[0105] Furthermore, an embodiment of the present invention also provides a hybrid index query method, that is, the method further includes:
[0106] A query request is received (the query request is a query request for metadata, which may also be referred to as a retrieval query request), wherein the query request includes a first index key and a second index key. Specifically, when an application (or client) issues a query request Q, the query request Q is sent to an index server, wherein the query request Q is a binary structure of [Plane ID (second index key), PageID (first index key)]. After that, the query request Q is processed according to the following steps:
[0107] Determine the legitimacy of the query request Q, where legitimacy refers to whether the Plane ID (second index key) and Page ID (first index key) included in the query request Q are within the range allowed by the system, that is, whether they are within the range included in the primary index structure and the secondary index structure. If so, proceed to the subsequent process, otherwise return a query failure;
[0108] According to the second index key and the first-level index structure, query the storage address of the first-level index structure. Specifically, use the Plane ID as the second index key to query whether the first-level index structure in the form of a hash table includes the storage address of the second-level index structure corresponding to the PlaneID. If the storage address is found, proceed to the subsequent process. Otherwise, return a query failure.
[0109] When the storage address of the primary index structure is queried, the storage address of the metadata is obtained according to the first index key and the secondary index structure. Specifically, the Page ID is used as the first index key. In the learning index instance of the secondary index structure, the core index component is used to query whether the metadata storage array contains the memory pointer of the Page's metadata (the storage address of the metadata). If so, the memory pointer of the metadata is obtained and then returned; if not, the query failure is returned.
[0110] Combine the following Figure 3 , specifically describes the process of the hybrid index query method provided by the embodiment of the present invention:
[0111] The client sends a retrieval query request Q and checks the legality of Q, that is, checks whether Q is legal. If so, continue with the subsequent process, otherwise return query failure. If Q is legal, query the primary index structure to determine whether the secondary index structure required by the query request Q exists (that is, whether the storage address of the secondary index structure is queried). If so, continue with the subsequent process, otherwise return query failure. If the secondary index structure required by the query request Q is queried, query the secondary index structure to determine whether the metadata required by the query request Q exists. If so, continue with the subsequent process, otherwise return query failure. If the metadata required by the query request Q is queried, return the memory pointer where the metadata is located.
[0112] The embodiment of the present invention further provides a method for updating metadata, that is, the method further includes at least one of the following methods:
[0113] Method 1: when the metadata corresponding to the metadata storage address is updated, in the metadata storage address, the metadata value in the metadata stored in the storage address is replaced with the metadata value in the updated metadata.
[0114] That is, it can be understood that in the first embodiment, when the metadata corresponding to the storage address of the metadata is updated, the metadata key remains unchanged, but the metadata value is changed.
[0115] Method 2: When the shard space in the storage space is added or deleted, metadata of the user data block is added (or called increased) or deleted in the storage address of the metadata corresponding to the user data block in the shard space.
[0116] That is, it can be understood that the addition or deletion of metadata will change the metadata key and destroy the mapping relationship of the existing learning index.
[0117] Specifically, in the second method, after the storage address of the metadata is obtained according to the query method of the mixed index, the newly added or deleted metadata is overwritten with the old metadata through a pointer according to the storage address of the metadata.
[0118] The embodiment of the present invention further provides a hybrid index retraining method, that is, the method further includes at least one of the following methods:
[0119] Method three: when some user data blocks are added (or called increased) or deleted in the shard space, the learning index instance corresponding to the shard space is retrained according to the storage address of the metadata corresponding to the added or deleted user data blocks.
[0120] Specifically, in this third method, the generation method of the core index component in the secondary index structure is re-executed, first the latest metadata storage array of Plane is read, then the generator is called to update the core index component in the secondary index structure, and then the retraining of the learning index instance is performed. Among them, the addition or deletion of metadata of the user data block may include at least one of the following situations: the data within the Plane range is garbage collected by the cloud block storage system (that is, the user data blocks in the allocated space perform garbage collection operations), the cloud block storage system performs capacity adjustment operations (increase or decrease storage space), and some shard spaces (and internal user data blocks) are added or deleted (including the addition or deletion of metadata keys and metadata values). Among them, the garbage collection operation refers to the operation of deleting obsolete user data blocks in Plane and retaining only the latest user data blocks, and the capacity adjustment operation refers to the operation of the cloud block storage system adding new storage space or reducing operation space.
[0121] It should be noted that the metadata storage space is sorted by the size of the metadata key to provide orderliness. Therefore, when some metadata keys are changed, the storage location of other metadata information will change, thereby destroying the mapping relationship established by the learning index. In this case, the learning index needs to be retrained to establish a new mapping relationship. Since the above situation rarely occurs when the system is running, the retraining overhead is not large.
[0122] Here, retraining the learning index instance may also be understood as updating the learning index instance or retraining the learning index instance.
[0123] It should also be noted that in most cases of normal system operation, user operations on storage space will only cause the metadata value corresponding to the user data block to be updated (the metadata key will not be changed). In the case where only the metadata value is updated, the learning index instance does not need to be retrained, thereby avoiding triggering retraining in normal operation and reducing overhead.
[0124] That is, in this method, when some user data blocks are added or deleted in a shard space, the learning index instance of the secondary index structure corresponding to the shard space needs to be retrained, but the primary index structure does not need to be updated because the shard space still exists.
[0125] Method 4: When the shard space in the storage space is added (or called added) or deleted as a whole, the corresponding secondary index structure is added or deleted according to the added or deleted shard space, and the primary index structure is updated. The corresponding secondary index structure is the secondary index structure corresponding to the added or deleted shard space.
[0126] That is, in this method, when a shard space is added or deleted as a whole, the secondary index structure corresponding to the shard space needs to be added or deleted as a whole, and the primary index structure corresponding to the secondary index structure needs to be added or deleted from the primary index hash table (that is, "updating the primary index structure").
[0127] It should also be noted that when the cloud block storage system is initialized (ie, the storage space is initialized), a primary index structure needs to be constructed, which can also be understood as updating the primary index structure.
[0128] The embodiment of the present invention proposes a fixed mapping method between the secondary index structure and the user data block, constructs a static metadata memory layout, and avoids the index update from reducing the prediction accuracy of the learning index. On this basis, the learning index retraining is only performed during system initialization, garbage collection, and system expansion to avoid triggering the retraining operation at runtime and affecting the service quality.
[0129] It should also be noted that if Figure 4 As shown, an embodiment of the present invention further provides a metadata index controller for cloud block storage, the metadata index controller is applicable to a metadata server, the metadata server includes hardware such as a processor, a DRAM memory, a network card, etc., and the metadata index controller includes:
[0130] Metadata storage module, metadata index construction module, learning index training module, learning index regeneration module, metadata query and update module.
[0131] The metadata storage module is used to store metadata information of the cloud block storage system in an array, including Page ID and storage offset and length corresponding to the Page. The Page ID is used as an index key, and the index value is a memory pointer, which points to the offset field and size field of the Page.
[0132] The metadata index building module is used to generate the core index component in the secondary index structure according to the metadata of the Page inside the Plane and the subscript information of the metadata in the metadata storage array;
[0133] The learning index training module is used to train the generator of the learning index instance in the secondary index structure by a machine learning method when the cloud block storage system is initialized;
[0134] The learning index regeneration module is used to trigger the execution of the metadata index construction module when the cloud block storage system is initialized, the data within the Plane range is garbage collected by the cloud block storage system, or the cloud block storage system is expanded;
[0135] The metadata index building module is also used to build a primary index structure in the form of a hash table;
[0136] The metadata query and update module is used to process index query or update requests.
[0137] The metadata index construction method provided by the embodiment of the present invention has low memory usage and high index construction efficiency, can effectively reduce the runtime memory space usage of the cloud block storage system, and improve the index query and update performance. Specifically, a hybrid index combining a hash table and a learning index is proposed as the metadata index of the cloud block storage system. Among them, the hash table is a primary index structure, and the learning index is a secondary index structure. In the hybrid index architecture, a learning index based on a piecewise linear model is used as a secondary index structure to directly index the massive metadata in the block storage system. The learning index uses a machine learning method to construct an auxiliary index structure based on a piecewise linear function according to the memory space distribution characteristics of the metadata, which can effectively reduce the memory usage of the auxiliary index structure while maintaining stable query performance. A fixed mapping method between the secondary index and the user data is proposed to construct a static metadata memory layout. The PB-level system logical space is divided into multiple small-size (default 32GB) Planes, and the secondary index is mapped one-to-one to the Plane. Under this allocation strategy, the Plane contains fixed user logical space data blocks, whose Page ID is determined at creation time and will not be modified at runtime, and the Plane will not insert new data blocks at runtime. A method for in-place overwriting and updating of secondary indexes is proposed. When the data of traditional indexes changes (i.e., the latest storage location of Page changes), it is necessary to insert new index items after deleting the old index items of Page, which causes structural changes and affects the performance of learning indexes. When updating the secondary index, the old index items are not deleted, but the old index values are updated in-place. Since the secondary index corresponding to Plane uses Page ID as the index value to construct the learning index, the structure of the learning index will not change during runtime, thereby avoiding the index update from reducing the prediction accuracy of the learning index. On the basis of constructing the static metadata memory layout and the index overwriting and updating method, the triggering conditions for model retraining and index regeneration of the learning index are simplified. Since the structure of the learning index will not change during runtime, model retraining and index regeneration are only performed during system initialization, garbage collection and system expansion, avoiding the frequent retraining operations during runtime from affecting the performance of the foreground service. In the hybrid index architecture, in order to improve the overall parallel access performance of the index, a hash table is used as the primary index to store all secondary index structures. On the one hand, the number of secondary indexes in a PB-level storage system is only in the hundreds of KB level, so the primary index will not occupy a large amount of memory; on the other hand, different secondary indexes are independently responsible for reading and writing user data within the Plane range. There is no dependency between different secondary indexes, and queries and updates can be processed in parallel.
[0138] Compared with traditional indexes, the metadata index construction method of the embodiment of the present invention has small memory space requirements and low memory operation occupancy. By introducing a learning index in a hybrid index architecture, the memory occupancy of the index structure itself can be effectively reduced, and the memory space requirements for metadata indexing can be reduced, which enables the cloud block storage system to better adapt to limited memory resources and obtain lower memory costs. Compared with traditional indexes and existing learning indexes, the access concurrency of the embodiment of the present invention is high. A two-layer hybrid index architecture is used. First, the massive metadata of hundreds of billions is divided into secondary index fragments according to a fixed range (i.e., Plane), and then a hash table is used to save all secondary indexes. Different secondary indexes can be accessed in parallel, thereby improving the overall concurrency capability of the metadata index. The existing learning index retraining and regeneration method is simplified, and the retraining and regeneration requirements of the learning index are greatly simplified. There is no need to repeatedly retrain the model and regenerate the index structure after the user data is updated. Thereby saving a lot of time and computing resources and improving the response speed and performance of the system.
[0139] Cloud block storage systems and cloud hard disk products are bound to face the need to reduce costs and increase efficiency of storage systems under the trend of large total data volume, diversified users, and extreme loads. While improving data reading and writing capabilities, they must also minimize cluster hardware deployment costs and operation and maintenance costs. The embodiments of the present invention can be applied to cloud block storage systems and cloud hard disk products of cloud service providers, providing them with a metadata index storage method with low memory space occupancy, thereby reducing the operating costs and hardware deployment costs of cloud storage systems, which is of great value for cloud computing providers to reduce costs and increase efficiency.
[0140] like Figure 5 As shown, the embodiment of the present invention also provides a metadata index construction device, including:
[0141] A first processing module 501 is used to divide the storage space into a plurality of shard spaces, each shard space including at least one user data block;
[0142] A second processing module 502, configured to extract metadata of the user data block and store the metadata;
[0143] A first construction module 503 is used to construct a learning type secondary index structure for the shard space, wherein the secondary index structure includes: first identification information of the shard space and a learning index instance, wherein the learning index instance includes: a correspondence between a first index key of the metadata and a storage address of the metadata;
[0144] The second construction module 504 is used to construct a primary index structure in the form of a hash table for the secondary index structure, wherein the primary index structure includes: a second index key of the first identification information, and a storage address of the secondary index structure.
[0145] Optionally, the metadata includes a metadata key and a metadata value:
[0146] The metadata key includes information indicating an offset of a logical address of the user data block in the shard space;
[0147] The metadata value includes at least one of the following:
[0148] Indicative information of the physical storage address of the user data block;
[0149] Size information of the user data block;
[0150] Verification information of the user data block;
[0151] Update information of the user data block;
[0152] Deletion information of the user data block.
[0153] Optionally, the secondary index structure further includes at least one of the following:
[0154] Indication information used to indicate whether the shard space is in a garbage collection state;
[0155] Indication information used to indicate the storage server of the slice space.
[0156] Optionally, the first building module 503 includes:
[0157] A first processing unit, configured to obtain a first index key of the metadata and a storage address of the metadata;
[0158] A second processing unit, configured to generate learning data according to the first index key and the corresponding storage address;
[0159] A third processing unit, used for dividing the learning data into training data and supervision data;
[0160] The fourth processing unit is used to use the machine learning model to obtain the learning index instance according to the training data and the supervision data.
[0161] Optionally, the device further comprises:
[0162] A receiving module, configured to receive a query request, wherein the query request includes a first index key and a second index key;
[0163] A first query module, configured to query a storage address of the secondary index structure according to the second index key and the primary index structure;
[0164] The second query module is used to obtain the storage address of the metadata according to the first index key and the secondary index structure when the storage address of the secondary index structure is queried.
[0165] Optionally, the device further comprises a third processing module, wherein the third processing module is used for at least one of the following:
[0166] In the case where the metadata corresponding to the metadata storage address is updated, in the metadata storage address, replacing the metadata value in the metadata stored at the storage address with the metadata value in the updated metadata;
[0167] When the shard space in the storage space is added or deleted, metadata of the user data block is added or deleted in the storage address of the metadata corresponding to the user data block in the shard space.
[0168] Optionally, the device further includes: a fourth processing module, wherein the fourth processing module is used for at least one of the following:
[0169] In the case where some user data blocks are added or deleted in the shard space, retraining the learning index instance corresponding to the shard space according to the storage address of the metadata corresponding to the added or deleted user data blocks;
[0170] When the shard space in the storage space is added or deleted as a whole, the corresponding secondary index structure is added or deleted according to the added or deleted shard space, and the primary index structure is updated.
[0171] It should be noted that the metadata index construction device described in the embodiment of the present invention is a device capable of executing the above-mentioned metadata index construction method, and all embodiments of the above-mentioned metadata index construction method are applicable to the device and can achieve the same or similar technical effects.
[0172] like Figure 6 As shown, an embodiment of the present invention also provides a metadata index construction device, including: a processor 601; and a memory 603 connected to the processor 601 through a bus interface 602, the memory 603 is used to store programs and data used by the processor 601 when performing operations, and the processor 601 calls and executes the programs and data stored in the memory 603.
[0173] The transceiver 604 is connected to the bus interface 602 and is used to receive and send data under the control of the processor 601. Specifically, the processor 601 is used to read the program in the memory 603 and execute the following process:
[0174] Dividing the storage space into a plurality of shard spaces, each shard space including at least one user data block;
[0175] Extracting metadata of the user data block and storing the metadata;
[0176] Constructing a learning-type secondary index structure for the shard space, the secondary index structure comprising: first identification information of the shard space and a learning index instance, the learning index instance comprising: a correspondence between a first index key of the metadata and a storage address of the metadata;
[0177] A primary index structure in the form of a hash table is constructed for the secondary index structure, wherein the primary index structure includes: a second index key of the first identification information, and a storage address of the secondary index structure.
[0178] Optionally, the metadata includes a metadata key and a metadata value:
[0179] The metadata key includes information indicating an offset of a logical address of the user data block in the shard space;
[0180] The metadata value includes at least one of the following:
[0181] Indicative information of the physical storage address of the user data block;
[0182] Size information of the user data block;
[0183] Verification information of the user data block;
[0184] Update information of the user data block;
[0185] Deletion information of the user data block.
[0186] Optionally, the secondary index structure further includes at least one of the following:
[0187] Indication information used to indicate whether the shard space is in a garbage collection state;
[0188] Indication information used to indicate the storage server of the slice space.
[0189] Optionally, the processor 601 is configured to:
[0190] Obtaining a first index key of the metadata and a storage address of the metadata;
[0191] Generate learning data according to the first index key and the corresponding storage address;
[0192] Dividing the learning data into training data and supervision data;
[0193] The learning index instance is obtained according to the training data and the supervision data using a machine learning model.
[0194] Optionally, the transceiver 604 is used for:
[0195] Receive a query request, the query request including a first index key and a second index key;
[0196] The processor 601 is further configured to:
[0197] According to the second index key and the primary index structure, query the storage address of the secondary index structure;
[0198] When the storage address of the secondary index structure is queried, the storage address of the metadata is obtained according to the first index key and the secondary index structure.
[0199] Optionally, the processor 601 is further configured to perform at least one of the following:
[0200] In the case where the metadata corresponding to the metadata storage address is updated, in the metadata storage address, replacing the metadata value in the metadata stored at the storage address with the metadata value in the updated metadata;
[0201] When the shard space in the storage space is added or deleted, metadata of the user data block is added or deleted in the storage address of the metadata corresponding to the user data block in the shard space.
[0202] Optionally, the processor 601 is further configured to perform at least one of the following:
[0203] In the case where some user data blocks are added or deleted in the shard space, retraining the learning index instance corresponding to the shard space according to the storage address of the metadata corresponding to the added or deleted user data blocks;
[0204] When the shard space in the storage space is added or deleted as a whole, the corresponding secondary index structure is added or deleted according to the added or deleted shard space, and the primary index structure is updated.
[0205] Among them, Figure 6In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 601 and memory represented by memory 603. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface 602 provides a user interface 605. The transceiver 604 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 601 is responsible for managing the bus architecture and general processing, and the memory 603 may store data used by the processor 601 when performing operations.
[0206] In addition, a specific embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, wherein the program, when executed by a processor, implements the steps in the metadata index construction method as described in any one of the above.
[0207] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0208] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0209] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0210] A specific embodiment of the present invention further provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.
[0211] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A metadata index construction method, characterized in that: include: Dividing the storage space into a plurality of shard spaces, each shard space including at least one user data block; Extracting metadata of the user data block and storing the metadata; Constructing a learning-type secondary index structure for the shard space, the secondary index structure comprising: first identification information of the shard space and a learning index instance, the learning index instance comprising: a correspondence between a first index key of the metadata and a storage address of the metadata; A primary index structure in the form of a hash table is constructed for the secondary index structure, wherein the primary index structure includes: a second index key of the first identification information, and a storage address of the secondary index structure.
2. The method according to claim 1, characterized in that The metadata includes metadata keys and metadata values: The metadata key includes information indicating an offset of a logical address of the user data block in the shard space; The metadata value includes at least one of the following: Indicative information of the physical storage address of the user data block; Size information of the user data block; Verification information of the user data block; Update information of the user data block; Deletion information of the user data block.
3. The method according to claim 1, characterized in that: The secondary index structure also includes at least one of the following: Indication information used to indicate whether the shard space is in a garbage collection state; Indication information used to indicate the storage server of the slice space.
4. The method according to claim 1, characterized in that: The learning index instance is obtained in the following way: Obtaining a first index key of the metadata and a storage address of the metadata; Generate learning data according to the first index key and the corresponding storage address; Dividing the learning data into training data and supervision data; The learning index instance is obtained according to the training data and the supervision data using a machine learning model.
5. The method according to claim 1, characterized in that The method further comprises: Receive a query request, the query request including a first index key and a second index key; According to the second index key and the primary index structure, query the storage address of the secondary index structure; When the storage address of the secondary index structure is queried, the storage address of the metadata is obtained according to the first index key and the secondary index structure.
6. The method according to claim 5, characterized in that The method further comprises at least one of the following: In the case where the metadata corresponding to the metadata storage address is updated, in the metadata storage address, replacing the metadata value in the metadata stored at the storage address with the metadata value in the updated metadata; When the shard space in the storage space is added or deleted, metadata of the user data block is added or deleted in the storage address of the metadata corresponding to the user data block in the shard space.
7. The method according to claim 1, characterized in that The method further comprises at least one of the following: In the case where some user data blocks are added or deleted in the shard space, retraining the learning index instance corresponding to the shard space according to the storage address of the metadata corresponding to the added or deleted user data blocks; When the shard space in the storage space is added or deleted as a whole, the corresponding secondary index structure is added or deleted according to the added or deleted shard space, and the primary index structure is updated.
8. A metadata index construction device, characterized in that: include: A first processing module, configured to divide a storage space into a plurality of shard spaces, wherein the shard space includes at least one user data block; A second processing module, used for extracting metadata of the user data block and storing the metadata; A first construction module is used to construct a learning type secondary index structure for the shard space, wherein the secondary index structure includes: first identification information of the shard space and a learning index instance, wherein the learning index instance includes: a correspondence between a first index key of the metadata and a storage address of the metadata; The second construction module is used to construct a primary index structure in the form of a hash table for the secondary index structure, wherein the primary index structure includes: a second index key of the first identification information, and a storage address of the secondary index structure.
9. A metadata index construction device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the metadata index construction method according to any one of claims 1 to 7.
10. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by a processor, the steps in the metadata index construction method according to any one of claims 1 to 7 are implemented.
11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the metadata index construction method according to any one of claims 1 to 7.
Citation Information
Cited By
DEM file processing method, electronic equipment and storage medium
CN119621668A
Intelligent document retrieval and generation system based on metadata driving
CN120104624A
Data storage method and apparatus, data query method and apparatus, and electronic device
WO2026130297A1
Memory sharing method and apparatus
WO2026218958A1
