Metadata query method and apparatus, and device

Abstract

A metadata query method and apparatus, and a device. In the present application, a data processing apparatus is connected to a host by means of a bus, a model layer of a hierarchical index is stored in the data processing apparatus, and the model layer comprises a target intermediate node. The hierarchical index is an index of metadata in a memory of the host. The target intermediate node is a node in the hierarchical index that indicates the relationship between an identifier of target data and a storage address of metadata of the target data. The data processing apparatus uses the identifier of the target data to search for the target intermediate node, so as to determine the storage address. The data processing apparatus accesses the memory on the basis of the storage address, so as to acquire the metadata. The data processing apparatus replaces a processor in the host to execute a metadata query operation, and the metadata query operation no longer consumes the computing power of the processor. The target intermediate node is stored in the data processing apparatus, thereby ensuring that the data processing apparatus quickly determines the storage address of the metadata in the memory of the host, and further improving the metadata query efficiency.

Description

Metadata query method, device and equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on March 28, 2024, with application number 202410377391.2 and application name "A metadata query method, device and equipment", the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of communication technology, and in particular to a metadata query method, apparatus, and device. Background Art

[0004] In some data access scenarios or some data processing scenarios, in order to access or process data, it is necessary to first query the metadata of the data to obtain the storage address of the data, and then perform data access operations on the storage address of the data or perform data processing operations after obtaining the data from the storage address of the data. Therefore, querying the metadata of the data is a key step in the data access or data processing process. Usually, the query operation of the metadata of the data is performed by the processor in the computing device. If the processor needs to perform a large number of data access operations or data processing operations, this means that the processor needs to complete a large number of metadata query operations, which not only occupies the computing power of the processor, but also affects the data access efficiency and data processing efficiency. Summary of the Invention

[0005] The embodiments of the present application provide a metadata query method, apparatus, and device for reducing the computing power occupied by a metadata query operation on a processor.

[0006] In a first aspect, embodiments of the present application provide a metadata query method, which is applied to a data processing device connected to a host computer via a bus. The data processing device stores a hierarchical index model layer, which includes a target intermediate node. The hierarchical index is an index of metadata in a host computer memory. The target intermediate node is a node in the hierarchical index that indicates a relationship between an identifier of target data and a storage address of the target data's metadata.

[0007] After obtaining the identifier of the target data, the data processing device uses the identifier of the target data to search for the target intermediate node and determine the storage address of the target data's metadata in the host's memory. After determining the storage address, the data processing device accesses the host's memory based on the storage address and obtains the metadata from the host's memory.

[0008] Through the above method, the data processing device performs metadata query operations instead of the processor in the host. The metadata query operation no longer consumes the computing power of the processor, avoiding the situation where the processor needs to perform a large number of metadata query operations, effectively improving the efficiency of metadata query. In addition, the target intermediate node is stored in the data processing device, ensuring that the data processing device can quickly determine the storage address of the metadata in the host's memory, further improving the efficiency of metadata query.

[0009] In one possible implementation, the hierarchical index includes a target leaf node, which includes metadata and a data status identifier. The data processing device may use the storage address determined by the target intermediate node as the storage address of the target leaf node in the host memory, and the data status identifier is used to indicate whether the metadata is the latest metadata.

[0010] Through the above method, the target leaf node contains not only metadata but also a data status identifier, which makes it easy to determine the status of the metadata through the data status identifier.

[0011] In one possible implementation, when the data processing device retrieves metadata from the host's memory, it may first read the data status identifier. If the data status identifier indicates that the metadata is the latest metadata, the data processing device retrieves the metadata from the host's memory. If the data status identifier indicates that the metadata is not the latest metadata, the data processing device may suspend the retrieval of metadata from the host's memory until the data status identifier indicates that the metadata is the latest metadata.

[0012] Through the above method, by identifying the data status identifier, it is ensured that the metadata acquired by the data processing device is the latest metadata, and it is ensured that the metadata read by the data processing device is the correct metadata.

[0013] In one possible implementation, the hierarchical index includes a leaf layer at the bottom and at least one model layer above the leaf layer, wherein the leaf layer includes a plurality of leaf nodes, and the plurality of leaf nodes include a target leaf node. Each leaf node includes metadata of the data, for example, the target leaf node includes metadata of the target data. Each model layer includes one or more intermediate nodes, and the intermediate nodes in at least one model layer include a target intermediate node. Any intermediate node is used to indicate the relationship between the identifier of the data and the storage address of the node in the next layer of the model layer, wherein the node in the next layer is a leaf node or an intermediate node. For example, if the intermediate node is in a model layer adjacent to the leaf layer, the intermediate node indicates the relationship between the identifier of the data and the storage address of the leaf node. If the model layer where the intermediate node is located is not adjacent to the leaf layer, the intermediate node indicates the relationship between the identifier of the data and the storage address of the intermediate node in the next model layer.

[0014] Through the above method, the hierarchical index is a simple layered structure. Starting from the top level of the hierarchical index, the intermediate nodes of the previous level can be used to determine the nodes in the next level until the leaf nodes in the leaf level are determined. This structure facilitates the rapid positioning of leaf nodes and enables fast query of metadata.

[0015] In one possible implementation, the hierarchical index is formed by sequentially performing a sampling operation, a clustering operation, and a model building operation.

[0016] The sampling operation is used to sample the metadata set in the memory to obtain multiple metadata samples. The clustering operation is used to cluster the multiple metadata samples to form multiple subsets, each subset including some metadata samples from the multiple metadata samples. The model construction is used to construct intermediate nodes based on the metadata samples included in each subset.

[0017] The sampling operation in this method ensures that the metadata in the metadata collection is pre-configured with a storage address when constructing the hierarchical index. This means that once the intermediate nodes in the model layer are constructed, the relationship between the data identifier and the storage address of the metadata for that data is determined. This storage address is the pre-configured storage address for the metadata.

[0018] In a possible implementation, the model building operation further includes: for any metadata in the metadata set except the metadata sample, determining the storage address of the leaf node according to the identifier of the metadata and the intermediate node, and storing the metadata at the storage address.

[0019] This method further demonstrates the principle of constructing hierarchical indexes by first building intermediate nodes and then writing metadata. In other words, once the model layer is determined, it is only necessary to use the intermediate nodes to determine the storage address and complete the metadata writing.

[0020] In a possible implementation, the intermediate node includes a linear regression model, which indicates the relationship between the identifier of the target data and the storage address.

[0021] Through the above method, the linear regression model is simpler and can effectively speed up the metadata query process.

[0022] In one possible implementation, a data processing device may receive a data message sent by an external device or a processor in a host computer. The data message carries an identifier of target data, and the data processing device may obtain the identifier of the target data from the data message. The data message is used to indicate any of the following: obtaining metadata of the target data, accessing the target data, or processing the target data. In the embodiments of the present application, to facilitate distinguishing the source of the data message, the data message sent by the external device is referred to as a request message, and the data message sent by the processor is referred to as an indication message.

[0023] Through the above method, when the data processing device receives the above-mentioned data message, it determines that metadata needs to be queried, and then completes the metadata query operation. The source of the data message is different, and the specific content indicated is also different, which is applicable to different scenarios.

[0024] In one possible implementation, the data processing device can also update the saved model layer. Taking the update of the saved target intermediate node as an example, the data processing device can obtain the target intermediate node in the hierarchical index from the host's memory, and use the target intermediate node obtained from the host's memory to update the target intermediate node saved in the data processing device.

[0025] Through the above method, the data processing device updates the target intermediate node to ensure that the included target intermediate node is consistent with the target intermediate node in the memory, so as to ensure that the metadata can be accurately obtained using the target intermediate node.

[0026] In a possible implementation, the data processing device obtains metadata from the host memory based on DMA, ensuring fast acquisition of the metadata.

[0027] In a possible implementation, the target leaf node includes metadata of the target data and metadata of data adjacent to the target data.

[0028] Through the above method, when the data processing device obtains the target leaf node, in addition to being able to obtain the metadata of the target data, it can also obtain the metadata of the data adjacent to the target data. When the data adjacent to the target data needs to be read subsequently, there is no need to search for the target intermediate node again, thereby achieving the effect of data pre-fetching.

[0029] In a possible implementation, the data processing device may exist in the host in the form of a network card or an offload card.

[0030] Through the above method, the specific form of the data processing device is relatively flexible, which effectively expands the scope of application.

[0031] In a second aspect, embodiments of the present application provide a metadata query method. This method is applied to a client, which is deployed on the user side, capable of interacting with the user, and connected to the server via a network. The implementation principle of this metadata query method is similar to that of the metadata query method described in the first aspect, differing only in the execution entity. The beneficial effects can be found in the description of the first aspect and will not be further elaborated here.

[0032] In this method, a client stores a model layer in a hierarchical index, which includes a target intermediate node. The hierarchical index is an index of metadata in a server-side memory. The client uses the identifier of the target data to search for the target intermediate node stored on the client side and determines the storage address of the target data's metadata in the server-side memory. The target intermediate node includes a node of a linear regression model, which indicates the relationship between the target data identifier and the storage address. Based on the storage address, the client accesses the memory and retrieves the metadata from the memory.

[0033] In one possible implementation, when the client determines the storage address of the metadata of the target data in the memory, it determines the storage address of the target leaf node in the memory. The target leaf node hierarchical index includes nodes of metadata and data status identifiers. The data status identifier is used to indicate whether the metadata is the latest metadata.

[0034] In one possible implementation, when the client retrieves metadata from the storage, it first identifies the data status identifier in the target leaf node. If the data status identifier indicates that the metadata is the latest metadata, the client retrieves the metadata from the storage. If the data status identifier indicates that the metadata is not the latest metadata, the client suspends the retrieval of the metadata from the storage until the data status identifier indicates that the metadata is the latest metadata.

[0035] In one possible implementation, the hierarchical index includes a leaf layer at the bottom and at least one model layer above the leaf layer, the leaf layer includes multiple leaf nodes, the multiple leaf nodes include a target leaf node, each leaf node includes metadata of the data, for example, the target leaf node includes metadata of the target data; each model layer includes one or more intermediate nodes, the intermediate nodes in at least one model layer include a target intermediate node, and any intermediate node is used to indicate the relationship between the identifier of the data and the storage address of the node in the next layer of the model layer, wherein the node in the next layer is a leaf node or an intermediate node.

[0036] In one possible implementation, a hierarchical index is formed by sequentially performing a sampling operation, a clustering operation, and a model building operation. The sampling operation is used to sample a metadata set in memory to obtain multiple metadata samples. The clustering operation is used to cluster the multiple metadata samples to form multiple subsets, each of which includes a portion of the metadata samples. The model building operation is used to construct an intermediate node based on the metadata samples included in each subset.

[0037] In a possible implementation, the model building operation further includes: for any metadata in the metadata set except the metadata sample, determining the storage address of the leaf node according to the identifier of the metadata and the intermediate node, and storing the metadata at the storage address.

[0038] In a possible implementation, the client receives a request message triggered by a user, the request message carries an identifier of the target data, and the request message is used to indicate any of the following: obtaining metadata of the target data, accessing the target data, and processing the target data.

[0039] In a possible implementation, the client obtains the target intermediate node in the hierarchical index from the memory, and uses the target intermediate node obtained from the memory to update the target intermediate node stored on the client side.

[0040] In a possible implementation, the client obtains metadata from a storage on the server based on RDMA.

[0041] In a possible implementation, the target leaf node includes metadata of the target data and metadata of data adjacent to the target data.

[0042] In a third aspect, an embodiment of the present application provides a hierarchical index construction method, which can be performed by a construction device, wherein:

[0043] The construction device samples the metadata set in the memory and obtains multiple metadata contained therein, each metadata being a metadata sample. Afterwards, the construction device clusters the multiple metadata samples to form multiple subsets, each subset including part of the metadata samples.

[0044] For any subset of the multiple subsets, the construction device constructs an intermediate node in the model layer based on the metadata samples included in the subset, and the intermediate node describes the relationship between the identifier of the metadata sample in the subset and the storage address of the metadata sample.

[0045] Through the above method, a hierarchical index is formed through the operations of sampling, clustering, and model building. Once the intermediate nodes in the hierarchical index are formed, the relationship between the metadata identifier and the metadata storage address is fixed. The storage address determined by the intermediate node is a pre-configured storage address for the metadata. Therefore, the construction method of the hierarchical index reflects the principle of first building the intermediate nodes in the model layer and then writing the metadata, that is, the storage address is configured for the metadata in advance. This hierarchical index allows metadata to be added to the metadata set without changing the structure of the hierarchical index or making large-scale changes.

[0046] In the embodiments of this application, the identifier of metadata (such as the identifier of the metadata sample, the identifier of the target metadata, etc.) is the same as the identifier of the data described by the metadata. In other words, this application does not distinguish between the identifier of metadata and the identifier of data. Taking the file system as an example, the identifier of a file is the identifier of the metadata of that file.

[0047] In one possible implementation, the hierarchical index also includes a leaf layer located below the model layer, and the leaf layer includes multiple leaf nodes; each metadata sample is included in a leaf node in the leaf layer, and the storage address of the metadata sample is the storage address of the leaf node.

[0048] Through the above method, metadata samples are included in leaf nodes, and leaf nodes can be determined based on intermediate nodes. The corresponding relationship between nodes is clearer, and the structure of the hierarchical index is simplified.

[0049] In one possible implementation, for any metadata in the metadata set except the metadata sample, the construction device determines the first storage address based on the metadata identifier and the intermediate node, stores the metadata at the first storage address, and forms a leaf node of the hierarchical index.

[0050] By using the above method, when initially constructing the hierarchical index, only part of the metadata in the metadata set (i.e., metadata samples) is targeted. To ensure the validity of the hierarchical index, the storage address for the remaining metadata can also be determined according to the instructions of the intermediate nodes in the hierarchical index.

[0051] In one possible implementation, the above-described method can also be used to determine the storage address of metadata subsequently added to the metadata collection, thereby storing the metadata. Specifically, when it is necessary to add target metadata to the metadata collection, the construction device determines a second storage address based on the identifier of the target metadata and the intermediate node, and stores the target metadata at the second storage address, forming a leaf node in the hierarchical index.

[0052] Through the above method, since the hierarchical index reserves storage space for metadata in advance, it is ensured that the hierarchical index can be updated in time when metadata collection needs to be added.

[0053] In one possible implementation, if the second storage address is occupied, the construction device determines an unoccupied third storage address within the target storage address range when storing the target metadata, and stores the target metadata at the third storage address. The length of the target storage address range is equal to the data read and write granularity of the memory, and the target storage address range includes the second storage address.

[0054] Through the above method, if the second storage address is occupied, a storage address can still be allocated thereto to ensure smooth writing of metadata.

[0055] In one possible implementation, if the second storage address is already occupied, the construction device splits the intermediate node into a first intermediate node and a second intermediate node, the first intermediate node indicates the relationship between the identifier of the metadata stored at the second storage address and the second storage address, and the second intermediate node indicates the relationship between the identifier of the target metadata and the third storage address.

[0056] Through the above method, if the target metadata is stored in the third storage address, the hierarchical index can be updated. The update only splits one of the intermediate nodes into two intermediate nodes, that is, the overall structure of the hierarchical index does not change significantly, and the update method is simpler and faster.

[0057] In a possible implementation, the leaf node further includes a data status identifier, which is used to indicate whether the metadata in the leaf node is the latest metadata.

[0058] Through the above method, the existence of the data status identifier can ensure that the status of the metadata in the leaf node is accurately determined when reading the leaf node.

[0059] In one possible implementation, the construction device adjusts the data status identifier during the process of updating the metadata in the leaf node, and the adjusted data status identifier is used to indicate that the metadata in the leaf node is not the latest metadata. After the metadata in the leaf node is updated, the data status identifier is adjusted, and the adjusted data status identifier is used to indicate that the metadata in the leaf node is the latest metadata.

[0060] Through the above method, the adjustment operation of the data status identifier can ensure that when reading a leaf node, it can be clear whether the metadata in the leaf node is the latest metadata.

[0061] In one possible implementation, the intermediate node includes a linear regression model, which indicates a relationship between an identifier of the metadata sample and a storage address of the metadata sample.

[0062] Through the above method, the linear regression model is simpler and can effectively speed up metadata query efficiency.

[0063] In a fourth aspect, an embodiment of the present application further provides a data processing device, which has the function of implementing the behavior in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The functions can be implemented by hardware, or by hardware executing corresponding software implementations. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the data processing device includes a search module and an acquisition module, which can perform the corresponding functions in the method example of the first aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0064] In the fifth aspect, an embodiment of the present application further provides a query device, which has the function of implementing the client behavior in the method example of the second aspect above. The beneficial effects can be found in the description of the second aspect and will not be repeated here. The function can be implemented by hardware, or by hardware executing the corresponding software implementation. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the query device includes a node search module and a remote access module, and optionally, a receiving module. These modules can perform the corresponding functions in the method example of the second aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0065] In a sixth aspect, the present application further provides a construction device. The beneficial effects and details can be found in the relevant description of the third aspect, which is only briefly described here. The construction device includes:

[0066] The sampling module is used to sample the metadata set in the memory to obtain multiple metadata samples.

[0067] The clustering module is used to cluster multiple metadata samples to form multiple subsets, each subset including part of the metadata samples.

[0068] The model construction module is used to construct an intermediate node in the model layer for any subset of the multiple subsets based on the metadata samples included in the subset, and the intermediate node describes the relationship between the identifier of the metadata sample in the subset and the storage address of the metadata sample.

[0069] In one possible implementation, the hierarchical index also includes a leaf layer located below the model layer, and the leaf layer includes multiple leaf nodes; each metadata sample is included in a leaf node in the leaf layer, and the storage address of the metadata sample is the storage address of the leaf node.

[0070] In one possible implementation, for any metadata in the metadata set except the metadata sample, the model building module determines the first storage address based on the metadata identifier and the intermediate node, stores the metadata at the first storage address, and forms a leaf node of the hierarchical index.

[0071] In one possible implementation, when the model building module needs to add target metadata to the metadata set, it determines the second storage address based on the identifier of the target metadata and the intermediate node, stores the target metadata at the second storage address, and forms a leaf node in the hierarchical index.

[0072] In one possible implementation, if the second storage address is occupied, the model building module determines an unoccupied third storage address within the target storage address range, and stores the target metadata at the third storage address. The length of the target storage address range is equal to the data read and write granularity of the memory, and the target storage address range includes the second storage address.

[0073] In one possible implementation, if the second storage address is occupied, the model building module splits the intermediate node into a first intermediate node and a second intermediate node, the first intermediate node indicates the relationship between the identifier of the metadata stored on the second storage address and the second storage address, and the second intermediate node indicates the relationship between the identifier of the target metadata and the third storage address.

[0074] In a possible implementation, the leaf node further includes a data status identifier, which is used to indicate whether the metadata in the leaf node is the latest metadata.

[0075] In one possible implementation, the construction module further includes an update module. During the process of updating metadata in a leaf node, the update module adjusts a data status identifier. The adjusted data status identifier is used to indicate that the metadata in the leaf node is not the latest metadata. After completing the metadata update in the leaf node, the update module adjusts the data status identifier. The adjusted data status identifier is used to indicate that the metadata in the leaf node is the latest metadata.

[0076] In one possible implementation, the intermediate node includes a linear regression model, which indicates a relationship between an identifier of the metadata sample and a storage address of the metadata sample.

[0077] In a seventh aspect, the present application further provides a computing device, comprising a data processing device and a host, wherein the data processing device and the host are connected via a bus. The data processing device executes the method provided in the first aspect or any possible implementation of the first aspect.

[0078] In an eighth aspect, the present application also provides a data processing device, which includes a processing module and a power supply circuit, the power supply circuit is used to power the processing module, and the processing module executes the method provided by the above-mentioned first aspect or any possible implementation of the first aspect.

[0079] Ninth aspect, the present application provides a computing device, which includes a memory and a processor. The processor is used to access the code in the memory to execute the method provided by the second aspect or any possible implementation of the second aspect or to execute the method provided by the third aspect or any possible implementation of the third aspect. When the computing device executes the method provided by the third aspect or any possible implementation of the third aspect, the structural diagram of the computing device is similar to the structure of the computing device in Figure 14 in the specification, except that the modules included in the internal memory of the computing device are the modules in the aforementioned construction device (such as sampling modules, clustering modules, model construction modules), and the present embodiment of the application will not repeat the structure of the computing device.

[0080] In a tenth aspect, the present application provides a computer-readable storage medium. When the computer-readable storage medium is executed by a computing device, the computing device executes the method provided in the first aspect or any possible implementation of the first aspect, executes the method provided in the second aspect or any possible implementation of the second aspect, or executes the method provided in the third aspect or any possible implementation of the third aspect. The storage medium stores computer program instructions. The storage medium includes but is not limited to volatile memory, such as random access memory, non-volatile memory, such as flash memory, hard disk drive (HDD), solid state drive (SSD).

[0081] In an eleventh aspect, the present application provides a computing device program product, which includes computer program instructions. When executed by a computing device, the computing device executes the method provided in the aforementioned first aspect or any possible implementation of the first aspect, executes the method provided in the aforementioned second aspect or any possible implementation of the second aspect, or executes the method provided in the aforementioned third aspect or any possible implementation of the third aspect. The computer program product can be a software installation package. When it is necessary to use the method provided in the aforementioned first aspect or any possible implementation of the first aspect, the method provided in the aforementioned second aspect or any possible implementation of the second aspect, or the method provided in the aforementioned third aspect or any possible implementation of the third aspect, the computer program product can be downloaded and executed on the computing device.

[0082] In the twelfth aspect, the present application also provides a computer chip, which is connected to a memory, and the chip is used to read and execute computer program instructions stored in the memory, execute the method provided in the aforementioned first aspect or any possible implementation of the first aspect, execute the method in the aforementioned second aspect and each possible implementation of the second aspect, or execute the method in the aforementioned third aspect and each possible implementation of the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] 1A to 1C are schematic diagrams of a hierarchical index structure provided in an embodiment of the present application;

[0084] FIG1D is a schematic diagram of the structure of an intermediate node and a leaf node provided in an embodiment of the present application;

[0085] FIG2 is a schematic diagram of constructing a hierarchical index according to an embodiment of the present application;

[0086] 3A-3B are a flowchart of metadata writing hierarchical indexes according to an embodiment of the present application;

[0087] FIG4 is a schematic diagram of compacting intermediate nodes within a hierarchical index according to an embodiment of the present application;

[0088] FIG5 is a schematic diagram of a hierarchical index structure provided in an embodiment of the present application;

[0089] FIG6 is a flowchart of using a hierarchical index to query metadata according to an embodiment of the present application;

[0090] FIG7 is a schematic diagram of the structure of a metadata query subsystem provided in an embodiment of the present application;

[0091] 8A to 8C are schematic structural diagrams of a storage system provided in an embodiment of the present application;

[0092] FIG9 is a schematic diagram of a metadata query method provided by an embodiment of the present application;

[0093] FIG10 is a schematic diagram of the structure of a metadata query subsystem provided in an embodiment of the present application;

[0094] FIG11 is a schematic diagram of a metadata query method provided in an embodiment of the present application;

[0095] FIG12 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0096] FIG13 is a schematic diagram of the structure of a query device provided in an embodiment of the present application;

[0097] FIG14 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] Before explaining the data processing method provided by this application, the concepts involved in this application are first explained:

[0099] 1. Metadata

[0100] Metadata, also known as intermediary data or relay data, is data about data, mainly information describing data properties, such as the data's storage address, data modification history, data size, and data creation date.

[0101] Thus, when searching for a file, the file name is first used to query the first-level index to obtain the system semantic metadata of the file's parent directory. Using the obtained system semantic metadata of the parent directory, the system jumps to the location of each file mapping metadata contained in the directory. The file name is then used to query the second-level index to obtain the file mapping metadata of the file. The file's storage address is obtained from the file mapping metadata, and the file is read from the storage address.

[0102] 2. Direct memory access (DMA).

[0103] DMA is a technology that enables efficient data transfer within a computing device. DMA enables data to be transferred directly between a computing device's peripherals and its memory, without requiring the device's processor to intervene or copy data.

[0104] The peripherals of the computing device here refer to some components other than the processor of the computing device, such as an accelerator card (also called an offload card), a network card, a hard disk, a hard disk device, etc. installed on the computing device.

[0105] 3. Remote direct memory access (RDMA).

[0106] RDMA is a technology that bypasses the kernel of a remote device's operating system to access data in its memory. By not going through the operating system, it not only saves a lot of processor resources, but also improves system throughput and reduces the system's network communication latency. It is particularly suitable for widespread application in large-scale parallel computer clusters.

[0107] RDMA has several major features: (1) Data is transferred between the network and the remote device; (2) There is no involvement of the operating system kernel, and all content related to sending and transmitting is offloaded to the smart network card; (3) Data is directly transferred between the user space virtual memory and the smart network card without involving the operating system kernel, and there is no additional data movement and copying.

[0108] 4. Unilateral RDMA.

[0109] Here, the two ends that need to exchange information are referred to as the client device (can be simply referred to as the client) and the server (in this embodiment of the application, the server can be understood as a node in the storage system). The client is deployed on the user side, and the user can initiate requests to the server through the client. The server can be deployed remotely. The server generally refers to the storage system, and specifically can be understood as a node in the storage system.

[0110] One-sided RDMA can be divided into RDMA read (READ) and RDMA write (WRITE).

[0111] Taking RDMA READ in one-sided RDMA as an example, the client can directly determine the location of the data in the server's memory. Therefore, the message initiated by the client to request data read carries the data's location information and is sent to the server. On the server side, the server's network card reads the data at this location information. During this process, the server-side processor is unaware of the client's operations. In other words, the server-side processor is unaware that the client has performed a read operation, thereby reducing the processor's involvement in the data transmission process and improving the system's business processing performance, resulting in high bandwidth, low latency, and low CPU utilization.

[0112] In an embodiment of the present application, the client can read the metadata of the data from the memory in the server through unilateral RDMA to obtain the storage address of the data on the server side described by the metadata.

[0113] An index is a data structure that is ordered according to a certain pattern. It enables fast data lookups. Common indexes include hash indexes, B-trees, and B+ trees. In an embodiment of the present application, a new index structure is provided that enables index offloading. Index queries are performed by modules other than the processor, reducing processor usage and freeing up processor computing power.

[0114] First, a new index structure provided in an embodiment of the present application is introduced. For the convenience of explanation, this index structure provided in an embodiment of the present application is called a hierarchical index.

[0115] 1) The structure of hierarchical index.

[0116] Before introducing the structure of the hierarchical index, the organizational form of the "data" targeted by the hierarchical index within the hierarchical index is first explained. The "data" here refers to the data that needs to be found with the help of the hierarchical index. The "data" can be metadata or data described by metadata. In the hierarchical index, each piece of data is configured with a data identifier, and there is a corresponding relationship between the data identifier and the data. The correspondence between the data identifier and the data can exist in the form of a key-value pair, that is, the data identifier is used as the "key", and the data itself or the storage address of the data is used as the "value" corresponding to the "key". In the embodiment of the present application, the correspondence between the data identifier and the data is only explained in the form of a key-value pair. The correspondence between the data identifier and the data can also exist in other forms, and the embodiment of the present application does not limit this.

[0117] The hierarchical index has a multi-layer structure. The following describes the structure of the hierarchical index by taking a two-layer hierarchical index as an example:

[0118] As shown in Figure 1A, it is a schematic diagram of the structure of a hierarchical index with a two-layer structure. The hierarchical index with a two-layer structure includes a leaf layer and a model layer. The leaf layer includes multiple leaf nodes, and each leaf node includes a piece of data. In actual applications, each leaf node can include the piece of data and the data identifier of the piece of data.

[0119] The model layer includes one or more intermediate nodes, each intermediate node indicates the relationship between the identifier of the data and the storage address of the data. Since the data is contained in the leaf node, the intermediate node indicates the relationship between the identifier of the data and the storage address of the leaf node containing the data. Inside any intermediate node, the intermediate node includes a model, and the data identifier is input to the model. The output of the model points to a certain leaf node, and the leaf node is a leaf node that includes the data corresponding to the data identifier. For example, the output of the model is the storage address of the leaf node. It can be seen from this that for any intermediate node, the intermediate node corresponds to one or more leaf nodes in the leaf layer. In other words, based on the data identifier of different data and with the help of the model included in the intermediate node, the one or more leaf nodes can be known.

[0120] Figure 1A illustrates only the simplest hierarchical index. A hierarchical index consists of a leaf layer and at least one model layer. The leaf layer and at least one model layer can be abstracted as a hierarchically arranged, pyramid-shaped node array. The so-called "pyramid-shaped node array" means that the number of nodes in each row can vary, with the bottom layer having the largest number of nodes and the top layer having the smallest number of nodes.

[0121] Each row of nodes constitutes a leaf layer or a model layer, where the bottom layer is the leaf layer and each layer above the leaf layer is a model layer.

[0122] The nodes at the bottom layer are the nodes in the leaf layer. For ease of explanation, the nodes at the bottom layer are referred to as leaf nodes. Each leaf node contains a piece of data. The leaf node contains the data in the form of a key-value pair. The key in the key-value pair is the data identifier of the data piece, and the value is the data piece or the storage address of the data piece.

[0123] Here, "a piece of data" is related to the granularity of data division. For example, in a file system, a piece of data can be a file or metadata of a file, and the data identifier of the piece of data is the file identifier. The piece of data can also be metadata of a directory, and the data identifier of the piece of data is the directory identifier. For another example, in a data system organized at block granularity, the piece of data can be a block of data or metadata of a block of data, and the data identifier of the piece of data is the identifier of the block of data. For another example, in a data system organized at object granularity, the piece of data can be an object or metadata of the object. The data identifier of the piece of data is the object identifier.

[0124] Each model layer includes one or more nodes. In order to distinguish them from the nodes in the leaf layer, the nodes in the model layer are called intermediate nodes. For an intermediate node in any model layer, the intermediate node includes a model, a data identifier is input to the model, and the output of the model points to the node of the next layer of the model layer. When the model layer is adjacent to the leaf layer, the output of the model will point to the leaf node. When the model layer is not adjacent to the leaf layer, the output of the model points to the intermediate node in the model layer below the model layer. The embodiment of the present application does not limit the way in which the output of the model included in the intermediate node points to the node of the next layer of the model layer. For example, the output of the model is the storage address of the node of the next layer of the model layer. For another example, the output of the model is the identifier of the node of the next layer of the model layer. It can be seen that the intermediate node in any model layer corresponds to the node of the next layer of the model layer.

[0125] In this hierarchical index, starting from the top model layer, data identifiers are used to search layer by layer (that is, the data identifiers are used as the input of the model in the intermediate nodes, and the nodes of the next layer are determined according to the output of the model, until the bottom leaf layer) until the leaf node is found.

[0126] 1B and 1C exemplarily illustrate a hierarchical index having a three-layer structure and a hierarchical index having more than three layers. In the embodiment of the present application, the number of nodes in the hierarchical index is not limited, and the number of nodes in adjacent layers can be different.

[0127] After introducing the structure of the hierarchical index, the structure of the nodes in the hierarchical index will be explained below. As shown in Figure 1D, it is a structural diagram of the nodes in the model layer and leaf layer provided in an embodiment of the present application.

[0128] For the intermediate node in the model layer, the intermediate node includes a model. Optionally, the intermediate node may also include attribute information of the intermediate node, such as a node status identifier for indicating the update status of the intermediate node (in a specific implementation, the node status identifier may indicate whether the intermediate node is the latest intermediate node, that is, whether the intermediate node is the intermediate node after the most recent update), a node validity identifier for identifying whether the intermediate node is valid, and information recording the size of the intermediate node.

[0129] For leaf nodes in the leaf layer, the data (and data identifier) ​​in the leaf node exists in the form of a key-value pair. The leaf node may include only one key-value pair, that is, only one copy of the data and its data identifier. The leaf node may also include multiple key-value pairs, that is, multiple copies of the data and their data identifiers.

[0130] When the multiple copies of data included in the leaf node are multiple metadata, then the data described by the multiple metadata are adjacent data. Here, "adjacent" between data can refer to the logical address of the data being adjacent, or it can refer to the position of the multiple copies of data in the data set to which the data belongs being adjacent. For example, the data described by the multiple metadata belong to the same data table, and are multiple adjacent columns or rows in the data table. For another example, the data described by the multiple metadata belong to the same file, and the positions of the data described by the multiple metadata in the file are adjacent. When the multiple copies of data included in the leaf node are data described by metadata, then the multiple copies of data can be adjacent data.

[0131] Optionally, the leaf node may also include attribute information of the leaf node, such as a data status identifier for indicating the update status of the data included in the leaf node, which data status identifier can indicate whether the data in the leaf node is the latest data, or a node validity identifier for identifying whether the leaf node is valid.

[0132] 2) Construction of hierarchical index.

[0133] As shown in Figure 2, the following describes how to construct a hierarchical index by taking the construction of a hierarchical index for metadata as an example:

[0134] Step 0, obtain a metadata set, which includes multiple metadata, and each metadata in the metadata set is stored in the form of a key-value pair. The metadata set includes multiple metadata, and the metadata included in the metadata set is the data targeted by the hierarchical index. In actual applications, the metadata included in the metadata set will change as the data is written / read. Taking the file system as an example, the metadata included in the metadata set is the metadata in the file system. When a new file is written in the file system, the metadata of the new file needs to be added to the metadata set. When a file in the file system is modified, the metadata of the file in the metadata set also needs to be modified. When a file in the file system is deleted, the metadata related to the file needs to be deleted in the metadata set.

[0135] For any metadata in the metadata set, the metadata is stored in the form of a key-value pair. For the sake of convenience, this key-value pair is referred to as the metadata key-value pair. In any metadata key-value pair, the metadata key is understood as the data identifier of the metadata, and the metadata value is the metadata.

[0136] Step 1: Sample the metadata set to obtain multiple metadata samples.

[0137] In step 1, sampling refers to obtaining partial metadata from the metadata set, where each metadata in the obtained partial metadata is a metadata sample. In other words, the multiple metadata samples are only partial metadata in the metadata set, and the multiple metadata samples can reflect the specific distribution or data characteristics of the metadata in the metadata set, such as the size order or similarity of the metadata in the metadata set. The embodiments of the present application do not limit the method for sampling the metadata set; multiple metadata can be randomly obtained from the metadata set as metadata samples, or other sampling algorithms can be used to sample the metadata set.

[0138] Step 2: cluster the multiple metadata samples to obtain at least one subset, each subset containing one or more metadata samples, and different subsets contain different metadata samples.

[0139] By clustering the multiple metadata samples, similar metadata can be aggregated together to form a subset. The embodiment of the present application does not limit the method of clustering the multiple metadata samples. For example, when executing step 2, based on the analysis of the keys of the multiple metadata samples, multiple value ranges for the keys are set, and the metadata identified by each key within each value range constitutes a subset. In the actual implementation process, a lightweight clustering algorithm can be called based on the keys of the multiple metadata samples to achieve clustering of the multiple metadata samples. Lightweight clustering algorithms refer to some clustering algorithms with lower complexity, for example, clustering based on the distribution of metadata keys. Another example is the K-means algorithm. Of course, when clustering the multiple metadata samples, the multiple metadata samples themselves can also be directly analyzed, and based on the data features of the multiple metadata samples themselves, the multiple metadata samples can be divided into multiple data ranges for the metadata, and the metadata samples within each data range constitute a subset.

[0140] To understand it from another perspective, any subset is a subset of a metadata set, and the boundary of each subset is formed by clustering multiple metadata samples.

[0141] Step 3, for any subset, build a model based on the metadata sample pairs contained in the subset. Input the key (that is, data identifier) ​​of the metadata sample into the model, and the output of the model will point to the metadata sample. For example, the output of the model is the storage address of the metadata sample. Here, the output of the model is only taken as an example of the storage address of the metadata sample. In actual applications, the output of the model can also be the identifier of the leaf node that includes the metadata sample. In this case, it is necessary to establish a correspondence between the identifier of the leaf node and the storage address of the metadata sample, so as to determine the storage address of the metadata sample based on the output of the model and the correspondence.

[0142] The embodiments of the present application do not limit the specific type of the model. For example, if the model is a linear regression model, the linear regression model can be identified by Y=ax+b. In Figure 2, Y=a1X+b1, Y=a2X+b2, and Y=a3X+b3 respectively represent different linear regression models. For example, the model is a neural network model, or a model that can represent nonlinear functions.

[0143] Assume that, when executing step 3, the model constructed for each subset is a linear regression model, and the output of the linear regression model is the storage address of the metadata sample. However, when constructing the linear regression model, there may be a small number of invalid metadata samples. In other words, the key (i.e., data identifier) ​​of the metadata sample input to the model cannot directly point to the metadata sample. In this case, the storage address of the metadata sample is adjusted so that the storage address of the adjusted metadata sample is consistent with the storage address of the output of the linear regression model.

[0144] As we can see from the subset construction process in Step 2, the subset defines a larger range, and the metadata sample represents only a portion of the metadata within that subset. Theoretically, the model should be valid for all metadata within that subset—not just the metadata sample within the subset, but also all other metadata within the subset excluding the metadata sample fingerprint. Mathematically, the range of metadata keys within the subset corresponds to the range of the model's input values.

[0145] At this point, the intermediate nodes in the model layer adjacent to the leaf layer in the hierarchical index are formed, and the construction of the model layer adjacent to the leaf layer in the hierarchical index is complete. When there are many intermediate nodes in the model layer adjacent to the leaf layer, further model layers can be constructed above the model layer adjacent to the leaf layer. For convenience of explanation, the model layer adjacent to the leaf layer is referred to as the first model layer, and the model layer adjacent to the first model layer is referred to as the second model layer.

[0146] The model in the intermediate node in the second model layer satisfies: the key of the metadata is input to the model, the output of the model points to the intermediate node in the first model layer, and the subset applicable to the intermediate node in the first model layer contains the metadata.

[0147] Of course, if the number of intermediate nodes in the second model layer is large, the model layer can also be constructed on the second model layer. This application is an embodiment and does not limit the construction method of any model layer above the first model layer. It only needs to ensure that the model layer can be searched layer by layer using the metadata key, and finally locate an intermediate node in the first model layer, where the subset applicable to the intermediate node contains the metadata.

[0148] In the embodiment of the present application, the second model layer and the model layers above the second model layer can be constructed in the same manner as the first model layer. Taking the construction of the second model layer as an example: first, the intermediate nodes in the second model layer are sampled to obtain multiple intermediate node samples.

[0149] Afterwards, the multiple intermediate node samples are clustered to form at least one node subset; in this step, the embodiment of the present application does not limit the method of clustering the multiple intermediate node samples. For example, based on the analysis of the models within the multiple intermediate node samples, the intermediate nodes containing similar models can be clustered to form a node subset. For another example, based on the subset to which the multiple intermediate node samples are applicable (that is, the effective range of the model), multiple intermediate node samples with similar effective ranges are classified into one category to form a node subset.

[0150] Afterwards, based on any node subset, a model is constructed, and the key of the metadata is input to the model, and the output of the model will point to the intermediate node sample in the node subset. The intermediate sample node pointed to corresponds to the leaf node including the metadata, that is, the key of the metadata is input to the model in the intermediate node sample, and the output of the model in the intermediate node sample will point to the leaf node including the metadata. The embodiment of the present application does not limit the way in which the model outputs the intermediate node sample. The output of the model can be the identifier of the intermediate node sample, and there is a corresponding relationship between the identifier of the intermediate node sample and the storage address of the intermediate node sample. The output of the model can also be the storage address of the intermediate node sample.

[0151] Similar to a subset, the node subset can be understood as a subset of multiple intermediate nodes in the first model layer. The model constructed based on the node subset is not only applicable to the intermediate node samples in the node subset, but also to the intermediate nodes in the node subset other than the intermediate node samples.

[0152] The construction process for each model layer is similar, embodying a "pre-allocation concept." Taking the first model layer as an example, its construction is based only on a portion of the metadata in the metadata set (i.e., the metadata samples obtained through sampling). For the model contained in any intermediate node, on the one hand, the model adheres to the characteristics of the metadata portion itself, ensuring that the model is also applicable to other metadata with the same characteristics as the metadata portion, such as other metadata in the subset other than the metadata sample fingerprint. On the other hand, once the model is formed, it provides a valid storage address range for metadata storage (this valid storage address range is the storage space indicated by the model output). This facilitates the allocation of storage addresses based on the model output when new metadata is added to the metadata set. Alternatively, once the first model layer is established, the output range of the model within each intermediate node is fixed. This is equivalent to pre-allocating storage addresses for metadata already included in the metadata set, as well as metadata that will be added to the metadata set in the future. Therefore, this hierarchical index construction method is more suitable for scenarios where metadata sets are updated.

[0153] For example, after completing step 3, the storage address of each metadata sample obtained by sampling is determined, and the key-value pair of any metadata sample is part of a leaf node. Optionally, in addition to the key-value pair of the metadata sample, the leaf node may also contain other information, such as a data status identifier that indicates whether the metadata in the leaf node is the latest metadata, and a node validity identifier used to identify whether the leaf node is valid. Among them, whether the metadata in the leaf node is the latest metadata refers to whether the metadata is the metadata after the most recent update is completed.

[0154] For any metadata other than metadata samples in the metadata collection, the search is started layer by layer starting from the top-level model layer of the hierarchical index, and the output of the model in the intermediate node of the previous model layer is used to determine the intermediate node of the next model layer, until the model layer adjacent to the leaf layer is searched, and the key of the metadata is input into the model in the intermediate node of the model layer adjacent to the leaf layer. According to the output of the model, the storage address is determined, and the metadata and key are stored at the storage address, thereby forming a leaf node, that is, the metadata and key are part of a leaf node. In addition to the key-value pair of the metadata, the leaf node can also contain a data status identifier and a node validity identifier.

[0155] As shown in Figure 3A, it is a schematic diagram of inserting a new leaf node in a hierarchical index. In Figure 3A, a hierarchical index including a leaf layer and three model layers is taken as an example. Among them, M0 is the intermediate node in the top model layer, M1 and M2 are the intermediate nodes in the next model layer, and M3, M4, M5, M6 and M7 are the intermediate nodes in the model layer adjacent to the leaf layer. Each square in the leaf layer is a leaf node, the black square is a leaf node that already includes metadata, and the white square is a leaf node that does not include metadata.

[0156] Assume that for metadata P in a metadata set, starting from the top model layer, the key of metadata P is input into the model in M0, and the output of the model in M0 points to M2. The key of metadata P is input into the model in M2, and the output of the model in M2 points to M5. The key of metadata P is input into the model in M5, and the output of the model in M5 points to a leaf node in the leaf layer (for example, the output of the model is a certain storage address). If the leaf node pointed to is not occupied (that is, the storage address of the output of the model is not occupied), that is, the leaf node does not include a key-value pair of metadata, then the metadata P and its key are stored in the leaf node. If the leaf node pointed to is already occupied (that is, the storage address of the output of the model is occupied and metadata is already stored), that is, the leaf node already includes a key-value pair of metadata, then, an unoccupied leaf node is searched for among the leaf nodes within the storage space range that includes the leaf node and whose data length is within a set value, and the metadata P and its key are written to the unoccupied leaf node, where the set value is equal to the data read and write granularity of the memory where the hierarchical index is located. For example, an unoccupied leaf node or an unoccupied storage address can be searched for within the storage space range with the leaf node as the center and the data length being the data read and write granularity of the memory, and the metadata P can be written into the unoccupied leaf node or the unoccupied storage address (writing metadata P into the unoccupied storage address can form a new leaf node).

[0157] The data read / write granularity of a memory is the minimum amount of data read or written when a read / write operation is performed on the memory. For example, when a processor accesses the memory, the data read / write granularity is the minimum amount of data read or written by the processor when performing a read / write operation on the memory. When an external device (such as the client mentioned below) accesses the memory based on RDMA, the data read / write granularity is the minimum amount of data read or written when a read / write operation is performed on the memory based on RDMA. When a data processing device in a host accesses the host's memory based on DMA, the data read / write granularity is the minimum amount of data read or written when a read / write operation is performed on the memory based on DMA.

[0158] This method is used to determine the storage location of the metadata and form a leaf node. When the metadata needs to be read from the memory later, as long as the leaf node pointed to by the intermediate node can be determined, the data read by performing a read operation must include the leaf node containing the metadata.

[0159] Of course, for any metadata that needs to be added to the metadata set later, the above method can also be used to add a leaf node including the metadata to the hierarchical index.

[0160] 3) Splitting of intermediate nodes within the hierarchical index.

[0161] It can be seen from the construction process of the hierarchical index that the hierarchical index is an index structure built based on partial metadata samples. The hierarchical index realizes the pre-allocation of the storage address of metadata to a certain extent. The hierarchical index is the storage address allocated in advance for metadata that may be added to the metadata collection in the future, so that after the hierarchical index is established, new metadata key-value pairs can be added to it at any time.

[0162] Furthermore, when the storage addresses pre-allocated for the metadata in the metadata set in the hierarchical index are insufficient, for example, after searching layer by layer, the leaf node pointed to by the model in the intermediate node is already occupied. In this case, it is necessary to reconfigure the storage address for the metadata to write the key-value pair of the metadata at the storage address to form a new leaf node. In order to ensure that the model in the intermediate node in the model layer can point to the new leaf node, an intermediate node in the model layer is split into two intermediate nodes, and the output of the model in one of the split intermediate nodes points to the new leaf node.

[0163] As shown in Figure 3B, still taking the hierarchical index shown in Figure 3A as an example, assuming that the leaf node pointed to by the model in M5 is occupied, then it is necessary to reselect a leaf node (i.e., storage address) for the metadata P to write the key-value pair of the metadata P. Then, it is necessary to split M5 into two intermediate nodes, namely M51 and M52, wherein the model in M51 can point to the occupied leaf node, and M51 occupies the storage address of M5. Storage space (i.e., storage address) is allocated for M52, and M52 points to the new leaf node. Among them, the construction method of the model in M51 and the model in M52 is similar to the method of constructing the model in step 3 of the embodiment shown in Figure 2, that is, the metadata included in the leaf node pointed to by the model in M5 and the newly added metadata are clustered to form two categories. Based on the metadata contained in each category, a model is formed. The two models finally formed based on the metadata contained in these two categories are the model in M51 and the model in M52. Some details will not be repeated here, and specific reference can be made to the above description.

[0164] From the construction of the hierarchical index and the splitting of the intermediate nodes in the hierarchical index, it can be seen that leaf nodes and intermediate nodes can be added to the hierarchical index (splitting an intermediate node into two intermediate nodes is equivalent to adding an intermediate node). In addition to adding leaf nodes and intermediate nodes, leaf nodes and intermediate nodes can also be deleted in the hierarchical index.

[0165] Any hierarchical index is constructed for a metadata set. When the data described by the metadata is deleted, the leaf node including the metadata can be deleted in the hierarchical index. The embodiment of the present application does not limit the method of deleting the leaf node. For example, the data at the storage address where the leaf node is located is directly cleared in the hierarchical index. For another example, the value of the node validity identifier in the leaf node is updated, and the updated node validity identifier indicates that the leaf node is invalid. When any leaf node is not stored in the storage space indicated by the output range of the model in a certain intermediate node, the intermediate node can be deleted. The embodiment of the present application does not limit the method of deleting the intermediate node. For example, the data at the storage address where the intermediate node is located is directly cleared in the hierarchical index. For another example, the value of the node validity identifier in the intermediate node is updated, and the updated node validity identifier indicates that the intermediate node is invalid.

[0166] 4) Compacting the intermediate nodes in the hierarchical index.

[0167] Since the construction process of the model layer is a kind of "pre-allocation", taking the second model layer as an example, the intermediate nodes in the second model layer (such as M2, M1), the output range of the model within the intermediate node is actually the storage space reserved by the hierarchical model for the intermediate nodes in the first model layer. In addition to storing the existing intermediate nodes (such as intermediate nodes M5, M6, and M7), the reserved storage space also needs to store the intermediate nodes that may be added in the subsequent first model layer (such as the split intermediate node M52 mentioned above).

[0168] For example, in the hierarchical index shown in FIG3A , there are some unoccupied small squares between the intermediate nodes M3, M4, M5, M6, and M7. These small squares are reserved and unoccupied storage space. This storage space is used to store intermediate nodes that need to be added to the second model layer later, such as the split intermediate nodes mentioned above.

[0169] However, if these storage addresses are not occupied for a long time, storage space will be wasted. Therefore, in the embodiment of the present application, the concept of "compacting intermediate nodes" is proposed in the hierarchical index.

[0170] "Compacting intermediate nodes" means compactly storing existing intermediate nodes. Specifically, when storing these intermediate nodes, there is no free storage space between them, and the storage addresses of existing intermediate nodes are continuous. A bitmap is used to represent the positional relationship between each intermediate node and the reserved, unoccupied storage space. Analysis of this bitmap can determine the positional relationship between existing intermediate nodes, and then read intermediate nodes from existing intermediate nodes based on this positional relationship.

[0171] This bitmap consists of multiple bits that describe the occupancy of the storage space reserved for intermediate nodes within a model layer in the hierarchical index. Each bit corresponds to the storage space reserved for storing an intermediate node. The value of the bit indicates whether the intermediate node already exists in the corresponding storage space.

[0172] For any bit, when the value of the bit is the first value, it indicates that an intermediate node exists in the corresponding storage space. When the value of the bit is the second value, it indicates that no intermediate node exists in the corresponding storage space.

[0173] Therefore, when reading an intermediate node in a model layer, the bit is first queried. By analyzing the bit, the position of the intermediate node in the model layer can be determined. Then, the storage space storing the intermediate nodes in the model layer is accessed and the intermediate node is read from that position.

[0174] As shown in Figure 4 , still taking the hierarchical index shown in Figure 3A as an example, when constructing this hierarchical index, storage space is reserved for the intermediate nodes of this model layer. However, the existing intermediate nodes M5, M6, and M7 only occupy a portion of this storage space. Some storage space remains between intermediate nodes M5, M6, and M7, which is used to store new intermediate nodes that will be written later. When actually storing intermediate nodes M5, M6, and M7, they are stored closely together, meaning that the storage addresses of intermediate nodes M5, M6, and M7 are continuous. Furthermore, a bitmap is constructed, comprising multiple bits, that describes the occupancy of the storage space indicated by the output range of the model within M2. The storage space indicated by the output range of the model within M2 includes the storage space occupied by intermediate nodes M5, M6, and M7, the storage space between intermediate nodes M5, M6, and M7, and the storage space after intermediate node M7. The storage space described by this bit refers to the storage space where each intermediate node is located, as determined by the model within M2. It does not represent the actual storage address of each intermediate node. Each bit in this bitmap corresponds to the storage space reserved for storing an intermediate node. A bit value of 1 indicates that an intermediate node exists in the corresponding storage space. A bit value of 0 indicates that no intermediate node exists in the corresponding storage space.

[0175] Then, the first, third, and fifth bits in the bitmap are 1, respectively indicating that there are intermediate nodes (i.e., M5, M6, and M7) in the corresponding storage space. The remaining bits are 0, indicating that there are no intermediate nodes in the corresponding storage space.

[0176] Assuming that the storage space where M7 is located is determined by the output of the model in M2 (that is, the storage space indicated by the output range of the model in M2), the bitmap describing the occupancy of the query space is queried to determine the positional relationship between M7 and other existing intermediate nodes. For example, by querying the bitmap, it can be known that M7 is the third of the existing intermediate nodes. Therefore, when reading M7, the third intermediate node read from the starting position of the storage space indicated by the output range of the model in M2 is the M7. Usually, when designing a hierarchical index, the size of each intermediate node is the same. According to the storage address of M7 output by the model in M2, the storage address can be regarded as the storage address assigned to it by the model, which may be different from the actual storage address of M7. Therefore, it is necessary to further query the bitmap. After determining the positional relationship between M7 and other existing intermediate nodes by querying the bitmap, it can be known that the actual storage address of M7 in the storage space occupied by the intermediate nodes M5, M6, and M7 is actually stored.

[0177] This approach allows intermediate nodes to be compactly stored within the reserved storage space, while free space is concentrated and available for other data. When adding a new intermediate node, simply write it in between, before, or after the existing intermediate nodes based on their relative position, and update the bit values ​​in the bitmap.

[0178] In the above example, the granularity of the storage space represented by the bitmap is the storage space indicated by the output range of the model within a certain intermediate node. In fact, the embodiments of the present application do not limit the granularity of the storage space represented by the bitmap. In actual applications, the storage space represented by the bitmap can also be set to all the storage space reserved for a model layer in the hierarchical index.

[0179] It is worth noting that the hierarchical index is usually stored in the memory, and the query and update of the hierarchical index will involve the storage space application granularity in the memory. The so-called storage space application granularity refers to the minimum granularity set when allocating storage space for data in the memory. That is, every time data needs to be written in the memory, storage space is always allocated based on the storage space application granularity, and the allocated storage space is a multiple of the storage space application granularity. For example, if the storage space application granularity is 4 bits, then each time storage space is allocated for data, it will be allocated in 4-bit units. Similarly, the allocation of storage space is also involved in the process of building a hierarchical index. This will determine the organization of leaf nodes and intermediate nodes in the hierarchical index. The storage space application granularity is not fixed. For the memory, users or computing devices are allowed to configure it according to the actual scenario. The memory can have multiple different storage space application granularities.

[0180] In order to be applicable to the storage space application granularity, the intermediate nodes in the model layer can be aggregated together, and the intermediate nodes in the leaf nodes can also be aggregated together. Therefore, as shown in Figure 5, another structural diagram of a hierarchical index is provided for the embodiment of the present application. The structure of the hierarchical index is as follows:

[0181] The hierarchical index includes a leaf layer and at least one model layer. The leaf layer includes at least one primary leaf node, and each primary leaf node includes multiple leaf nodes. The size of each primary leaf node is equal to the storage space application granularity of the memory where the primary leaf is located. The model layer includes at least one primary intermediate node, and each primary intermediate node includes multiple intermediate nodes. The size of each primary intermediate node is equal to the storage space application granularity of the memory where the primary intermediate node is located. The top-level model layer includes a primary intermediate node, and the primary intermediate node includes an intermediate node. For the description of leaf nodes and intermediate nodes, please refer to the above content and will not be repeated here.

[0182] Optionally, in addition to the intermediate node, any primary intermediate node may also include other information; the primary intermediate node includes an intermediate node header, and the intermediate node header may include attribute information of the primary intermediate node. The attribute information of the primary intermediate node includes but is not limited to: the currently occupied storage space in the primary intermediate node, the currently unoccupied storage space in the primary intermediate node, the number of intermediate nodes currently included in the primary intermediate node, the creation time of the primary intermediate node, the update record of the primary intermediate node, the valid intermediate nodes in the primary intermediate node, and the invalid intermediate nodes.

[0183] In addition to the leaf node, any primary leaf node may also include other information; the primary leaf node includes a leaf node header, and the leaf node header may include attribute information of the primary leaf node. The attribute information of the primary leaf node includes but is not limited to: the storage space currently occupied in the primary leaf node, the storage space currently unoccupied in the primary leaf node, the number of leaf nodes currently included in the primary leaf node, the creation time of the primary leaf node, the update record of the primary leaf node, the valid leaf nodes in the primary leaf node, and the invalid leaf nodes.

[0184] In this hierarchical index, both primary leaf nodes and primary intermediate nodes are organized at the granularity of memory space allocation. Therefore, when a certain intermediate node is needed from the hierarchical index, the primary intermediate node to which the intermediate node belongs can be read. The intermediate node can then be retrieved from the read primary intermediate node. Similarly, when a certain leaf node is needed from the hierarchical index, the primary leaf node to which the leaf node belongs can be read. The leaf node can then be retrieved from the read primary leaf node.

[0185] The following describes the method of querying the hierarchical index using the hierarchical index shown in Figure 6. The hierarchical index shown in Figure 6 includes a leaf layer and three model layers. The current top-level model layer includes a primary intermediate node JM0, and the primary intermediate node JM0 includes an intermediate node M0. The model layer of the intermediate layer includes a primary intermediate node JM1, and the primary intermediate node JM1 includes two intermediate nodes M1 and M2. The bottom-level model layer includes two primary intermediate nodes, respectively marked as primary intermediate node JM2 and primary intermediate node JM3, the primary intermediate node JM2 includes two intermediate nodes M3 and M4, and the primary intermediate node JM3 includes three intermediate nodes M5, M6 and M7. The leaf layer includes multiple primary leaf nodes, and Figure 6 only shows one of the primary leaf nodes JL1. The primary leaf node includes several leaf nodes, among which a certain leaf node L1 stores metadata P. In FIG6 , each solid-line small square represents a storage space that can be used to store an intermediate node or a leaf node. The characters in the solid-line small square represent the intermediate nodes that have been written. The unfilled characters in the solid-line small square indicate that no intermediate node has been written in the square.

[0186] Taking the search for metadata P from a hierarchical index as an example, first, read the primary intermediate node JM0 at the top level of the hierarchical index, obtain the intermediate node M0 from the primary intermediate node JM0, and input the key of the metadata P into the model within the intermediate node M0. Assume that the output of the model within the intermediate node M0 points to the intermediate node M2. For example, the output of the model within the intermediate node M0 is the storage address of the intermediate node M2, and the storage address of the intermediate node M2 ​​includes the base address, the offset, and the length of the intermediate node M2. The base address is the starting storage address of the primary intermediate node JM1 where the intermediate node M2 ​​is located. Moving the length indicated by the offset on the basis of the base address is the starting storage address of the intermediate node M2.

[0187] After determining the storage address of the intermediate node M2 ​​(such as the base address included therein), read the primary intermediate node JM1 where the intermediate node M2 ​​is located in the hierarchical index, and then obtain the intermediate node M2 ​​from the primary intermediate node JM1 according to the storage address of the intermediate node M2 ​​(such as the offset and length included therein).

[0188] It should be noted that the above process of obtaining the intermediate node M2 ​​does not take into account the scenario of intermediate node compression. In the presence of intermediate node compression, after the storage address of the intermediate node M2 ​​is output from the model in the intermediate node M0, it is necessary to query the bit map, which describes the occupancy of the storage space indicated by the model in the intermediate node M0. The position of the intermediate node M2 ​​in the existing intermediate node in the primary intermediate node JM1 is determined according to the bit map, and then the actual storage address of the intermediate node M2 ​​is determined. The intermediate node M2 ​​is obtained from the primary intermediate node JM1 according to the actual storage address of the intermediate node M2. For the method of determining the actual storage address of the intermediate node in the intermediate node compression scenario, please refer to the above description and will not be repeated here.

[0189] After acquiring the intermediate node M2, the key of the metadata P is input into the model within the intermediate node M2. Assume that the output of the model within the intermediate node M2 ​​points to the intermediate node M5. Similar to the method for acquiring the intermediate node M2, when acquiring the intermediate node M5, the primary intermediate node JM3 to which the intermediate node M5 belongs is read, and the intermediate node M5 is acquired from the primary intermediate node JM3.

[0190] After acquiring the intermediate node M5, the key of the metadata P is input into the model within the intermediate node M5. The output of the model within the intermediate node M5 points to the leaf node L1. Similar to the method for acquiring the intermediate node M2, when acquiring the leaf node L1, the primary leaf node JL1 to which the leaf node L1 belongs is read, the leaf node L1 is acquired from the primary leaf node JL1, and the metadata P is then read from the leaf node.

[0191] In the hierarchical index shown in FIG6 , if the storage address of any leaf node or intermediate node includes a base address, an offset, and a length, the base address needs to be aligned with the storage space application granularity of the memory, that is, the interval between any two adjacent base addresses is equal to the storage space application granularity of the memory, so that the atomicity of the leaf node can be guaranteed. The offset in the storage address output by the model in the intermediate node is not less than or greater than zero, and is less than the storage space application granularity of the memory. If the offset in the storage address output by the model in the intermediate node is less than zero or greater than the storage space application granularity of the memory, it means that the intermediate node is unavailable and there is an error in the model in the intermediate node.

[0192] As can be seen from the above description of hierarchical indexes, they offer significant advantages. The pre-allocation principle embodied in hierarchical indexes pre-allocates storage space for the data targeted by the hierarchical index and also reserves storage space for intermediate nodes within the model layer. This facilitates the addition of data and the splitting of intermediate nodes. In other words, once a hierarchical index is constructed, if data needs to be added or intermediate nodes need to be split, the overall structure of the hierarchical index remains largely unchanged. The new data is simply written to the pre-allocated storage space, and the intermediate nodes being split are stored in the reserved storage space. Compared to common hash indexes, this hierarchical index eliminates the need for a long chain structure, making data queries within the hierarchical index more efficient. Compared to common tree-like indexes (such as B+ trees), hierarchical indexes utilize the model to form intermediate nodes, effectively reducing the number of layers within the hierarchical index and improving search efficiency.

[0193] It is worth noting that in the above description of the hierarchical index, the execution subject of constructing the hierarchical index, realizing the splitting of the intermediate nodes, and realizing the compaction of the intermediate nodes is not limited. The execution subject is the computing device where the metadata collection is located (such as the host 20 or the server 50 mentioned in the embodiment of the present application), such as the hardware component in the computing device or the software module running in the computing device. The execution subject can also be a device outside the computing device, which can configure the hierarchical index into the computing device after constructing the hierarchical index. The device can be a hardware device, such as a server, a processor, etc., or a software model. The embodiment of the present application does not limit the specific form of the device.

[0194] Since the hierarchical index has a simple structure, it is convenient to realize efficient data search, and it also provides the possibility of unloading the hierarchical index. In an embodiment of the present application, a hierarchical index is constructed for the metadata of the data to speed up the query efficiency of the metadata. In order to reduce the processor occupancy of the operation of searching the hierarchical index, the metadata query operation is unloaded to the data processing device (such as a network card), and the data processing device (such as a network card) can save part of the information in the hierarchical index, such as storing all or part of the model layers in the hierarchical index except the leaf layer in the data processing device (such as a network card). For the data processing device, the data processing device can usually query the saved intermediate node to determine the storage address of the metadata on the host by querying the model layer in the hierarchical index. The data processing device can read the metadata from the storage address to complete the metadata query operation.

[0195] The host's memory stores a complete hierarchical index, and the host's processor can promptly update the hierarchical index. The data processing device can synchronously update the portion of information stored in the hierarchical index when the host's processor updates the hierarchical index. The method by which the data processing device synchronously updates the portion of information stored in the hierarchical index will be described below and will not be further elaborated here.

[0196] Similarly, metadata query operations can be offloaded to the client, which can store partial information from the hierarchical index. For example, the client can store all or part of the model layers in the hierarchical index except for the leaf layers. The client can determine the storage address of the metadata on the server by querying the intermediate nodes in the model layers in the hierarchical index. The client can then read the metadata from this storage address to complete the metadata query operation.

[0197] A complete hierarchical index is stored on the server side, and the server processor can update the hierarchical index in a timely manner. When the processor on the server side updates the hierarchical index, the client can synchronously update part of the information in the hierarchical index stored in itself.

[0198] In an embodiment of the present application, metadata query operations are offloaded to a component other than the processor or to a module (such as a client) outside the computing device where the processor resides. The component other than the processor or the module outside the computing device where the processor resides completes the metadata query and obtains the metadata for the data. The processor does not need to search the hierarchical index to complete the metadata query, which can effectively reduce the processor's computing power and improve data access efficiency.

[0199] Before describing the metadata query method provided in the embodiments of this application, we first describe the structure of the metadata query subsystem to which this metadata query method applies. This embodiment of the application provides two different metadata query subsystems. The following describes these two different metadata query subsystems and the metadata query methods within each of these subsystems.

[0200] The first metadata query subsystem:

[0201] FIG7 is a schematic diagram of the structure of a metadata query subsystem provided in an embodiment of the present application. The metadata query subsystem 10 includes a host 20 and a data processing device 30. The host 20 is a common computing device. The host 20 includes, but is not limited to, a personal computer, a server, a mobile phone, a tablet computer, or a smart car.

[0202] The metadata query subsystem 10 can communicate with other devices outside the metadata query subsystem, receive request messages sent by other devices to the metadata query subsystem 10, and process the request messages. The present embodiment does not limit the type of request message received by the metadata query subsystem 10. The request message can be a request message for accessing data, a request message for obtaining metadata about the data, or a request message for processing the data.

[0203] Within the metadata query subsystem 10, the host computer 20 and the data processing device 30 both possess data processing capabilities. The host computer 20 is the primary processing core within the metadata query subsystem 10 and is capable of performing the data operations required by the metadata query subsystem 10. From the perspective of request message processing, the host computer 20 is capable of processing request messages received by the metadata query subsystem 10. In this embodiment of the present application, the addition of the data processing device 30 to the metadata query subsystem 10 allows the data processing device 30 to process some request messages on behalf of the host computer 20 or assist the host computer 20 in completing some of the operations required to process the request messages.

[0204] The data processing device 30 is capable of independently completing a metadata acquisition operation for the data, that is, the data processing device 30 is capable of acquiring metadata for the data based on the data identifier. If the request message is a request message for acquiring metadata, or the data operation required to process the request message includes a metadata query operation for the data, the data processing device 30 is capable of processing the request message on behalf of the host 20 (if the request message is a request message for acquiring metadata), or is capable of assisting the host 20 in completing the metadata query operation for the data in the data operation performed to process the request message.

[0205] The data processing device 30 side saves the model layer in the hierarchical index configured for metadata. The data processing device 30 uses the data identifier carried in the request message to query the model layer in the hierarchical index, and obtains the storage address of the metadata in the host by querying the intermediate nodes in the model layer, that is, determines the storage address of the leaf node including the metadata to be queried, obtains the metadata from the memory of the host 20 based on the storage address, that is, obtains the leaf node, and then obtains the metadata included in the leaf node.

[0206] The data processing device 30 is connected to the host 20. The data processing device 30 can be an external device of the host 20 or be deployed inside the host 20. For example, the data processing device 30 is located on the motherboard or backplane of the host 20. The data processing device 30 (such as the processing module 301 in the data processing device 30) exchanges data with the host 20 (such as the processor 201 in the host 20) through the bus 204. The bus 204 can be a peripheral component interconnect express (PCIe) bus, or a bus of a compute express link (CXL), a universal serial bus (USB) protocol, or other protocols.

[0207] To put it another way, the data processing device 30, as a module with data processing capabilities attached to the host 20, assumes some of the functions of the host 20. In other words, some of the functions of the host 20 are offloaded to the data processing device 30, which processes data and performs some operations on behalf of the host 20, thereby reducing the pressure on the host 20, especially the pressure on the processor 201 in the host 20, and freeing up the computing power of the processor 201. In this embodiment of the present application, the data processing device 30 performs metadata query operations on behalf of the host 20.

[0208] The embodiments of the present application do not limit the specific form of the data processing device 30. For example, the data processing device 30 can be installed in the host 20 as an offload card or an accelerator card. In another example, the data processing device 30 can be installed in the host 20 as a network card of the host 20. In addition to performing metadata query operations, the data processing device 30, as a network card, can also complete data processing based on network protocols, such as implementing data encapsulation and transmission.

[0209] Next, the internal structures of the host computer 20 and the data processing device 30 will be described.

[0210] Host 20 includes a network card 203, a processor 201, and a memory 202. Network card 203 is used to communicate with devices external to host 20. For example, an external device can send a request message to host 20 via network card 203. After processing the request message, processor 201 in host 20 then feeds back the processing result to the external device via network card 203.

[0211] In this embodiment of the present application, if data processing device 30 is installed in host 20 in the form of a network card, then data processing device 30 acts as "network card 203," enabling direct message transmission between host 20 and external devices. In other words, request messages sent by external devices to host 20 first pass through data processing device 30. After receiving the request message, data processing device 30 can parse the request message. If the request message is for metadata, or if the data operation required to process the request message includes a metadata query operation, data processing device 30 can complete the metadata query operation.

[0212] If the data processing device 30 is installed in the host 20 in a non-network card form (such as an offload card or an accelerator card), then after receiving the request message sent by the external device, the network card 203 transmits the request message to the processor 201 of the host 20. When the request message is a request message for requesting metadata or the data operation required to process the request message includes a data metadata query operation, the processor 201 can instruct the data processing device 30 to complete the data metadata query operation or instruct the completion of the data processing operation 30 requested by the request message.

[0213] Processor 201 is the computing and control core of host 20. It can be a central processing unit (CPU) or other specific integrated circuit. Processor 201 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0214] The memory 202 is used to store metadata and hierarchical indexes configured for the metadata. The memory 202 may be a dynamic random access memory (DRAM). In addition to DRAM, the memory 202 may also be other random access memories, such as static random access memory (SRAM). In addition, the memory 202 may also be a read-only memory (ROM). As for the read-only memory, for example, it may be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc. The memory 202 may also be a flash memory medium (FLASH), a hard disk drive (HDD), or a solid state drive (SSD).

[0215] This embodiment does not limit the number of memories 202. The processor 201 is connected to the memory 202 via a double data rate (DDR) bus or other types of buses. The memory 202 can be understood as the internal memory of the host 20, which is also called the main memory. The processor 201 executes some methods that the host 20 needs to execute in the embodiment shown in Figure 9 below by calling computer program instructions in the memory. The memory 202 can also be understood as a memory in the host 20 other than the internal memory, such as the memory 202 can be a persistent memory in the host 20, or a memory that the host can access and is located remotely.

[0216] The data processing device 30 includes a processing module 301 and a memory 302. The processing module 301 and the memory 302 are connected via a bus. The system bus can be a PCIe-based line, or a bus using CXL, USB, or other protocols. Although not shown, the data processing device 30 may also include a power supply circuit to provide power to the processing module 301.

[0217] Processing module 301 is the primary computing unit and core unit of data processing device 30. It performs the primary functions of data processing device 30. For example, metadata queries required by data processing device 30 can be performed by processing module 301. Processing module 301 can invoke computer program instructions stored in memory 302 to execute methods required by data processing device 30, as shown in the embodiment illustrated in FIG. 9 below. Alternatively, processing module 301 can invoke computer program instructions programmed into processing module 301 to execute methods required by data processing device 30, as shown in the embodiment illustrated in FIG. 9 below.

[0218] The embodiments of the present application do not limit the specific type of the processing module. The processing module 301 may be a data processing unit (DPU), a DSP, an ASIC, an FPGA, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. Any component with processing capabilities may serve as the processing module 301.

[0219] Memory 302 can support metadata query operations performed by processing module 301. Memory 302 is used to store data and / or computer program instructions required to perform metadata query operations. For example, memory 302 can store model layers in a hierarchical index constructed for metadata. Memory 302 can also be used to store computer program instructions required to perform metadata query operations.

[0220] The type of the memory 302 is similar to that of the memory 202 , and details thereof can be found in the above content, which will not be described again. The memory 302 is understood to be the internal memory of the data processing device 30 .

[0221] As can be seen from the description of the metadata query subsystem, the metadata query subsystem can be regarded as a computing device with an additional data processing device 30. In practical applications, the metadata query subsystem can be applied to a storage system as a node in the storage system.

[0222] The embodiments of the present application do not limit the specific type of storage system. Several storage systems applicable to the metadata query subsystem are listed below.

[0223] As shown in FIG8A , it is a schematic diagram of a system architecture provided in an embodiment of the present application, wherein the system architecture includes an application server 100 , a switch 110 , and a storage system 120 .

[0224] Users access data through applications. The computers running these applications are called "application servers." Application server 100 can be either a physical machine or a virtual machine. Physical application servers include, but are not limited to, desktop computers, servers, laptops, and mobile devices. Application servers access data from the storage system via Fibre Channel switch 110. However, switch 110 is optional; application server 100 can also communicate directly with storage system 120 over the network.

[0225] The storage system 120 shown in Figure 8A is a centralized storage system. A centralized storage system is characterized by a unified entry point through which all data from external devices must pass. This entry point is the centralized storage system's engine 121. Engine 121 is the core component of the centralized storage system, implementing many of the system's advanced functions.

[0226] As shown in Figure 8A, there are one or more controllers in the engine 121. Figure 8A uses the example of an engine containing two controllers for illustration. There is a mirror channel between controller 0 and controller 1. Then, after controller 0 writes a copy of data into its memory 124, it can send a copy of the data to controller 1 through the mirror channel, and controller 1 stores the copy in its local memory 124. In this way, controller 0 and controller 1 back up each other. When controller 0 fails, controller 1 can take over the business of controller 0. When controller 1 fails, controller 0 can take over the business of controller 1, thereby avoiding the unavailability of the entire storage system 120 due to hardware failure. When there are 4 controllers deployed in the engine 121, there is a mirror channel between any two controllers, so any two controllers back up each other.

[0227] Controller 0 can execute the metadata query method provided in the embodiments of the present application. In terms of hardware, as shown in FIG8A , controller 0 includes at least a network card 122, a processor 123, and memory 124. Network card 122 can be used to communicate with application server 100, thereby providing storage services for application server 100. Network card 122 can also be used to communicate with hard disk 125 to expand the capacity of the storage system. Through network card 122, engine 121 can connect to more hard disks 125, thereby forming a very large storage resource pool.

[0228] In this embodiment of the present application, the model layer in the hierarchical index configured for metadata is stored in network card 122. A request message initiated by an external storage system (a server or other storage system) to the storage system will first reach network card 122. If the request message is a request message for obtaining metadata, or if the data operation required to process the request message includes a data metadata query operation, network card 122 can complete the data metadata query operation or process the request message.

[0229] The processor 123 can be a CPU or other module with processing functions. The processor 123 is the main processing core inside the storage system and can maintain the storage system, such as updating the metadata of the data, completing the persistent storage of the data, backing up the data, etc. In an embodiment of the present application, the processor 123 can update the hierarchical index configured for the metadata stored in the memory 124. In addition, if the data operations required to process the request message include other operations in addition to the metadata query operation of the data, the processor 123 can obtain the metadata queried by the network card 122 from the network card 122, and use the metadata to point to the other operations.

[0230] Memory 124 refers to internal memory that directly exchanges data with the processor. Memory can be either random access memory (RAM) or ROM, such as DRAM or SCM. Memory can also include other random access memories, such as SRAM, PROM, and EPROM. Memory 124 can also be DIMMs or SSDs.

[0231] The memory 124 stores hierarchical indexes configured for metadata, and may also store computer query instructions required for the processor 123 to perform data operations.

[0232] The hardware components and software structure of controller 1 (and other controllers not shown in FIG8A ) are similar to those of controller 0 and are not described again here.

[0233] The hard disk 125 is used to provide storage resources, such as storing data, and can be a magnetic disk or other types of storage media, such as a solid state drive or a shingled magnetic recording hard disk.

[0234] The metadata query method provided in the embodiment of the present application is not only applicable to centralized storage systems, but also to distributed storage systems.

[0235] FIG8B is a schematic diagram of the system architecture of a distributed storage system provided in an embodiment of the present application. The distributed storage system includes a server cluster. The server cluster includes one or more servers 130 ( FIG8B shows three servers 130, but is not limited to two servers 130). Each server 130 can communicate with each other. A server 130 is a device with both computing and storage capabilities, such as a server or desktop computer.

[0236] The server 130 can execute the data query method provided in the embodiment of the present application. In terms of hardware, as shown in FIG8B , the server 130 includes at least a processor 132, a memory 133, a network card 134, and a hard disk 105. The processor 132, the memory 133, the network card 134, and the hard disk 105 are connected via a bus.

[0237] The functions and specific types of the processor 132, memory 133, network card 134, and hard disk 105 are similar to those of the processor 123, memory 124, network card 122, and hard disk 125 mentioned above. For details, please refer to the previous description and will not be repeated here. The difference is that in the distributed storage system, the servers 130 can communicate with each other, and communication between the servers 130 can be achieved through the network card 122 in the server 130.

[0238] The hard disk 105 is used to provide storage resources, such as storing data, and can be a magnetic disk or other types of storage media, such as a solid state drive or a shingled magnetic recording hard disk.

[0239] As shown in Figure 8C, this is a system architecture diagram of another distributed storage system provided in an embodiment of the present application. In this distributed storage system, the computing functions and storage functions required by the distributed storage system are deployed on different nodes, where nodes with computing functions are called computing nodes, and nodes with storage functions are called storage nodes.

[0240] In FIG8C , the distributed storage system includes a computing node cluster and a storage node cluster. The computing node cluster includes one or more computing nodes 140 ( FIG8C shows two computing nodes 140, but is not limited to two computing nodes 140). Each computing node 140 can communicate with each other. A computing node 140 is a computing device, such as a server, a desktop computer, or a storage array controller.

[0241] Computing node 140 has computing capabilities and can receive and process data requests from outside the storage system. If the data request is for accessing data or for processing data, computing node 140 can send a request message to storage node 150 to request that the data be retrieved from the storage node.

[0242] 8C , computing node 140 includes at least processor 142, memory 143, and network card 144. Processor 142 may be a CPU, configured to process data requests and trigger network card 144 to send request messages to storage nodes.

[0243] Memory 143 is used to store computer program instructions required by processor 142 to process data requests. It can also temporarily store data. When the data stored in memory 143 reaches a certain threshold, compute node 140 can send a request message to storage node 150 via network interface card 144 to persistently store the data. Network interface card 144 is used to communicate with storage node 150.

[0244] Any computing node 140 can access any storage node 150 in the storage node cluster via the network. The storage node cluster includes multiple storage nodes 150 (three storage nodes 150 are shown in FIG8C , but are not limited to three storage nodes 150). A storage node 150 includes one or more control units 151, a network card 154, and multiple hard disks 155. The network card 154 is used to communicate with the computing node 140. The hard disk 155 is used to store data and can be a magnetic disk or other type of storage medium, such as a solid-state drive or a shingled magnetic recording hard disk.

[0245] The functions of the control unit 151 (equivalent to the processor in the storage node 150), the network card 154 and the multiple hard disks 155 are similar to those of the aforementioned processor 142, the network card 144 and the hard disk 125. For details, please refer to the above description and will not be repeated here.

[0246] The above examples only illustrate several possible storage systems. In actual applications, a metadata server for storing metadata can be independently deployed within a storage system, and the data described by the metadata can be stored on storage nodes outside of the metadata server. Because the metadata server stores metadata, it is responsible for metadata management, including but not limited to querying, modifying, adding, and deleting metadata. Within the metadata server, metadata query operations can be offloaded to a network card or offload card within the metadata server. In other words, the structure of the metadata server is similar to that of the metadata query subsystem 10 shown in FIG7 , and metadata query operations can be offloaded to the data processing device 30.

[0247] In the above description of the storage system, the network card is used as a data processing device 30 to replace the processor in the node where the network card is located to complete the metadata query operation. In actual applications, a data processing device 30 in the form of a non-network card can also be added to the node within the storage node to replace the processor in the node to complete the metadata query operation.

[0248] The following describes a metadata query method provided by an embodiment of the present application based on the metadata query subsystem shown in FIG7 . As shown in FIG9 , a metadata query method provided by an embodiment of the present application is shown.

[0249] Step 900: A hierarchical index is stored in the memory 202 of the host 20, and the data processing device 30 stores the model layers in the hierarchical index. The hierarchical index is constructed for the metadata set. The construction method of the hierarchical index is described above and will not be repeated here. The hierarchical index can be constructed by the host 20 and stored in the memory 202 of the host 20. That is, a complete hierarchical index is stored in the memory 202 of the host 20, and the data processing device 30 only stores the model layers in the hierarchical index. For example, the data processing device 30 can store all the model layers in the hierarchical index, or it can keep some of the model layers in the hierarchical index.

[0250] The data processing device 30 obtains and saves the model layer in the hierarchical index. The embodiment of the present application does not limit the manner in which the data processing device 30 obtains the model layer in the hierarchical index. The data processing device 30 can actively obtain the model layer in the hierarchical index from the memory 202 of the host 20. For example, the data processing device 30 accesses the memory 202 of the host 20 based on DMA, and obtains the model layer in the hierarchical index from the memory 202 of the host 20. For another example, the data processing device 30 sends an index acquisition request to the processor 201 of the host 20, and the index acquisition request is used to request to obtain the model layer in the hierarchical index; after receiving the index retrieval request, the processor 201 of the host 20 can transmit the model layer in the hierarchical index to the data processing device 30. The processor 201 in the host 20 can also actively transmit the model layer in the hierarchical index to the data processing device 30. For example, after the processor 201 in the host 20 configures a hierarchical index for metadata, it can transmit the model layer in the hierarchical index to the data processing device 30, instructing the data processing device 30 to save the model layer in the hierarchical index.

[0251] It should be noted that, in the metadata query subsystem, the model layer of the hierarchical index is stored in both the memory 202 of the host 20 and the data processing device 30. If the hierarchical index includes multiple model layers, the output of the model contained in the intermediate node of any model layer other than the model layer adjacent to the leaf layer in the multiple model layers points to the intermediate node in the next model layer.

[0252] For the data processing device 30, for any model layer other than the model layer adjacent to the leaf layer among the multiple model layers stored in the data processing device 30, the output of the model included in the intermediate node in the model layer points to the intermediate node in the model layer stored in the data processing device 30. Assuming that the output of the model in the intermediate node is the storage address of the intermediate node in the next model layer, the storage address is the storage address of the intermediate node in the next model layer in the data processing device 30. Assuming that the output of the model in the intermediate node is the identifier of the intermediate node in the next model layer, the data processing device 30 also includes a correspondence between the identifier of the intermediate node and the storage address of the intermediate node, and the storage address of the intermediate node involved in the correspondence is the storage address of the intermediate node in the data processing device 30.

[0253] Among the multiple model layers stored in the data processing device 30, the model layer adjacent to the leaf layer, the output of the model included in the intermediate node in the model layer points to the leaf node, and the leaf node is located in the memory 202 of the host 20. Assuming that the model in the intermediate node outputs the storage address of the leaf node, the storage address of the leaf node is the storage address of the leaf node in the memory 202. Assuming that the model in the intermediate node outputs the identifier of the leaf node, the data processing device 30 also includes a correspondence between the identifier of the leaf node and the storage address of the leaf node, and the storage address of the leaf node involved in the correspondence is the storage address of the leaf node in the memory 202.

[0254] For the host 20, for any model layer other than the model layer adjacent to the leaf layer among the multiple model layers stored in the memory 202 of the host 20, the output of the model included in the intermediate node in the model layer points to the intermediate node in the model layer stored in the memory 202 of the host 20. Assuming that the output of the model in the intermediate node is the storage address of the intermediate node in the next model layer, the storage address is the storage address of the intermediate node in the next model layer in the memory 202 of the host 20. Assuming that the output of the model in the intermediate node is the identifier of the intermediate node in the next model layer, the memory 202 of the host 20 also includes a correspondence between the identifier of the intermediate node and the storage address of the intermediate node, and the storage address of the intermediate node involved in the correspondence is the storage address of the intermediate node in the memory 202 of the host 20.

[0255] Among the multiple model layers stored in the memory 202 of the host 20, the model layer adjacent to the leaf layer, the output of the model included in the intermediate node in the model layer points to the leaf node, and the leaf node is located in the memory 202 of the host 20. Assuming that the model in the intermediate node outputs the storage address of the leaf node, the storage address of the leaf node is the storage address of the leaf node in the memory 202. Assuming that the model in the intermediate node outputs the identifier of the leaf node, the memory 202 of the host 20 also includes a correspondence between the identifier of the leaf node and the storage address of the leaf node, and the storage address of the leaf node involved in the correspondence is the storage address of the leaf node in the memory 202.

[0256] Step 901: When the data processing device 30 determines that metadata needs to be queried, it obtains an identifier for the metadata to be queried. In this embodiment of the present application, the metadata identifier is the identifier of the data described by the metadata. This identifier is the key in the metadata key-value pair in the hierarchical index. For ease of explanation, the data described by the metadata to be queried is referred to as the target data, the metadata is the metadata of the target data, and the obtained identifier is the identifier of the target data.

[0257] There are many situations in which the data processing device 30 determines that metadata needs to be queried, and several of them are listed below:

[0258] Case 1: The data processing device 30 receives a request message sent by an external device, and the data processing device 30 determines that metadata needs to be queried according to the request message.

[0259] When data processing device 30 is installed in host 20 in the form of a network card, request messages sent by external devices to host 20 will first pass through data processing device 30. Upon receiving the request message from the external device, data processing device 30 will parse the request message and determine whether processing the request message requires performing a metadata query operation.

[0260] For example, when the request message is used to request metadata, the request message carries the identifier of the target data, and the data processing device 30 determines that processing the request message requires executing a metadata query operation.

[0261] For another example, when the request message is used to request access to target data, that is, to read the target data or write the target data, the request message carries the identifier of the target data. If the request message is used to request reading the target data, when processing the request message, it is necessary to first query the metadata of the target data, and then read the target data based on the metadata. If the request message is used to request writing the target data, when processing the request message, it is necessary to update the metadata for the target data, first query the metadata of the target data in the hierarchical index, and then update the metadata of the target data. Therefore, processing the request message requires executing a metadata query operation.

[0262] For another example, when a request message is used to request a processing operation on target data, the request message carries an identifier of the target data. For example, if the request message requests a calculation on the target data, processing the request message requires first obtaining the target data and then performing the calculation on the data. To obtain the target data, a metadata query must first be performed to obtain the metadata of the target data. Therefore, the data processing device 30 determines that processing the request message requires performing a metadata query operation.

[0263] The embodiment of the present application does not limit the specific content of the request message. Any request message that needs to obtain the metadata during the processing of the request message is applicable to the embodiment of the present application.

[0264] Case 2: The data processing device 30 receives an instruction message from the processor 201 in the host 20, which is used to instruct to query metadata of target data, or to access target data, or to perform a processing operation on target data. The instruction message carries the identifier of the target data.

[0265] For example, if data processing device 30 is installed on host 20 in a non-network card format, a request message sent by an external device to host 20 will first pass through the network card in host 20 and reach processor 201 in host 20. Processor 201 parses the request message. If processing the request message requires performing a metadata query operation, an example of performing a metadata query operation for processing the request message can be found in the aforementioned content and will not be repeated here. Processor 201 can send an instruction message to data processing device 30 to instruct it to query metadata. Processor 201 can also transmit the request message as an instruction message to data processing device 30, and data processing device 30 will process the request message.

[0266] For another example, when the processor 201 in the host 20 needs to process the stored data in order to maintain the data in the hard disk or memory 202 (such as the processor 201 in the host 20 needs to back up, compress, and delete the stored data), the processor 201 in the host 20 can send an indication message carrying the identifier of the target data, and instruct the data processing device 30 to query the metadata of the data to be processed or directly instruct the data processing device 30 to process the target data through the indication message.

[0267] Step 902: The data processing device 30 queries the model layer of the stored hierarchical index according to the identifier of the target data, and determines the storage address of the metadata (ie, the storage address of the target leaf node including the metadata of the target data).

[0268] After obtaining the identifier of the target data (i.e., the key of the metadata), the data processing device 30 queries the model layer of the stored hierarchical index, and uses the identifier of the data to start from the top model layer, and jump layer by layer according to the output of the model included in the intermediate node in the model layer, until it reaches the last model layer (i.e., the model layer adjacent to the leaf layer), and determines the storage address of the metadata of the target data according to the output of the model included in the target intermediate node in the model layer, that is, determines the storage address of the target leaf node including the metadata of the target data. The method of querying the model layer in the hierarchical index using the key of the metadata can be found in the above description of the hierarchical index, which will not be described in detail here.

[0269] When the output of the model included in the intermediate node in the model layer is the storage address of the target leaf node of the leaf layer in the memory 202 of the host 20, the data processing device 30 can directly obtain the output of the model included in the intermediate node in the model layer.

[0270] When the output of the model included in the intermediate node in the model layer is the identifier of the target leaf node, the data processing device 30 determines the storage address of the target leaf node including the metadata using the identifier of the target leaf node output by the model based on the correspondence between the leaf node number and the storage address of the leaf node.

[0271] Step 903: The data processing device 30 accesses the memory 202 of the host 20, and obtains the target leaf node from the memory 202 of the host 20 through DMA according to the storage address of the target leaf node.

[0272] The data processing device 30 obtains the storage address of the target leaf node, that is, it can clearly determine the location of the target leaf node in the memory 202 of the host 20. The data processing device 30 can obtain the target leaf node from the memory 202 of the host 20 through DMA.

[0273] Since the target leaf node includes the metadata, when the data processing device 30 obtains the target leaf node from the memory 202 of the host 20 via DMA, there may be a situation where the processor 201 in the host 20 needs to update the metadata included in the target leaf node.

[0274] If the processor 201 in the host 20 obtains the target leaf node from the memory 202 of the host 20 via DMA before the metadata included in the target leaf node is updated, the metadata included in the target leaf node obtained by the data processing device 30 is not updated metadata, and is not the latest metadata. If the processor 201 in the host 20 obtains the target leaf node from the memory 202 of the host 20 via DMA while the processor 201 in the host 20 is in the process of updating the metadata included in the target leaf node, the metadata included in the target leaf node obtained by the data processing device 30 may be metadata that is not fully updated and is also not the latest metadata.

[0275] In order to ensure that the metadata included in the target leaf node obtained by the data processing device 30 from the memory 202 of the host 20 is the latest metadata, or to ensure that the data processing device 30 can accurately determine whether the metadata included in the target leaf node obtained from the memory 202 of the host 20 is the latest metadata.

[0276] In addition to the metadata and its corresponding identifier, the target leaf node may also include a data status identifier for indicating whether the metadata is the latest metadata.

[0277] There are many ways in which the data status identifier indicates whether metadata is up to date. Here are a few examples:

[0278] First, the specific value of the data status identifier represents the update status of the current metadata, such as not updated, updated, or updating.

[0279] By identifying the specific value of the data status identifier, the update status of the current metadata can be determined, and then it can be determined whether the metadata included in the current target leaf node is the latest metadata.

[0280] For example, when the data status identifier is equal to the first value, it indicates that the metadata in the target leaf node does not need to be updated. When the data status identifier is equal to the second value, it indicates that the metadata in the target leaf node is in a state to be updated, that is, the metadata needs to be updated, but the metadata has not been updated yet. When the data status identifier is equal to the third value, it indicates that the metadata in the target leaf node is in an update state, that is, the metadata needs to be updated, and the metadata is being updated. When the data status identifier is equal to the fourth value, it indicates that the metadata in the target leaf node is in an update-completed state, that is, the metadata needs to be updated, and the metadata has been updated.

[0281] When the data processing device 30 obtains the target leaf node from the memory 202 of the host 20 via DMA, it first determines the value of the data status identifier in the target leaf node. If the data status identifier is the first value or the fourth value, the metadata in the target leaf node obtained by the data processing device 30 is the latest metadata. If the data status identifier is the second value or the third value, the currently obtained target leaf node is discarded, and attempts to obtain the target leaf node from the memory 202 of the host 20 via DMA continue until the data status identifier is identified as the fourth value. At this time, the metadata in the target leaf node obtained by the data processing device is the latest metadata.

[0282] Second, the specific value of the data status identifier directly indicates whether the current metadata is the latest metadata.

[0283] For example, when the data status identifier is equal to the first set value, it indicates that the metadata in the target leaf node is the latest metadata. When the data status identifier is equal to the second set value, it indicates that the metadata in the target leaf node is not the latest metadata, such as the metadata currently needs to be updated or is being updated.

[0284] When the data processing device 30 obtains the target leaf node from the memory 202 of the host 20 via DMA, it first determines the value of the data status identifier in the target leaf node. If the data status identifier is the first set value, the metadata in the target leaf node obtained by the data processing device 30 is the latest metadata. If the data status identifier is the second set value, the currently obtained target leaf node is discarded, and attempts to obtain the target leaf node from the memory 202 of the host 20 via DMA continue until it is recognized that the data status identifier is the second set value. At this time, the metadata in the target leaf node obtained by the data processing device is the latest metadata.

[0285] For another example, the data status identifier in the target leaf node is initially an even number. When the processor 201 in the host 20 needs to update the metadata in the target leaf node, it increments the data status identifier in the target leaf node by one, and the data status identifier in the target leaf node becomes an odd number. After the processor 201 in the host 20 completes the metadata update in the target leaf node, it increments the data status identifier in the target leaf node by one, and the data status identifier in the target leaf node becomes an even number.

[0286] When the data processing device 30 obtains the target leaf node from the memory 202 of the host 20 via DMA, it first determines the value of the data status identifier within the target leaf node. If the data status identifier is an even number, the metadata within the target leaf node obtained by the data processing device 30 is the latest metadata. If the data status identifier is an odd number, the currently obtained target leaf node is discarded, and attempts to obtain the target leaf node from the memory 202 of the host 20 via DMA continue until it is recognized that the data status identifier becomes an even number. At this time, the metadata within the target leaf node obtained by the data processing device is the latest metadata.

[0287] The third type, the specific value of the data status identifier represents the updated version of the metadata.

[0288] Each time the metadata in the target leaf node is updated, the data status identifier is incremented by one. Assume that the initial value of the data status identifier in the target leaf node is an even number, such as 0. When the data processing device 30 obtains the target leaf node from the memory 202 of the host 20 via DMA, it first determines the value of the data status identifier in the target leaf node. After reading the target leaf node, the data processing device 30 obtains the target leaf node from the memory 202 of the host 20 via DMA again, and determines the data status identifier in the target leaf node. If the data status identifier in the target leaf node in the memory 202 of the current host 20 is the same as the value of the data status identifier identified before the data processing device 30 reads the target leaf node, it means that the metadata included in the target leaf node read by the data processing device 30 from the memory 202 of the host 20 via DMA is the latest metadata, and the target leaf node has not been updated during the process of reading the target leaf node. If the data status identifier in the target leaf node in the memory 202 of the current host 20 is different from the data status identifier value identified before the data processing device 30 reads the target leaf node, it means that in the process of the data processing device 30 reading the target leaf node from the memory 202 of the host 20 through DMA, the processor 201 of the host 20 updates the target leaf node, and the metadata included in the target leaf node read by the data processing device 30 is not the latest metadata. The data processing device 30 can read the target leaf node from the memory 202 of the host 20 again until the metadata identifier included in the target leaf node in the memory 202 of the host 20 before and after reading the target leaf node is the same, then it can be determined that the target leaf node currently being read is the latest metadata.

[0289] Step 904: The data processing device 30 reads the metadata from the target leaf node.

[0290] After the data processing device 30 obtains the target leaf node, if the target leaf node only includes one metadata, the data processing device 30 may read the metadata from the target leaf node.

[0291] If the target leaf node includes multiple metadata (that is, the target leaf node includes key-value pairs of the multiple metadata), for example, the target leaf node includes metadata of adjacent data and its corresponding identifier, the data processing device 30 can use the identifier of the target data to obtain the metadata from the target leaf node. For other metadata in the target leaf node except the metadata, the data processing device 30 can temporarily save the other metadata so that when the other metadata needs to be read later, the data processing device 30 can directly obtain the saved other metadata without querying the model layer in the hierarchical index, thereby achieving the effect of "data prefetching".

[0292] After obtaining the metadata, the data processing device 30 may perform subsequent operations. The following describes the subsequent operations performed by the data processing device 30 after obtaining the metadata using the two cases listed in step 901:

[0293] Corresponding to the first situation, the data processing device 30 receives a request message sent by an external device, and the data processing device 30 determines that metadata needs to be queried according to the request message.

[0294] When the request message is used to request metadata, the data processing device 30 may directly feed back the metadata to the external device after obtaining the metadata.

[0295] When the request message is used to request access to the target data, that is, the target data needs to be read or written, the data processing device 30 can access the target data directly based on the metadata after obtaining the metadata. The data processing device 30 can also transmit the request message and the queried metadata to the processor 201 in the host 20, and the processor 201 in the host 20 accesses the target data according to the metadata.

[0296] When the request message is used to request processing of the target data, the data processing device 30, after obtaining the metadata, can directly obtain the target data based on the metadata and perform processing operations on the target data. The data processing device 30 can also transmit the request message and the queried metadata to the processor 201 in the host 20. The processor 201 in the host 20 reads the target data based on the metadata and performs processing operations on the target data.

[0297] Corresponding to the second situation, the data processing device 30 receives an instruction message from the processor 201 in the host 20 , where the instruction message is used to instruct to query metadata, or to access target data, or to perform a processing operation on the target data.

[0298] When the indication message is used to request obtaining metadata, the data processing device 30 may feed back the obtained metadata to the processor 201 of the host 20 after obtaining the metadata.

[0299] When the indication message is used to request access to the target data, that is, to read or write the target data, the data processing device 30 can directly access the target data based on the metadata after obtaining the metadata. After accessing the data, the data processing device 30 returns a first response message to the processor 201 of the host 20. The first response message is used to feedback the read target data or indicate that the target data was successfully written.

[0300] When the indication message is used to request a processing operation on the target data, the data processing device 30, after obtaining the metadata, can directly obtain the target data based on the metadata and perform the processing operation on the target data. After accessing the data, the data processing device 30 returns a second response message to the processor 201 of the host 20, indicating that the target data processing was successful.

[0301] In the embodiment shown in FIG. 9 , the data processing device 30 can replace the processor 201 of the host 20 to perform metadata query operations, thereby reducing the occupancy of the processor 201 and releasing the computing power of the processor 201 .

[0302] In an embodiment of the present application, the processor 201 of the host 20 may not perform metadata query operations, but the processor 201 of the host 20 can manage metadata, and the processor 201 of the host 20 can update metadata. The update of metadata by the processor 201 of the host 20 includes but is not limited to: adding metadata, modifying metadata, and deleting metadata.

[0303] When the processor 201 of the host 20 updates the metadata, it also needs to update the hierarchical index configured for the metadata. The update of the hierarchical index includes but is not limited to: adding new leaf nodes, deleting existing leaf nodes, and splitting intermediate nodes.

[0304] For the methods of adding new leaf nodes and splitting intermediate nodes, please refer to the above description of hierarchical indexing, which will not be repeated here.

[0305] Since the processor 201 of the host 20 may update the hierarchical index, in order to ensure that the model layer in the hierarchical index stored in the data processing device 30 is the latest model layer, the data processing device 30 synchronizes the model layer of the hierarchical index. The data processing device 30 replaces the model layer or intermediate node stored in the data processing device 30 with the model layer or intermediate node obtained from the host 20 (the memory 202 of the host 20). It is ensured that the model layer of the hierarchical index stored in the data processing device 30 is consistent with the model layer of the hierarchical index stored in the memory 202 of the host 20.

[0306] The embodiment of the present application does not limit the manner in which the data processing device 30 synchronizes the model layers of the hierarchical index. Two manners are listed below:

[0307] Method 1: The data processing device 30 initiates synchronization based on the model miss rate of the hierarchical index.

[0308] A miss rate is set for each stored intermediate node in data processing device 30. The miss rate for each intermediate node indicates the probability that a leaf node cannot be retrieved using that intermediate node. The miss rate is determined based on the ratio of data identifiers that cannot be processed within a fixed time period to data identifiers that need to be processed within the fixed time period. An unprocessable data identifier refers to a leaf node for which metadata corresponding to that data identifier cannot be retrieved using that data identifier and the intermediate node.

[0309] A threshold value of the miss rate is set in the data processing device 30. If the miss rate of the model layer of a certain intermediate node in the current data processing device 30 is greater than or equal to the threshold value, the data processing device 30 synchronizes the intermediate node, all the saved model layers or the model layer where the intermediate node is located. The data processing device 30 reads the intermediate node and model layer from the memory 202 of the host 20 through DMA; or requests the processor 201 of the host 20 to obtain the intermediate node and model layer. The data processing device 30 replaces the model layer or intermediate node saved by the data processing device 30 with the model layer or intermediate node obtained from the host 20 (the memory 202 of the host 20).

[0310] Since the processor 201 of the host 20 may update the model layer of the hierarchical index, in order to ensure that the model layer of the hierarchical index read by the data processing device 30 from the memory 202 of the host 20 via DMA is the latest model layer, a node status identifier for indicating the update status of the intermediate node is included in the intermediate node, and the data processing device 30 can determine whether the read intermediate node is the latest intermediate node by identifying the value of the node status identifier. The setting principle of the value of the node status identifier and the way in which the data processing device 30 determines whether the read intermediate node is the latest intermediate node by identifying the value of the node status identifier are similar to the setting principle of the value of the data status identifier and the way in which the data processing device 30 determines whether the metadata included in the read leaf node is the latest metadata by identifying the value of the data status identifier. For details, please refer to the above description and will not be repeated here.

[0311] Method 2: The data processing device 30 periodically synchronizes the model layers of the hierarchical index.

[0312] The data processing device 30 can synchronize the model layer of the hierarchical index once every set time interval. If the time interval between the current moment and the last synchronization is equal to the set time interval, the data processing device 30 obtains the model layer of the hierarchical index from the memory 202 of the host 20 through DMA; or requests the processor 201 of the host 20 to obtain the model layer of the hierarchical index. The data processing device 30 replaces the model layer or intermediate node saved by the data processing device 30 with the model layer or intermediate node obtained from the host 20 (the memory 202 of the host 20).

[0313] Since the processor 201 of the host 20 may update the model layer of the hierarchical index, in order to ensure that the model layer of the hierarchical index read by the data processing device 30 from the memory 202 of the host 20 via DMA is the latest model layer, a node status identifier for indicating the update status of the intermediate node is included in the intermediate node, and the data processing device 30 can determine whether the read intermediate node is the latest intermediate node by identifying the value of the node status identifier. The setting principle of the value of the node status identifier and the way in which the data processing device 30 determines whether the read intermediate node is the latest intermediate node by identifying the value of the node status identifier are similar to the setting principle of the value of the data status identifier and the way in which the data processing device 30 determines whether the metadata included in the read leaf node is the latest metadata by identifying the value of the data status identifier. For details, please refer to the above description and will not be repeated here.

[0314] Method three: the host 20 actively triggers the synchronization of the model layer in the data processing device 30 .

[0315] When the hierarchical index is updated, the host 20 may send a synchronization request to the data processing device 30 to transmit the model layer or intermediate node in the updated hierarchical index to the data processing device 30 through the synchronization request.

[0316] The host 20 may also periodically send a synchronization request to the data processing device 30 to transmit the model layer or intermediate node in the current hierarchical index to the data processing device 30 through the synchronization request.

[0317] The data processing device 30 replaces the model layer or intermediate node stored in the data processing device 30 with the model layer or intermediate node in the synchronization request.

[0318] The second metadata query subsystem:

[0319] As shown in FIG10 , it is a schematic structural diagram of a metadata query subsystem provided in an embodiment of the present application. The metadata query subsystem 10 includes a client 40 and a server 50 .

[0320] The client 40 and the server 50 are deployed on different computing devices, and the client 40 and the server 50 are connected via a network.

[0321] Client 40 is deployed on the user side and can interact directly with the user. The user reads and writes data by interacting with client 40. From the perspective of client 40, client 40 can initiate a request message when triggered by the user. The information requested by the request message can be found in the description above and will not be repeated here.

[0322] In the case that the request message is a request message for requesting to obtain metadata or the data operation required to process the request message includes a metadata query operation on the data, the client 40 can process the request message on behalf of the server 50 (in the case that the request message is a request message for requesting to obtain metadata), or can assist the server 50 in completing the metadata query operation on the data in the data operation performed to process the request message.

[0323] The client 40 side stores a model layer of a hierarchical index, that is, the model layer of the hierarchical index can be stored in the memory of the computing device where the client 40 is located (such as the memory of the computing device or a persistent storage such as the hard disk of the computing device). The client 40 can use the data identifier carried in the request message to query the model layer of the hierarchical index and determine the storage address of the leaf node that includes the metadata corresponding to the data identifier. The client 40 can access the memory on the server side 50, read the leaf node from the memory on the server side 50, and then obtain the metadata included in the leaf node.

[0324] The embodiments of the present application do not limit the specific form of the client 40. The client 40 can be hardware, such as a computing device, or a component in the computing device (such as a processor). The client 40 can also be a software module, such as an application running on a processor, an application running in a virtual machine or container, etc.

[0325] The server 50 is the party that stores data. The memory on the server 50 side stores the data and the metadata of the data. The server 50 configures a hierarchical index for the metadata of the data. The memory can be understood as the memory on the computing device where the server 50 is located or the memory accessible to the server 50. The specific type of the memory is not limited in this application. The specific type of the memory is the same as the specific type of memory 202. For details, please refer to the above description and will not be repeated here.

[0326] The server 50 may itself have the function of querying metadata. In this embodiment of the present application, this function can be offloaded to the client 40, with the client 40 performing the metadata query operation. When the data operations required to process the request message triggered by the user include other operations in addition to the metadata query operation, the server 50 obtains the metadata from the client 40, uses the metadata to read the data, and performs other operations on the read data.

[0327] The embodiments of the present application do not limit the specific form of the server 50. The server 50 can be hardware, such as a computing device, a computing device cluster, or a component in the computing device (such as a processor). The server 50 can also be a software module, such as an application running on a processor, an application running in a virtual machine or container, etc.

[0328] As can be seen from the description of the metadata query subsystem, the metadata query subsystem can utilize the client 40 deployed on the user side to complete metadata query operations. This allows the server 50 to perform other operations besides metadata query operations, thereby reducing the computing power occupied by metadata query operations on the server 50. In actual applications, the metadata query subsystem can also be applied to storage systems. The server 50 can be regarded as a storage system or a node in the storage system. The client 40 can be a front-end program provided by the storage system to users, such as the application server or application running on the application server mentioned in Figure 8A.

[0329] The following describes a metadata query method provided by an embodiment of the present application based on the metadata query subsystem shown in FIG10 . FIG11 shows a metadata query method provided by an embodiment of the present application.

[0330] Step 1100: The server 50 side constructs a hierarchical index for the metadata in the memory. The client 40 saves the model layer in the hierarchical index. The way in which the server 50 side constructs a hierarchical index for the metadata in the memory is similar to the way in which the host constructs a hierarchical index for the metadata in the memory in step 900. Please refer to the above description for details and will not be repeated here. The way in which the client 40 saves the model layer in the hierarchical index is similar to the way in which the data processing device saves the model layer in the hierarchical index in step 900. The difference is that the interaction between the host processor and the data processing device in step 900 is implemented based on the bus 204, and the data processing device can access the host's memory through DMA. In step 1101, the client 40 and the server 50 interact based on the network, and the client 40 can access the memory on the server 50 side through unilateral RDMA. Please refer to the above description for details and will not be repeated here.

[0331] Step 1101: When the client 40 determines that it needs to query metadata, it obtains the identifier of the metadata to be queried. Here, the metadata to be queried is referred to as the metadata of the target data, and the identifier of the metadata is the identifier of the target data.

[0332] There are many situations in which the client 40 determines that it is necessary to query metadata. For example, when the request message triggered by the user is used to request metadata, the request message carries the identifier of the target data, and the client 40 determines that processing the request message requires executing a metadata query operation. For another example, when the request message triggered by the user is used to request access to the target data, that is, to read the target data or write the target data, the request message carries the identifier of the target data, and the client 40 determines that processing the request message requires executing a metadata query operation. For another example, when the request message triggered by the user is used to request processing operations on the target data, the request message carries the identifier of the target data, such as when the request message requests calculations on the target data, and the client 40 determines that processing the request message requires executing a metadata query operation. Regarding the manner in which processing a request message requires executing a metadata query operation, please refer to the relevant description in step 901 and will not be repeated here.

[0333] The embodiment of the present application does not limit the specific content of the request message. Any request message that needs to obtain the metadata during the processing of the request message is applicable to the embodiment of the present application.

[0334] Step 1102: Client 40 queries the model layer of the stored hierarchical index based on the identifier of the target data to determine the storage address of the metadata in the memory of server 50. The manner in which client 40 executes step 1102 is similar to the manner in which the data processing device executes step 902. For details, please refer to the previous description and will not be repeated here.

[0335] Step 1103: Client 40 retrieves the target leaf node from the memory on server 50 via RDMA based on the storage address. The client 40 executes step 1103 in a manner similar to the data processing device executing step 903. The difference is that in step 903, the data processing device reads the leaf node via DMA, while in step 1103, client 40 reads the leaf node via RDMA. For details, please refer to the previous description and will not be repeated here.

[0336] Step 1104: The client 40 reads the metadata from the target leaf node.

[0337] After obtaining the metadata, the client 40 may perform subsequent operations. The following uses the several request messages listed in step 1101 as examples to illustrate the subsequent operations performed by the client 40 after obtaining the metadata:

[0338] When the request message is used to request metadata, after obtaining the metadata, the client 40 may directly feed back the metadata to the user.

[0339] When the request message is used to request access to the target data, that is, the target data needs to be read or written, the client 40 can access the target data through the storage address of the target data included in the metadata via RDMA after obtaining the metadata. The client 40 can also transmit the request message and the queried metadata to the server 50, and the server 50 accesses the target data based on the metadata.

[0340] When the request message is used to request data processing, after obtaining the metadata, the client 40 can use RDMA to obtain the target data at the storage address included in the metadata and perform processing operations on the target data. The client 40 can also transmit the request message and the queried metadata to the server 50, which then reads the target data based on the metadata and performs processing operations on the target data.

[0341] Based on the same inventive concept as the method embodiment, the present embodiment further provides a data processing device for executing the method performed by data processing device 30 in the method embodiment described above. As shown in Figure 12, data processing device 1200 includes a search module 1201 and an acquisition module 1202. Specifically, in data processing device 1200, each module is connected via a communication path.

[0342] Search module 1201 is configured to use the identifier of the target data to search for a target intermediate node stored in the data processing device, and determine the storage address of the metadata of the target data in the host memory. The target intermediate node is a node in a hierarchical index that indicates the relationship between the identifier of the target data and the storage address. The hierarchical index is an index of the metadata in the host memory.

[0343] The acquisition module 1202 is configured to access the host memory based on the storage address and acquire metadata from the host memory.

[0344] As a possible implementation, when determining the storage address, the search module 1201 determines the storage address of the target leaf node in the host's memory. The target leaf node is a node in the hierarchical index that includes metadata and a data status identifier. The data status identifier is used to indicate whether the metadata is the latest metadata.

[0345] As a possible implementation, the acquisition module 1202 acquires the metadata from the memory of the host when the data status identifier indicates that the metadata is the latest metadata.

[0346] As a possible implementation, the hierarchical index includes a leaf layer at the bottom and at least one model layer above the leaf layer, wherein the leaf layer includes multiple leaf nodes, and the multiple leaf nodes include a target leaf node. Each leaf node includes metadata for the data; each model layer includes one or more intermediate nodes, and any intermediate node is used to indicate the relationship between the identifier of the data and the storage address of the node in the next layer of the model layer, wherein the node in the next layer is a leaf node or an intermediate node, and the intermediate nodes in at least one model layer include the target intermediate node.

[0347] As a possible implementation, the hierarchical index is formed through sampling operations, clustering operations, and model building operations in sequence.

[0348] The sampling operation is used to sample the metadata set in the memory to obtain multiple metadata samples. The clustering operation is used to cluster the multiple metadata samples to form multiple subsets, each subset including a portion of the metadata samples. The model construction is used to construct intermediate nodes based on the metadata samples included in each subset.

[0349] As a possible implementation, the model building operation further includes: for any metadata in the metadata set except the metadata sample, determining the storage address of the leaf node according to the identifier of the metadata and the intermediate node, and storing the metadata at the storage address.

[0350] As a possible implementation, the intermediate node includes a linear regression model, which indicates the relationship between the identifier of the target data and the storage address.

[0351] As a possible implementation, the acquisition module 1202 can also receive a data message sent by an external device or a processor in a host, where the data message carries an identifier of the target data, and the data message is used to indicate any of the following: obtaining metadata of the target data, accessing the target data, and processing the target data.

[0352] As a possible implementation, the acquisition module 1202 acquires the target intermediate node in the hierarchical index from the memory of the host, and uses the target intermediate node acquired from the memory of the host to update the target intermediate node stored in the data processing device.

[0353] As a possible implementation, the acquisition module 1202 acquires metadata from a memory of the host based on DMA.

[0354] As a possible implementation, the target leaf node includes metadata of the target data and metadata of data adjacent to the target data.

[0355] Based on the same inventive concept as the method embodiment, the present application also provides a query device for executing the method performed by client 40 in the method embodiment. As shown in Figure 13, query device 1300 includes a node search module 1301 and a remote access module 1302. Specifically, in query device 1300, each module is connected via a communication path.

[0356] Node search module 1301 is configured to use the identifier of the target data to search for a target intermediate node stored on the client side, and determine the storage address of the metadata of the target data in the memory of the server side. The target intermediate node is a node in the hierarchical index that includes a linear regression model. The linear regression model indicates the relationship between the identifier of the target data and the storage address. The hierarchical index is an index of the metadata in the memory of the server side.

[0357] The remote access module 1302 is configured to access the memory based on the storage address and obtain metadata from the memory.

[0358] As a possible implementation, the node search module 1301 determines the storage address of the target leaf node in the memory when determining the storage address. The target leaf node hierarchical index includes nodes with metadata and data status identifiers. The data status identifier is used to indicate whether the metadata is the latest metadata.

[0359] As a possible implementation, when the remote access module 1302 obtains metadata from the memory, if the data status identifier indicates that the metadata is the latest metadata, the remote access module 1302 obtains the metadata from the memory.

[0360] As a possible implementation, the hierarchical index includes a leaf layer at the bottom layer and at least one model layer above the leaf layer, the leaf layer includes multiple leaf nodes, the multiple leaf nodes include a target leaf node, and each leaf node includes metadata of the data; each model layer includes one or more intermediate nodes, and any intermediate node is used to indicate the relationship between the identifier of the data and the storage address of the node in the next layer of the model layer, wherein the node in the next layer is a leaf node or an intermediate node, and the intermediate nodes in at least one model layer include a target intermediate node.

[0361] As a possible implementation, the hierarchical index is formed through sampling operations, clustering operations, and model building operations in sequence.

[0362] The sampling operation is used to sample the metadata set in the memory to obtain multiple metadata samples. The clustering operation is used to cluster the multiple metadata samples to form multiple subsets, each subset including a portion of the metadata samples. The model construction is used to construct an intermediate node based on the metadata samples included in each subset.

[0363] As a possible implementation, the model building operation further includes: for any metadata in the metadata set except the metadata sample, determining the storage address of the leaf node according to the identifier of the metadata and the intermediate node, and storing the metadata at the storage address.

[0364] As a possible implementation, the construction device also includes a receiving module 1303, which receives a request message triggered by a user. The data message carries an identifier of the target data, and the data message is used to indicate any one of the following: obtaining metadata of the target data, accessing the target data, and processing the target data.

[0365] As a possible implementation, the receiving module 1303 obtains the target intermediate node in the hierarchical index from the memory, and uses the target intermediate node obtained from the memory to update the target intermediate node stored on the client side.

[0366] As a possible implementation, when the remote access module 1302 obtains metadata from the memory, it obtains the metadata from the memory of the server based on RDMA.

[0367] As a possible implementation, the target leaf node includes metadata of the target data and metadata of data adjacent to the target data.

[0368] The division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0369] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for enabling a terminal device (which can be a personal computer, mobile phone, or network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0370] The present application further provides a computing device 1400 as shown in Figure 14. The computing device 1400 includes a bus 1401, a processor 1402, a communication interface 1403, and a memory 1404. The processor 1402, the memory 1404, and the communication interface 1403 communicate with each other via the bus 1401.

[0371] Among them, the processor 1402 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0372] The memory 1404 may be a dynamic random access memory (DRAM). In addition to DRAM, the memory 1404 may also be other random access memories, such as static random access memory (SRAM). In addition, the memory 1404 may also be a read-only memory (ROM). For example, the read-only memory may be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc. The memory 1404 may also be a flash memory (FLASH), a hard disk drive (HDD), or a solid state drive (SSD).

[0373] The memory 1404 stores computer program instructions, and the processor 1402 calls the computer program instructions to execute the steps performed by the client 40 in the method described in Figure 11. The memory 1404 may also include other software modules required for running processes such as an operating system (such as multiple modules of the query device 1300). The operating system may be LINUX TM ,UNIX TM ,WINDOWS TM wait.

[0374] The descriptions of the processes corresponding to the above figures have different focuses. For parts that are not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0375] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes computer program instructions that, when loaded and executed on a computer, fully or partially generate the process or functions described in FIG. 4 of the embodiment of the present invention.

[0376] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line, or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD).

[0377] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A metadata query method, characterized in that: The method is applied to a data processing device, wherein the data processing device is connected to a host via a bus, and the method comprises: searching for a target intermediate node stored in the data processing device using the identifier of the target data, and determining a storage address of metadata of the target data in a memory of the host, wherein the target intermediate node is a node in a hierarchical index indicating a relationship between the identifier of the target data and the storage address, and the hierarchical index is an index of the metadata in the memory of the host; Based on the storage address, the memory of the host is accessed, and the metadata is obtained from the memory of the host.

2. The method according to claim 1, wherein Determining a storage address of the metadata of the target data in the memory of the host includes: Determine the storage address of the target leaf node in the memory of the host, the target leaf node is a node in the hierarchical index that includes the metadata and a data status identifier, and the data status identifier is used to indicate whether the metadata is the latest metadata.

3. The method according to claim 2, wherein The acquiring the metadata from the memory of the host comprises: In a case where the data status identifier indicates that the metadata is the latest metadata, the metadata is acquired from a memory of the host.

4. The method according to any one of claims 1 to 3, wherein: The hierarchical index includes a leaf layer at the bottom and at least one model layer above the leaf layer, wherein the leaf layer includes multiple leaf nodes, and each leaf node includes metadata of the data; each model layer includes one or more intermediate nodes, and any of the intermediate nodes is used to indicate the relationship between the identifier of the data and the storage address of the node in the next layer of the model layer, wherein the node in the next layer is a leaf node or an intermediate node, and the intermediate nodes in the at least one model layer include the target intermediate node.

5. The method according to any one of claims 1 to 4, characterized in that The hierarchical index is formed by sampling, clustering and model building operations in sequence. The sampling operation is used to sample the metadata set in the memory to obtain a plurality of metadata samples; The clustering operation is used to cluster the multiple metadata samples to form multiple subsets, each subset including a portion of the metadata samples; The model building is used to build the intermediate nodes based on the metadata samples included in each subset.

6. The method according to claim 5, wherein The model building operation also includes: For any metadata in the metadata set except the metadata sample, the storage address of the leaf node is determined according to the identifier of the metadata and the intermediate node, and the metadata is stored at the storage address.

7. The method according to any one of claims 4 to 6, characterized in that The intermediate node includes a linear regression model, which indicates the relationship between the identifier of the target data and the storage address.

8. The method according to any one of claims 1 to 7, wherein: The method further comprises: A data message sent by an external device or a processor in the host is received, where the data message carries an identifier of the target data, and the data message is used to indicate any one of the following: obtaining metadata of the target data, accessing the target data, and processing the target data.

9. The method according to any one of claims 1 to 8, wherein: The method further comprises: A target intermediate node in the hierarchical index is obtained from the memory of the host, and the target intermediate node stored in the data processing device is updated using the target intermediate node obtained from the memory of the host.

10. The method according to any one of claims 1 to 9, wherein: The acquiring the metadata from the memory of the host comprises: The metadata is obtained from the memory of the host based on direct memory access (DMA).

11. The method according to any one of claims 1 to 10, wherein: The target leaf node includes metadata of the target data and metadata of data adjacent to the target data.

12. The method according to any one of claims 1 to 11, wherein: The data processing device is a network card in the host.

13. A data processing device, characterized in that: The data processing device is connected to the host via a bus, and the data processing device includes: a search module, configured to use the identifier of the target data to search for a target intermediate node stored in the data processing device, and determine a storage address of metadata of the target data in the memory of the host, wherein the target intermediate node is a node in a hierarchical index indicating a relationship between the identifier of the target data and the storage address, and the hierarchical index is an index of the metadata in the memory of the host; The acquisition module is used to: access the memory of the host based on the storage address, and acquire the metadata from the memory of the host.

14. The device according to claim 13, wherein The search module is used to: Determine the storage address of the target leaf node in the memory of the host, the target leaf node is a node in the hierarchical index that includes the metadata and a data status identifier, and the data status identifier is used to indicate whether the metadata is the latest metadata.

15. The device according to claim 14, wherein The acquisition module is used to: In a case where the data status identifier indicates that the metadata is the latest metadata, the metadata is acquired from a memory of the host.

16. The device according to any one of claims 13 to 15, characterized in that The hierarchical index includes a leaf layer at the bottom and at least one model layer above the leaf layer, wherein the leaf layer includes multiple leaf nodes, and each leaf node includes metadata of the data; each model layer includes one or more intermediate nodes, and any of the intermediate nodes is used to indicate the relationship between the identifier of the data and the storage address of the node in the next layer of the model layer, wherein the node in the next layer is a leaf node or an intermediate node, and the intermediate nodes in the at least one model layer include the target intermediate node.

17. The device according to any one of claims 13 to 16, characterized in that The hierarchical index is formed by sampling, clustering and model building operations in sequence. The sampling operation is used to sample the metadata set in the memory to obtain a plurality of metadata samples; The clustering operation is used to cluster the multiple metadata samples to form multiple subsets, each subset including a portion of the metadata samples; The model building is used to build the intermediate nodes based on the metadata samples included in each subset.

18. The device according to claim 17, wherein The model building operation also includes: For any metadata in the metadata set except the metadata sample, the storage address of the leaf node is determined according to the identifier of the metadata and the intermediate node, and the metadata is stored at the storage address.

19. The device according to any one of claims 16 to 18, characterized in that The intermediate node includes a linear regression model, which indicates the relationship between the identifier of the target data and the storage address.

20. The device according to any one of claims 13 to 19, characterized in that The acquisition module is further used to: A data message sent by an external device or a processor in the host is received, where the data message carries an identifier of the target data, and the data message is used to indicate any one of the following: obtaining metadata of the target data, accessing the target data, and processing the target data.

21. The device according to any one of claims 13 to 20, characterized in that The acquisition module is further used to: A target intermediate node in the hierarchical index is obtained from the memory of the host, and the target intermediate node stored in the data processing device is updated using the target intermediate node obtained from the memory of the host.

22. The device according to any one of claims 13 to 21, characterized in that The acquisition module is used to: The metadata is obtained from the memory of the host based on direct memory access (DMA).

23. The device according to any one of claims 13 to 22, characterized in that The target leaf node includes metadata of the target data and metadata of data adjacent to the target data.

24. A computing device, characterized in that The computing device includes a host and a data processing device; The data processing device is configured to execute the method according to any one of claims 1 to 12.

25. A data processing device, characterized in that: The data processing device includes a processing module and a power supply circuit, where the power supply circuit is used to supply power to the processing module; the processor is used to execute the method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that When the computer-readable storage medium is executed by a computing device, the computing device executes the method according to any one of claims 1 to 12.

Images