Metadata prefetching method and apparatus, storage medium, electronic device, and program

Abstract

The application discloses a metadata prefetching method and device, a storage medium, an electronic device and a program, relates to the field of distributed storage system metadata management, and comprises the following steps: acquiring a first information group corresponding to a metadata operation event in a target storage area of each node; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining a prefetching priority corresponding to any two metadata through the updated three-dimensional correlation degree; generating a hot correlation list of the metadata in each node according to the prefetching priority; and sending the hot correlation lists corresponding to the multiple nodes to the client respectively, so that the client performs prefetching on the target metadata in the multiple nodes according to the multiple hot correlation lists. Through the application, the problem that the metadata prefetching method in the distributed storage system depends on the static path correlation of the metadata and has poor cross-node adaptability, resulting in low prefetching accuracy, is solved, and the prefetching accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of metadata management in distributed storage systems, and in particular to methods and apparatuses for prefetching metadata, storage media, electronic devices, and programs. Background Technology

[0002] With the development of big data and cloud computing technologies, distributed storage systems have become the core carrier for massive data storage. The performance of distributed storage systems directly determines the efficiency of upper-layer applications (such as artificial intelligence (AI) training, scientific computing, and log analysis). In distributed storage systems, the file system is responsible for data organization and management, while metadata (including file paths, attributes, permissions, storage locations, etc.) serves as the "index" of the data, and its access performance is a key bottleneck restricting the overall system response speed. In scenarios with massive amounts of small files, the frequency of metadata access is much higher than the frequency of data access (the ratio can reach more than 10:1), and traditional centralized metadata servers (MDS) are prone to latency spikes due to excessive load.

[0003] To address the issue of latency spikes caused by excessive load, distributed file systems (MDS) achieve load balancing through dynamic subtree partitioning of the metadata server, combined with directory and index node (inode) prefetching to reduce client interaction. However, this approach has the following limitations: the prefetching strategy relies on static path associations (such as directory hierarchy) and does not consider the dynamic context of file access (such as process associations and time series); association degree calculation is based solely on access frequency, ignoring spatiotemporal decay characteristics (such as recent accesses should have higher weight than historical accesses); and the association data of multiple MDS nodes in a distributed environment is not coordinated, leading to cross-node prefetching failure. In other words, related technologies rely on static path associations of metadata in distributed storage systems, have poor cross-node adaptability, and suffer from low prefetching accuracy.

[0004] Therefore, the problem that the prefetching method for metadata in a distributed storage system relies on static path associations of metadata and has poor cross-node adaptability, resulting in low prefetching accuracy, has not yet been effectively solved. Summary of the Invention

[0005] This application provides a method and apparatus for prefetching metadata, a storage medium, an electronic device, and a program to at least solve the problem in the related art that the prefetching method for metadata in a distributed storage system relies on static path associations of metadata and has poor cross-node adaptability, resulting in low prefetching accuracy.

[0006] This application provides a metadata prefetching method, comprising: obtaining a first information group corresponding to metadata operation events in a target storage area of ​​each node, wherein each node is a metadata server node; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining the prefetching priority common to the two metadata based on the updated three-dimensional correlation degree; generating a hot association list of metadata in each node according to the prefetching priority; and sending the hot association lists corresponding to multiple nodes to a client respectively, so as to instruct the client to prefetch the target metadata in the multiple nodes according to the multiple hot association lists.

[0007] This application also provides a metadata prefetching device, comprising: an acquisition module, configured to acquire a first information group corresponding to metadata operation events in a target storage area of ​​each node, wherein each node is a metadata server node; an update module, configured to update the three-dimensional correlation degree between any two metadata in each node according to the first information group; a determination module, configured to determine the prefetching priority common to the two metadata based on the updated three-dimensional correlation degree; and an instruction module, configured to generate a hot association list of metadata in each node according to the prefetching priority, and send the hot association lists corresponding to multiple nodes to a client respectively, so as to instruct the client to prefetch the target metadata in the multiple nodes according to the multiple hot association lists.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the prefetching method for metadata as described above when executing the computer program.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described metadata prefetching methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described metadata prefetching methods.

[0011] This application obtains a first information group corresponding to metadata operation events in the target storage area of ​​each node, wherein each node is a metadata server node; updates the three-dimensional correlation between any two metadata items in each node based on the first information group; determines the prefetch priority common to the two metadata items based on the updated three-dimensional correlation; generates a hot association list of metadata in each node based on the prefetch priority; and sends the hot association lists corresponding to multiple nodes to the client respectively, instructing the client to prefetch the target metadata in the multiple nodes according to the multiple hot association lists. This application solves the problem in related technologies where the prefetching method for metadata in a distributed storage system relies on static path associations of metadata and has poor cross-node adaptability, resulting in low prefetch accuracy, thereby improving the prefetch accuracy of metadata. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a hardware structure block diagram of a computer terminal for a metadata prefetching method according to an embodiment of this application.

[0014] Figure 2 This is a flowchart of a metadata prefetching method according to an embodiment of this application;

[0015] Figure 3 This is a distributed storage system architecture diagram according to an optional embodiment of this application;

[0016] Figure 4 This is a schematic diagram of the interaction flow of key modules according to an optional embodiment of this application;

[0017] Figure 5 This is a framework diagram of a metadata prefetching device according to an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The specific application environment architecture or specific hardware architecture on which the execution of the metadata prefetching method depends is described here.

[0022] The methods and embodiments provided in this application can be executed on server devices, mobile terminals, computer terminals, or similar computing devices. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a metadata prefetching method according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor unit (MPU), or programmable logic device (PLD)) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0023] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for determining the interaction state in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0025] Figure 2 This is a flowchart of a metadata prefetching method according to an embodiment of this application, which can be applied to... Figure 1 In computer terminals, it is used in high-frequency metadata access scenarios, such as Figure 2 As shown, the process includes the following steps:

[0026] Step S202: Obtain the first information group corresponding to the metadata operation event in the target storage area of ​​each node, wherein each node is a metadata server node;

[0027] Step S204: Update the three-dimensional correlation degree between any two metadata in each node according to the first information group;

[0028] Step S206: Determine the prefetch priority corresponding to any two metadata items by using the updated three-dimensional correlation degree;

[0029] Step S208: Generate a hot association list of metadata in each node according to the prefetch priority, and send the hot association lists corresponding to multiple nodes to the client respectively, so as to instruct the client to prefetch the target metadata in the multiple nodes according to the multiple hot association lists.

[0030] Through the above steps, a first information group corresponding to metadata operation events in the target storage area of ​​each node is obtained, wherein each node is a metadata server node; the three-dimensional correlation degree between any two metadata in each node is updated according to the first information group; the prefetch priority corresponding to any two metadata is determined by the updated three-dimensional correlation degree; a hot association list of metadata in each node is generated according to the prefetch priority, and the hot association lists corresponding to multiple nodes are sent to the client respectively to instruct the client to prefetch the target metadata in the multiple nodes according to the multiple hot association lists. This application solves the problem in related technologies where the prefetching method for metadata in a distributed storage system relies on static path associations of metadata and has poor cross-node adaptability, resulting in low prefetch accuracy, thereby improving the prefetch accuracy of metadata.

[0031] In an exemplary embodiment, before obtaining the first information group corresponding to the metadata operation event in the target storage area of ​​each node, the method further includes: intercepting the metadata operation event through a kernel-mode hook function and extracting the first information group corresponding to the metadata operation event, wherein the first information group includes: a first identifier of the metadata operated by the metadata operation event, a second identifier of the access process executing the metadata operation event, and a timestamp of the metadata operation event; and storing the first information group in the target storage area of ​​each node.

[0032] In other words, metadata operation events (including system calls such as open, stat, readdir, and mkdir) are intercepted in real time through kernel hook functions. The first information group of metadata is extracted. This first information group is specifically represented by a triplet of information: <metadata unique identifier (ID), accessing process process identifier (PID), timestamp>. It is stored in the target storage area, which can be a local circular buffer (the capacity of this buffer is configurable, and it retains 24 hours of data by default).

[0033] By implementing hook functions at the kernel level to capture and analyze metadata operation events in real time, information on metadata access is effectively collected. This enables the system to accurately track metadata access patterns, providing a solid data foundation for subsequent high-performance prefetching strategies. This helps to significantly improve metadata access efficiency and user experience, especially under high-concurrency and data-intensive workloads. It can better understand and predict metadata access dynamics, thereby optimizing prefetching decisions, reducing unnecessary network transmission, speeding up access, and reducing latency.

[0034] In an exemplary embodiment, updating the three-dimensional correlation between any two metadata in each node according to the first information group includes: updating the spatial correlation in the three-dimensional correlation according to the first information group and the co-occurrence count of the two metadata; updating the temporal decay in the three-dimensional correlation according to the first information group and the decay coefficient; and updating the process correlation in the three-dimensional correlation according to the first information group.

[0035] Furthermore, updating the spatial correlation in the three-dimensional correlation degree based on the co-occurrence count of the first information group and any two metadata includes: determining the hierarchical distance between any two metadata through the directory tree of the metadata corresponding to each node; and determining the access counts corresponding to each of the two metadata through the first information group; and updating the spatial correlation in the three-dimensional correlation degree based on the co-occurrence count, the hierarchical distance, and the access count.

[0036] Specifically, updating the spatial correlation in the three-dimensional correlation degree using the co-occurrence count, the hierarchical distance, and the access count includes: determining the larger access count among the access counts corresponding to any two metadata items, and determining the ratio of the co-occurrence count to the larger access count; updating the spatial correlation in the three-dimensional correlation degree using the ratio and the hierarchical distance.

[0037] Optionally, the formula for calculating spatial correlation is:

[0038] ;

[0039] x and y represent any two metadata items. x is the metadata corresponding to a metadata operation event that is intercepted in real time by a kernel-mode hook function, and y is the candidate prefetched metadata relative to x. d is the hierarchical distance between metadata x and y in the directory tree. Access count x and access count y represent the access counts corresponding to any two metadata items respectively. Co-occurrence count is the number of times x and y are accessed consecutively within a sliding window (default 1 hour).

[0040] Calculating and periodically updating spatial correlation can accurately diagnose hierarchical dependencies between metadata, significantly improve the intelligence and efficiency of the prefetching mechanism, ensure that cache resources are prioritized for storing metadata with high access probability, thereby significantly reducing access latency and optimizing the overall performance of the distributed storage system.

[0041] Furthermore, updating the time decay in the three-dimensional correlation degree based on the first information group and the decay coefficient includes: determining the access time of the i-th access of the target metadata in any two metadata within a first time interval, wherein each node is allowed to obtain the first information group from the target storage area at intervals of the first time interval; updating the time decay in the three-dimensional correlation degree through the access frequency corresponding to the i-th access, the second time interval, and the decay coefficient, wherein the second time interval is the time interval between the current time and the access time.

[0042] Here, the i-th access can be the i-th access to metadata x, where x is the metadata corresponding to the metadata operation event intercepted in real time by the kernel-mode hook function. The formula for determining the time decay is:

[0043] .

[0044] in, Let λ be the frequency of the i-th access, and λ be the attenuation coefficient. The second event interval represents the time interval between the current access time and the i-th access time.

[0045] By calculating and periodically updating the time decay rate, the influence of historical access records on prefetching decisions can be dynamically adjusted, ensuring that metadata accessed more frequently in the recent period receives a higher prefetching priority. This effectively improves the accuracy of prefetching and the immediacy of system response, reduces latency, and optimizes the user experience.

[0046] Optionally, after updating the time decay in the three-dimensional correlation degree according to the first information group and the decay coefficient, the method further includes: determining the contribution ratio of the time decay in the prefetch priority; and reducing the decay coefficient if the contribution ratio is higher than a preset ratio within m consecutive periods, where m is a positive integer.

[0047] Furthermore, updating the process correlation in the three-dimensional correlation based on the first information group includes: determining the process sets that access any two metadata items respectively through the first information group; and updating the process correlation in the three-dimensional correlation based on the process sets corresponding to the two metadata items respectively.

[0048] Specifically, updating the process correlation degree in the three-dimensional correlation degree through the process sets corresponding to any two metadata includes: determining the intersection and union between the process sets corresponding to any two metadata; and updating the process correlation degree in the three-dimensional correlation degree through the intersection and the union.

[0049] Optionally, the formula for calculating process correlation is:

[0050] ;

[0051] in, For the set of processes that access x, The set of processes that access y.

[0052] Determining process correlation helps identify and quantify the binding relationship between a specific process and its frequently accessed metadata, thereby prioritizing the metadata related to these processes in prefetching decisions, significantly improving the targeting and efficiency of prefetching, reducing invalid prefetching across processes, and optimizing the resource utilization and access performance of distributed storage systems.

[0053] In an exemplary embodiment, determining the prefetch priority corresponding to any two metadata items through the updated three-dimensional correlation degree includes: determining the dynamic weights corresponding to the spatial correlation degree, the time decay degree, and the process correlation degree respectively; and performing a weighted calculation on the updated spatial correlation degree, the updated time decay degree, and the updated process correlation degree through the dynamic weights to obtain the prefetch priority corresponding to any two metadata items.

[0054] The formula for calculating the prefetch priority is as follows:

[0055] ;

[0056] Where α, β, and γ are dynamic weights (summing to 1).

[0057] In an exemplary embodiment, generating a hot association list of metadata in each node according to the prefetch priority includes: filtering multiple metadata pairs in descending order of prefetch priority, wherein the prefetch priority of each metadata pair is higher than the prefetch priority of other metadata pairs in each node besides the multiple metadata pairs; generating the hot association list from the multiple metadata pairs, wherein the hot association list includes: a third identifier of the metadata in each metadata pair, the prefetch priority, and a validity period, wherein the validity period is determined by the prefetch priority.

[0058] In other words, after each update of the prefetch priority of metadata pairs, the top 20 metadata pairs with the highest prefetch priority for each node are selected in descending order of prefetch priority to form a hot association list. The hot association list contains the fields <source metadata ID, target metadata ID, relevance, validity period>, where the validity period is dynamically set based on the relevance (5 minutes if the relevance is ≥0.8, otherwise 2 minutes). The source metadata corresponds to metadata x mentioned above, and the target metadata corresponds to metadata y mentioned above.

[0059] In an exemplary embodiment, instructing the client to prefetch target metadata from the plurality of nodes based on the plurality of hot-association lists includes: instructing the client to filter the hot-association lists pushed by the plurality of nodes respectively through a metadata index to obtain a list to be prefetched, wherein the metadata index is an index of metadata cached locally by the client; instructing the client to send a batch prefetch request to the plurality of nodes according to the correspondence between the fourth identifier of the metadata included in the list to be prefetched and the plurality of nodes, so as to prefetch the target metadata from the plurality of nodes, wherein the batch prefetch request is encoded using a predefined binary protocol.

[0060] Furthermore, instructing the client to send a batch prefetch request to the plurality of nodes according to the correspondence between the fourth identifier of the metadata included in the prefetch list and the plurality of nodes includes: instructing the client to group the metadata included in the prefetch list based on the correspondence to obtain a plurality of groups, wherein each group uniquely corresponds to a target identifier in the node identifier of the plurality of nodes; instructing the client to send a batch prefetch request to the node corresponding to the target identifier, wherein the batch prefetch request is a batch prefetch request for the metadata in each group.

[0061] In other words, the client aggregates the hot-related lists received from all MDS nodes and compares them with the locally cached metadata index (record <metadata ID, cache time, access count>). It filters cached and non-expired metadata, retaining uncached or soon-to-expire entries. The filtered prefetch list is then grouped by MDS node (resulting in multiple groups as in the above embodiment). Each group generates a batch input / output (I / O) request (i.e., the batch prefetch request in the above embodiment) and sends it to the corresponding MDS through the client interface.

[0062] By centrally processing multiple hot-association lists and sending batch prefetch requests on the client side, the number of interactions between the client and storage nodes is significantly reduced. By prefetching multiple expected access metadata at once, network latency and bandwidth consumption are greatly reduced, improving the overall throughput and access speed of the distributed storage system and optimizing the user's data access experience.

[0063] In an optional embodiment, after determining the prefetch priority corresponding to any two metadata items through the updated three-dimensional correlation degree, the method further includes, in the case of subtree migration requirements: receiving a first acquisition request sent by a subtree migration coordinator when each node triggers subtree migration, wherein the first acquisition request is used to request acquisition of an association summary table, wherein the association summary table includes: association entries of all metadata of the subtree; sending the association summary table to the subtree migration coordinator to instruct the subtree migration coordinator to send the association summary table to the target node of the subtree migration.

[0064] Optionally, if each node is the target node of the subtree migration, the association summary table sent by the source node of the subtree migration through the subtree migration coordinator is parsed; a preload instruction for the first n metadata pairs in the parsed association summary table is generated, wherein the first n metadata pairs are the first n metadata pairs in the parsed association summary table after prefetching priority is sorted from high to low, and n is a positive integer; the first n metadata pairs are preloaded into memory through the preload instruction.

[0065] In other words, subtree migration in the distributed storage system is executed through a tree migration coordinator (deployed in the MDS cluster manager and working in conjunction with the MON component). Specifically, the subtree migration coordinator listens for subtree migration events in the MDS cluster. When it detects that the source MDS has triggered a subtree migration, it sends a request to retrieve the association summary table (equivalent to the first retrieval request) to the source MDS. After the target MDS receives the association summary table, it parses and extracts the top 5 metadata entries with the highest correlation, generates a preload instruction (including loading order and cache duration), calls the MDS local cache interface (such as mds_cache_add(), which can be set to cache_ttl=300s (5 minutes)) to load these metadata entries into memory, and sets the cache priority to "highest". It should be noted that the associated metadata entries are any two metadata entries that are related, and the correlation is equivalent to the prefetch priority in the above embodiment. The Time To Live (TTL) is also mentioned.

[0066] In an exemplary embodiment, after preloading the first n metadata pairs into memory via the preloading instruction, the method further includes: generating a metadata location mapping table corresponding to the first n metadata pairs; and broadcasting the metadata location mapping table to the client via a preset broadcast mechanism.

[0067] After the target MDS completes preloading, once loading is complete, the subtree migration coordinator is notified through the mds_notify() interface to trigger location synchronization, which generates a metadata location mapping table (<metadata ID, target MDS Internet Protocol (IP) address, port>), and synchronizes it to all client prefetch agents through the distributed storage Monitor broadcast mechanism.

[0068] To better understand the process of the above metadata prefetching method, the implementation flow of the above metadata prefetching method will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0069] The following are some of the prefetching schemes for metadata in distributed storage systems and their drawbacks in related technologies:

[0070] Option 1: Implement dynamic subtree partitioning through a metadata server, and migrate frequently accessed metadata subtrees to idle MDS nodes to balance the load; the client enables the directory prefetching mechanism by default, and loads the directory entries (dentry) and inode information of the entire directory in one I / O operation, reducing the number of interactions with the MDS.

[0071] The disadvantages of Option 1 are: the prefetch strategy only relies on static directory hierarchy association and does not consider the dynamic context of file access (such as process association and time series characteristics); the dynamic subtree partitioning is only based on load statistics and does not combine metadata association, which makes it easy for associated metadata to be scattered across nodes after subtree migration, which will increase access latency.

[0072] Option 2: Use a lightweight pattern matching algorithm to discover explicit relationships (such as hyperlinks and references) between files, store the relationship information in the extended attributes of the file metadata, and implement metadata prefetching based on the relationship without modifying the application programming interface (API).

[0073] The disadvantages of Scheme 2 are: it only relies on explicit static associations between files, without considering time decay characteristics (the association weight of recent accesses should be higher than that of historical accesses) and process context (such as file associations specific to a particular process), and the prefetch accuracy is limited by the completeness of the explicit associations; it does not support the collaboration of association data between multiple MDS nodes, and cross-node metadata associations cannot be perceived.

[0074] Option 3: Based on the continuous access characteristics of file paths, when accessing certain metadata, the metadata of its adjacent paths (such as the inodes of other files in the same directory) is prefetched, which is suitable for large file continuous access scenarios.

[0075] The disadvantages of Scheme 3 are: it only applies to access patterns with continuous paths, and has no prefetching effect on non-contiguous or cross-directory metadata access (such as the associated access of log files and configuration files); it does not consider the dynamic weight changes of metadata access, and historical access and recent access are given equal weight, resulting in a high proportion of invalid prefetching.

[0076] In summary, existing technologies suffer from low prefetching accuracy due to reliance on static path associations of metadata in distributed storage systems and poor cross-node adaptability. To address this issue, this application discloses an optional embodiment of a metadata prefetching optimization method and system based on distributed storage systems (equivalent to the metadata prefetching method in the above embodiments), aiming to solve problems such as low metadata prefetching accuracy and insufficient cross-node collaboration in distributed storage. The core innovations are: constructing a three-dimensional association model of "spatiotemporal features - process context - subtree topology," proposing a multi-dimensional association prediction algorithm, and accurately calculating metadata prefetching priority by dynamically weighting spatial association degree, temporal decay degree, and process association degree; designing a distributed association collaboration mechanism, achieving lightweight synchronization of multi-MDS node association data based on an association summary table and gossip protocol, breaking through the cross-node prefetching bottleneck; introducing reinforcement learning to dynamically optimize weight parameters to adapt to different load scenarios; and adopting a plug-in design, ensuring compatibility with existing distributed storage system architectures without modifying core logic. The system achieves full-process optimization through an MDS association engine, client prefetching agent, and subtree migration coordinator, significantly improving metadata prefetching efficiency.

[0077] The technical solution adopted in the optional embodiments of this application is applied to a distributed storage system cluster. Figure 3 This is a distributed storage system architecture diagram according to an optional embodiment of this application. (e.g.) Figure 3 As shown, the distributed storage system adopts a decentralized, distributed architecture. Its core components include clients, a Monitor node (MON), an MDS (Multi-Level Data Storage), and an Object Storage Device (OSD). These modules collaborate via high-speed networks (such as Ethernet and InfiniBand) to achieve data storage and management. The layered architecture of the distributed storage system is explained below:

[0078] 1) Client Layer. Positioning of the Client Layer: The client is the entry point for users to interact with the distributed storage system, providing three interfaces: block, object, and file.

[0079] The core components of the client layer include: the librados library, which encapsulates the underlying API and is responsible for interacting with the cluster; and the Controlled Replication Under Scalable Hashing (CRUSH) client module, which caches the CRUSH Map locally and calculates data storage locations. Inputs include object names and storage pool rules (such as the number of replicas and fault domain level). The processing steps for the input are: hashing the object name to generate a random number; traversing the CRUSH bucket level (e.g., host → rack → data center) according to the storage pool rules and selecting OSDs based on weights; and ensuring balanced data distribution to meet redundancy strategies (e.g., deploying replicas across racks).

[0080] 2) Control Plane (Monitor / MDS). The Monitor maintains cluster metadata (OSD Map, CRUSH Map, authentication information) and ensures consistency across multiple nodes using the Paxos algorithm. The MDS (in file scenarios) manages file system metadata (directory tree, permissions) and caches frequently accessed metadata to accelerate access. The control plane also interacts with clients: it pulls the latest cluster map when a client first connects and updates it synchronously via heartbeats thereafter.

[0081] 3) Data Plane (OSD Cluster). An OSD cluster consists of OSD nodes. The hardware components of an OSD node include: a Central Processing Unit (CPU), memory, disks (Hard Disk Drive (HDD) / Solid State Drive (SSD)), and a network interface (Gigabit / 10 Gigabit Ethernet card). OSD nodes are responsible for data storage, replication, erasure coding, and consistency maintenance, and are aware of the cluster topology through an OSD Map.

[0082] Based on the above-described distributed storage system architecture, the optional embodiments of this application add three core modules to the distributed storage system architecture through a plug-in design. Figure 4 This is a schematic diagram of the key module interaction flow according to an optional embodiment of this application, which is described below in conjunction with... Figure 4 The three newly added core modules will be explained.

[0083] New Module 1: MDS Association Engine (deployed on each MDS node). Includes log collection unit, association calculation unit, and list generation unit.

[0084] The log collection unit intercepts metadata operation events (including system calls such as open, stat, readdir, and mkdir) in real time through kernel hook functions, extracts triple information <metadata unique identifier ID, accessing process PID, timestamp>, and stores it in a local circular buffer (capacity is configurable, and data is retained for 24 hours by default).

[0085] The association calculation unit loads the offline-trained basic association model, updates the three-dimensional association degree (spatial association degree S, temporal decay degree T, and process association degree C) every 10 seconds based on real-time data from a circular buffer, and calculates the prefetch priority P(x,y) using a multi-dimensional association expectation algorithm. Optionally, the prefetch priority can be calculated by executing a multi-objective differential evolution algorithm (MDAEA) based on an adaptive ensemble approach.

[0086] The list generation unit filters the top 20 metadata pairs by priority every 30 seconds to generate a "hot-related list". The list contains the fields <source metadata ID, target metadata ID, relevance, validity period>, where the validity period is dynamically set according to the relevance (5 minutes when the relevance is ≥0.8, otherwise 2 minutes).

[0087] It's important to note that "Source Metadata ID" refers to metadata that is currently accessed or has a high access frequency (serving as the "starting point" for triggering prefetching); "Target Metadata ID" refers to metadata that is highly related to the source metadata and is expected to be accessed in subsequent accesses (serving as the "object" for prefetching). For example, when " / logs / 2024-08-01" (the source) is frequently accessed, based on three-dimensional correlation calculations, if " / logs / 2024-08-02" (the target) is closely related to it, then <Source ID, Target ID> will be included in the hot association list to guide the client to prefetch the target metadata when accessing the source metadata, reducing latency in subsequent accesses. This naming reflects the relationship between the "prefetch trigger (source metadata)" and the "prefetch object (target metadata)". The "source" and "target" metadata in the hot association list are only used for prefetching logical association, that is, predicting the access demand for target metadata through the access behavior of source metadata, and then loading the target metadata into the client cache in advance. This is a "cache preheating" operation and does not involve the migration of the storage location of the metadata itself.

[0088] New Module 2: Client Prefetch Agent (deployed in the distributed storage client user space). Includes list filtering unit, batch prefetching unit, and effect feedback unit.

[0089] The list filtering unit receives the hot-related list pushed by MDS, compares it with the locally cached metadata index (record <metadata ID, cache time, access count>), filters out cached and non-expired metadata, and retains entries that are not cached or are about to expire.

[0090] It should be noted that the metadata is distributed across multiple MDS nodes (based on dynamic subtree partitioning). Each MDS node's list generation unit generates a hot-association list containing <source metadata ID, target metadata ID> for the metadata it manages (where the target metadata may belong to this node or other nodes), and pushes it to the client. The client aggregates the hot-association lists received from all MDS nodes, serving as the basis for subsequent filtering and grouping.

[0091] The batch prefetching unit groups the filtered list of items to be prefetched by MDS nodes, generates a batch I / O request for each group (using a custom binary protocol encoding, including a list of metadata IDs and priorities), and sends it to the corresponding MDS through the client interface.

[0092] The target metadata may be distributed across different MDS nodes (for example, the source metadata belongs to MDS-01, while the associated target metadata may belong partly to MDS-01 and partly to MDS-02).

[0093] Grouping by MDS node allows multiple prefetch requests for the same node to be merged into a single batch I / O request. All prefetch operations for that node can be completed in a single network interaction, avoiding multiple scattered requests to the same node (reducing overhead such as handshakes and verification). Centralized filtering and grouping by the client allows for global coordination of prefetching needs across all MDSs, optimizing batch request merging strategies (such as dynamically adjusting request order based on node load and network conditions).

[0094] The effect feedback unit records whether the metadata prefetched each time is actually accessed (hit), generates a feedback report every 5 minutes (including hit rate, invalid prefetch amount, and average access interval), and sends it to the corresponding MDS associated engine through an asynchronous channel.

[0095] New Module 3: Subtree Migration Coordinator (deployed in MDS cluster manager, working in conjunction with the MON component). Includes data migration unit, preload triggering unit, and location synchronization unit.

[0096] The data migration unit listens for subtree migration events in the MDS cluster. When it detects that the source MDS has triggered a subtree migration, it sends a request to the source MDS to obtain the associated summary table. It packages the summary table (containing all associated entries of the metadata of the subtree) into JavaScript Object Notation (JSON) format and sends it to the target MDS through a reliable transmission protocol (which may be Transmission Control Protocol, or TCP).

[0097] The preloading trigger unit, after receiving the association summary table from the target MDS, parses and extracts the top 5 metadata entries with the highest association degree, generates a preloading instruction (including loading order and cache duration), calls the MDS local cache interface (such as mds_cache_add(), which can set cache_ttl=300s (5 minutes)) to load these metadata entries into memory, and sets the cache priority to "highest". It should be noted that the associated metadata entries are any two metadata entries that are related, and the association degree is equivalent to the prefetch priority in the above embodiment.

[0098] The location synchronization unit: After the target MDS completes preloading, it notifies the subtree migration coordinator through the mds_notify() interface to trigger location synchronization, that is, to generate a metadata location mapping table (<metadata ID, target MDSIP, port>), and synchronize it to all client prefetch agents through the distributed storage Monitor broadcast mechanism.

[0099] It should be noted that the batch prefetching units of the client prefetching agent need to update the target node information based on these broadcast new locations to ensure that subsequent prefetching requests can be correctly sent to the migrated MDS nodes, avoiding prefetching failures due to location invalidation. After the target MDS receives the association summary table and completes preloading, its MDS association engine will incorporate these preloaded metadata into the local association calculation scope, update the three-dimensional association degree based on the new metadata distribution (such as the need for recalibration after the topological changes of the spatial association factor tree after migration), and generate a new hot association list to ensure the accuracy of the association calculation after migration.

[0100] In summary, the subtree migration coordinator is a functional extension and collaborative guarantee of the first two new modules in the "subtree migration scenario": the first two provide the associated data foundation to ensure the accuracy of preloading during migration; the latter, through location synchronization and preloading, ensures that the first two can still run efficiently when the distributed environment changes. Together, the three constitute a closed loop of full-scenario optimization for metadata prefetching.

[0101] Based on the above module functions, optional embodiments of this application propose module interaction details and protocol specifications:

[0102] The communication protocol between MDS and the client uses a custom binary protocol format as follows:

[0103] Header (16 bytes): Magic number "0xCEPHASSOC" (8 bytes) + version number (2 bytes) + number of entries (2 bytes) + checksum (4 bytes, using CRC32); where the Cyclic Redundancy Check 32-bit (CRC32 for short) is included.

[0104] Body: Each entry contains metadata ID (64-bit integer encoding) + relevance (single-precision floating-point number, 0-1) + validity period (32-bit integer, in seconds), total length = number of entries × 16 bytes;

[0105] Compression: When the number of entries is ≥5, Z-standard compression library (Zstandard, abbreviated as zstd) is enabled. The compression level increases with the number of entries (level 3 for 5-10 entries, level 5 for 11-20 entries).

[0106] The subtree migration coordinator interacts with MON: It listens for subtree migration events through the mds_map function of MON, calls mon_command("tell mds.{src_mds} get_assoc_summary {subtree_path}") to obtain the associated summary table, and after receiving it, calls mon_command("tell mds.{dest_mds} load_assoc_summary {data}") to trigger preloading. The interface response timeout is set to 5 seconds.

[0107] Optionally, the optional embodiments of this application overcome the limitations of static association through multi-dimensional association modeling, innovatively integrating a three-dimensional model of "spatiotemporal features - process context - subtree topology" through a multi-dimensional association expectation algorithm. Multi-dimensional association degree includes spatial association degree, temporal decay degree, and process association degree. Among them, spatial association degree S quantifies directory hierarchy dependency (such as the dependency relationship between " / data / logs" and " / data / logs / 2024"), solving the locality blind spot of traditional path prefetching. Temporal decay degree T dynamically adjusts the historical access weight through an exponential function (e.g., the access weight within 1 hour is the same as that 24 hours ago). (Multiple times), adapting to dynamic changes in access patterns. Process association degree C identifies dedicated files through PID clustering (e.g., processes 1234 always access " / app / config"), avoiding invalid cross-process associations.

[0108] Specifically, a three-dimensional association model of "spatiotemporal features - process context - subtree topology" is constructed, which involves building models for calculating spatial association degree, temporal decay degree, and process association degree respectively. Finally, the metadata prefetching priority is calculated by comprehensively calculating the dynamic weights, using the following formula:

[0109] ;

[0110] Here, x and y represent any two metadata items in the metadata set managed by the metadata server, such as " / logs / 2024-08-01" (x) and " / logs / 2024-08-02" (y). The prefetch priority P(x,y) is the prefetch priority of metadata y based on metadata x, that is, the degree of necessity for the system to prefetch y when x is accessed. For example, if P(x,y) is high when x is accessed frequently, it indicates that y is closely related to x, and y should be prefetched first to improve access efficiency. In other words, the core of the association calculation unit is to dynamically calculate the three-dimensional association degree between pairs of metadata items in the metadata set based on the offline initial model and real-time data, and then fuse them into a prefetch priority using a fixed formula to guide accurate prefetching.

[0111] Where S(x,y) represents spatial relevance, used to quantify the dependencies of metadata in the subtree topology. Based on the hierarchical distance d of the directory tree (e.g., d=1 for the same parent directory, d=2 for every level apart), the formula for calculating spatial relevance is:

[0112] ;

[0113] Where d represents the hierarchical distance (integer) between metadata x and y in the directory tree. For example, if x is " / logs / 2024-08" and y is " / logs / 2024-08 / 01", then the hierarchical distance d = 1 (i.e., y is a direct subdirectory of x); if y is " / logs / 2024-09", then d = 2 (i.e., x and y belong to the same parent directory " / logs" but are not direct subdirectories). The co-occurrence count is the number of times x and y are accessed consecutively within a sliding window (default 1 hour) (integer). For example, if x is accessed 15 times along with y within 1 hour, then the co-occurrence count = 15.

[0114] Where T(x,y) represents the time decay rate, and the weight of historical visits is dynamically adjusted based on an exponential function, using access frequency weighting within a sliding window. The weight of the i-th visit within the window is... Access weight outside the window decays to zero. The formula for calculating the time decay is:

[0115] .

[0116] in, Let λ be the frequency of the i-th access, and λ be the attenuation coefficient. This represents the time interval between the current access time and the i-th access time.

[0117] Where C(x,y) represents the process correlation degree, which is calculated using the Jaccard coefficient. The calculation formula is as follows:

[0118] ;

[0119] in, For the set of processes that access x, Let y be the set of processes accessing y. α, β, and γ are dynamic weights (summing to 1). It should be noted that the basic association model mentioned above refers to the initial framework and initial parameters of the three formulas used to calculate spatial association, temporal decay, and process association. The initial parameters include: the initial co-occurrence count of spatial association S, the initial decay coefficient λ of temporal decay T (e.g., λ=0.01), the initial process set association rule for process association C, and the initial values ​​of the dynamic weights α, β, and γ (e.g., 0.3, 0.4, 0.3). The purpose of these initial parameters is to provide an initial parameter baseline for online calculation, and they will be dynamically updated based on real-time data. In other words, the formulas represent the fixed calculation logic, while the model represents the initial parameters and framework of the formulas.

[0120] Optionally, the dynamic weights can be optimized in real time by reinforcement learning. The reward function used by reinforcement learning is: prefetch hit rate - invalid prefetch cost, specifically "invalid prefetch cost = number of invalid prefetch entries × average size of a single metadata entry".

[0121] Furthermore, a Markov decision process is constructed using prefetch hit rate and invalid prefetch overhead as state characteristics. When the system load switches from a high-performance computing (HPC) scenario to a big data scenario, the process correlation weight β is automatically increased (from 0.2 to 0.4). Additionally, a system stability parameter is introduced to dynamically adjust the learning rate. When the hit rate fluctuation is ≤5% for 10 consecutive minutes, the learning rate is reduced by 50% to avoid parameter oscillations in a stable state.

[0122] Dynamic weights α, β, γ: Initial values ​​(0.3, 0.4, 0.3) are adapted to mixed workloads; for small file-intensive scenarios (such as log collection), spatial correlation S contributes more, so α is increased to 0.4 and β is decreased to 0.3; for long-cycle access scenarios (such as scientific computing), time decay T is more critical, so β ​​is increased to 0.5 and α is decreased to 0.2; the weight adjustment step size is 0.1, and each adjustment must satisfy α+β+γ=1.

[0123] Optionally, for the decay coefficient λ: the initial value is determined through offline training. Metadata access logs of 10 typical loads (including HPC, big data, and AI training) are collected, and the time decay degree T under different λ values ​​is calculated to match the actual access. The λ=0.01 with the highest average matching degree is selected as the benchmark value. If the contribution ratio of the time decay degree of a certain type of load (such as AI training) (the proportion of T in P(x,y)) is ≥60% during runtime, λ is automatically reduced to 0.005 (to enhance the weight of recent access).

[0124] Optionally, an initial value λ=0.01 is determined for offline training. The core logic is to select the optimal value by comparing the matching degree between the time decay rate and the actual access behavior under different λ values.

[0125] Specifically, metadata access logs are collected for two typical workloads (simplified example, the original text lists 10 types): Workload A (HPC scenario): Historical access records (timestamp unit: minutes) of metadata x (e.g., " / simulation / step1") under a certain directory: (current), , , (1 hour ago). Load B (AI training scenario): No historical access records for data y (e.g., " / model / epoch5"): , , , .

[0126] First, the time decay T is calculated according to the formula for different values ​​of λ to determine the initial value of λ:

[0127] When λ = 0.005:

[0128] Load A:

[0129] ;

[0130] Load B:

[0131] .

[0132] When λ=0.01:

[0133] Load A:

[0134] ;

[0135] Load B:

[0136] .

[0137] When λ=0.02:

[0138] Load A:

[0139] ;

[0140] Load B:

[0141] .

[0142] Calculate the matching degree:

[0143] The "matching degree" is defined as: Matching degree = Number of times the metadata pairs with high T values ​​in the Top 20 were actually accessed / 20. Here, the Top 20 metadata pairs with high T values ​​refer to the Top 20 being sorted based on the time decay factor T.

[0144] In actual access for load A, recent accesses (t less than or equal to 30 minutes) account for 80%. When λ=0.01, the T value is more consistent with the weight of recent accesses, and the matching degree is 75%. In actual access for load B, recent accesses (t less than or equal to 30 minutes and 15 minutes) account for 90%. When λ=0.01, the matching degree is 80%. The average matching degree is (75%+80%) / 2=77.5%, which is higher than λ=0.005 (70%) and λ=0.02 (65%). Therefore, λ=0.01 is chosen as the initial value.

[0145] Secondly, the process of adjusting λ during runtime will be explained, taking the process of lowering it to 0.005 as an example. The core logic is that when the contribution ratio of T in the prefetch priority P continues to be... A score of 60% indicates a need to increase access weight and lower λ.

[0146] Calculate the contribution percentage of T in the prefetch priority. .

[0147] Taking AI training load as an example, at a certain moment, the calculation is: S=0.3, T=0.6, C=0.2 (dynamic weights α=0.2, β=0.5, γ=0.3), then the prefetch priority is:

[0148] ;

[0149] The calculated contribution ratio of T is (The reason why T is higher than P here is because the weight coefficient is increased. In actual scenarios, T is the original value and needs to be calculated in combination with the weight. The simplified example highlights the trend.)

[0150] If for three consecutive cycles (once every 10 minutes), the contribution percentage of T is consistently... If 60% (e.g., 142.8%, 130%, 120%), then the load is considered to have a high dependence on recent visits, and λ is reduced from 0.01 to 0.005. After the reduction, the weight of recent visits increases (e.g., visits within 10 minutes of t). Compared to 0.905 when λ=0.01), the T value is closer to the actual access pattern, and the budget hit rate is improved.

[0151] In summary, the initial value of λ is determined by comparing offline data, and is dynamically adjusted at runtime according to load characteristics, ultimately achieving a precise match between the time decay degree T and the actual access behavior.

[0152] Furthermore, in an optional embodiment of this application, to address the cross-MDS prefetch failure problem, an innovative association summary table synchronization mechanism is designed for the distributed architecture of a distributed storage system with multiple MDS nodes. Lightweight synchronization is achieved using the gossip protocol (transmitting only entries with an association degree ≥ 0.6), and the synchronization frequency (2 minutes) matches the existing cluster state synchronization cycle, avoiding additional network overhead. During synchronization, if there are associated entries with the same metadata ID, the entry with the latest timestamp is used.

[0153] When a client accesses cross-MDS metadata, the primary MDS initiates a relational query to the secondary MDS and merges the results to generate a global hot relational list, solving the problem that traditional prefetching is limited to a single MDS node. The relational summary table structure and storage field definitions are as follows:

[0154] 1) Metadata ID: 64-bit integer (based on distributed storage inode number extension, the first 32 bits are the MDS ID, and the last 32 bits are the local inode number); the format of the associated metadata ID is the same as the metadata ID format; 2) Association degree: single-precision floating-point number (retaining 3 decimal places); 3) Last update time: 64-bit timestamp (in milliseconds); 4) Storage and expansion: LevelDB storage is used (path / var / lib / icfs / mds / {fs_name} / leveldb / ), with a single table capacity limit of 1 million records; when the number of records reaches the limit, records that have not been updated for 7 days are discarded according to the "last update time", and a new table is automatically created (naming rule is assoc_summary_{timestamp}.ldb), and the table entries are traversed in reverse order of timestamp during querying.

[0155] It should be noted that the global hot association list is a cross-node prefetch list generated by the client after aggregating all MDS node hot association lists (such as <1001,1002> of MDS-01 and <2001,2002> of MDS-02), deduplicating and filtering (removing cached entries). It is the client-side aggregation result of the "single node hot association list" mentioned above.

[0156] For the association summary table maintenance mechanism, the local storage adopts a Least Recently Used (LRU) cache structure with a capacity of 5% of the MDS node memory, prioritizing the retention of entries with an association degree ≥ 0.5; the synchronization strategy is to synchronize every 2 minutes via the gossip protocol, with the synchronization packet size limited to 1MB (containing approximately 1000 entries), and truncation is performed in descending order of association degree when the size exceeds this limit; the eviction rule is to mark entries with an association degree ≤ 0.3 for 3 consecutive calculations as "cold data" and delete them during the next cache cleanup.

[0157] It should be noted that all the newly added modules mentioned above are integrated through the dynamic plug-in interface of the distributed storage system (such as the "module" mechanism of MDS and the "libicfsfs" extension of the client) without modifying the core code: the hot-association list is pushed through the extended attribute field of the caps mechanism, reusing the existing permission synchronization channel; the subtree migration coordinator is implemented based on the event callback interface of MON, decoupled from the dynamic subtree partitioning mechanism, to ensure compatibility.

[0158] It should also be noted that the optimized processing logic for MDS in abnormal scenarios in the optional embodiments of this application includes: 1) MDS node failure: The association summary table adopts dual storage of "memory + disk". The memory table is synchronized to the disk every 30 seconds (path is / var / lib / icfs / mds / {fs_name} / assoc_summary / ); if the MDS fails, the backup MDS loads the association summary table in the disk first when it starts. Missing entries are supplemented by gossip synchronization of other MDS nodes, and the recovery time is ≤10 seconds. 2) Prefetch request timeout: The client prefetch agent sets a timeout threshold (default 2 seconds). After the timeout, the request is marked as "to be retried" and retried according to the exponential backoff strategy (1s, 2s, 4s), up to 3 times; if it still fails, the metadata ID is recorded to the "low priority list" and will not be prefetched again within 10 minutes. 3) Subtree migration interruption: The source MDS generates a snapshot of the associated summary table before migration. If the migration is interrupted (e.g., due to network failure), the target MDS discards the received incomplete data, the source MDS restores the snapshot and re-triggers the migration, and records the reason for the interruption through the MON component. If there are 3 consecutive interruptions, the migration priority of the subtree will be temporarily increased and bandwidth will be allocated preferentially.

[0159] For example, a distributed cluster deploys a distributed storage system version, containing 3 MDS nodes (mds-01, mds-02, mds-03) and 10 clients, mainly running log collection (process PID=8080) and data analysis (process PID=9090) workloads. Performing operations such as metadata prefetching includes the following steps 1 to 5.

[0160] Step 1: System initialization and basic model training.

[0161] 1.1 Deploy the MDS association engine, client prefetch agent, and subtree migration coordinator plugin, and configure basic parameters (such as sliding window size = 1 hour, association threshold = 0.6).

[0162] 1.2 Upon initial startup, the MDS associated engine's log collection unit collects 24 hours of historical access data and performs offline calculations:

[0163] Spatial correlation S: The co-occurrence frequency of files and directories under the " / logs" directory reaches 15 times, and the initial value of S = 0.6;

[0164] Process dependency C: PID=8080 accesses " / logs" 80% of its total accesses, C initial value=0.7.

[0165] 1.3 Initialize the decay coefficient λ=0.01 of the time decay degree T (obtained by fitting historical data), and the dynamic weights α=0.3, β=0.4, γ=0.3.

[0166] Step 2: Online correlation calculation and hot list generation.

[0167] 2.1 The log collection unit intercepts access to " / logs / 2024-08-01" by PID=8080 in real time and records the triplet.<ID=1001,PID=8080, t=10:00:00> ;

[0168] 2.2 Every 10 seconds, the correlation calculation unit updates the correlation degree:

[0169] T(1001,1002) (1002 is the ID of " / logs / 2024-08-02"): Based on 3 accesses in the past hour, T=0.5 is calculated;

[0170] The prefetch priority P(1001,1002) = 0.3 × 0.6 + 0.4 × 0.5 + 0.3 × 0.7 = 0.59;

[0171] 2.3 Every 30 seconds, the list generation unit filters the top 20 metadata pairs, and "<1001,1002,0.59,5 minutes>" is selected into the hot-related list.

[0172] Step 3: Hotlist push and client prefetching.

[0173] 3.1 MDS-01 pushes the hot-association list to the client along with the access permissions of the " / logs" directory through the extended attribute fields of the caps mechanism;

[0174] 3.2 The client prefetches the proxy's list filtering unit and compares it with the local cache. If 1002 is not cached, the entry is retained.

[0175] 3.3 The batch prefetching unit groups according to MDS nodes (1002 belongs to mds-01), generates batch requests and compresses and encodes them, and sends them to mds-01;

[0176] 3.4 After prefetching is completed, the effect feedback unit records the cache status of 1002 and waits for subsequent access verification.

[0177] Step 4: Subtree migration and association collaboration.

[0178] 4.1 Due to excessive load on the " / logs" subtree, mds-01 triggered a subtree migration to mds-02;

[0179] 4.2 The data migration unit of the subtree migration coordinator requests the association summary table from mds-01 to obtain the association entries containing metadata such as 1001 and 1002;

[0180] 4.3 After receiving the MDS-02, the preload trigger unit loads the top 5 metadata (including 1002) into the cache in descending order of relevance.

[0181] 4.4 The location synchronization unit broadcasts the mapping relationship <1002,mds-02,6789> to all clients, and the clients update their local metadata location index.

[0182] Step 5: Optimize reinforcement learning weights.

[0183] 5.1 The effect feedback unit statistics show that the prefetch hit rate within 1 hour is 65% (below the threshold of 70%), and the invalid prefetch cost is 0.2 (below the threshold of 0.3).

[0184] 5.2 Adjusting dynamic weights in reinforcement learning algorithm: Increase β to 0.5 (to enhance the effect of time decay), decrease α to 0.2, and keep γ at 0.3;

[0185] 5.3 The system stability is calculated to be 92% (hit rate fluctuation = 3% over 10 consecutive minutes), and the learning rate is reduced by 50% to stabilize the parameters.

[0186] In summary, the beneficial effects of the technical solutions in the optional embodiments of this application are mainly reflected in four dimensions: performance, reliability, compatibility, and resource efficiency, as detailed below:

[0187] Performance improvements: By integrating spatiotemporal features, process context and subtree topology association through a multi-dimensional association expectation algorithm, the accuracy of metadata prefetching is improved by more than 40% compared with traditional directory prefetching, and the prefetching hit rate across MDS nodes is improved to more than 80%. Combined with batch prefetching and dynamic weight optimization, the latency of metadata access is reduced by 50%-70%, and the throughput in high-frequency access scenarios is increased by more than 3 times.

[0188] Enhanced reliability: The associated summary table adopts a dual storage mechanism of "memory + disk", and the recovery time of associated data after the MDS node crashes is ≤10 seconds; the snapshot rollback and retry strategy when the subtree migration is interrupted ensures that the metadata associated information is not lost; the exponential backoff retry mechanism for prefetch request timeout reduces prefetch failures caused by network fluctuations, and the overall system availability is improved to over 99.9%.

[0189] Excellent compatibility: It adopts a plug-in design and integrates through dynamic plug-in interfaces. It can be deployed directly without modifying the core logic such as MDS dynamic subtree partitioning and CASS mechanism. It is compatible with existing prefetching strategies and supports superimposed use without replacing existing system components.

[0190] Resource efficiency optimization: Lightweight synchronization based on correlation filtering (transmitting only entries with correlation ≥ 0.6) reduces data transmission across MDS nodes by up to 60%; dynamic weights adapt to different load scenarios, reducing invalid prefetch overhead by 70% and increasing MDS cache utilization to over 85%, significantly reducing memory and network resource waste.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0192] This embodiment also provides a metadata prefetching device for implementing the above embodiments and preferred embodiments, which will not be repeated hereafter. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0193] Figure 5 This is a framework diagram of a metadata prefetching device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0194] The acquisition module 52 is used to acquire the first information group corresponding to the metadata operation event in the target storage area of ​​each node, wherein each node is a metadata server node;

[0195] Update module 54 is used to update the three-dimensional correlation between any two metadata in each node according to the first information group;

[0196] Determining module 56 is used to determine the prefetch priority corresponding to any two metadata items through the updated three-dimensional correlation degree;

[0197] Instruction module 58 is used to generate a hot association list of metadata in each node according to the prefetch priority, and send the hot association lists corresponding to multiple nodes to the client respectively, so as to instruct the client to prefetch the target metadata in the multiple nodes according to the multiple hot association lists.

[0198] The aforementioned metadata prefetching device acquires a first information group corresponding to metadata operation events in the target storage area of ​​each node, wherein each node is a metadata server node. The three-dimensional correlation between any two metadata items in each node is updated based on the first information group. The prefetching priority common to the two metadata items is determined using the updated three-dimensional correlation. A hot association list of metadata in each node is generated based on the prefetching priority, and multiple hot association lists corresponding to multiple nodes are sent to the client to instruct the client to prefetch target metadata in multiple nodes according to the multiple hot association lists. This application solves the problem in related technologies where metadata prefetching methods in distributed storage systems rely on static path associations of metadata and have poor cross-node adaptability, resulting in low prefetching accuracy, thereby improving the prefetching accuracy of metadata.

[0199] In an exemplary embodiment, the device further includes a storage module, configured to intercept the metadata operation event via a kernel-mode hook function and extract the first information group corresponding to the metadata operation event before obtaining the first information group in the target storage area of ​​each node, wherein the first information group includes: a first identifier of the metadata operated by the metadata operation event, a second identifier of the access process executing the metadata operation event, and a timestamp of the metadata operation event; and store the first information group in the target storage area of ​​each node.

[0200] In an exemplary embodiment, the update module 54 is further configured to update the spatial correlation in the three-dimensional correlation based on the co-occurrence count of the first information group and any two metadata; and update the temporal decay in the three-dimensional correlation based on the first information group and the decay coefficient; and update the process correlation in the three-dimensional correlation based on the first information group.

[0201] Furthermore, the update module 54 is also used to determine the hierarchical distance between any two metadata items through the directory tree of the metadata corresponding to each node; and to determine the access counts corresponding to each of the two metadata items through the first information group; and to update the spatial correlation in the three-dimensional correlation degree through the co-occurrence count, the hierarchical distance, and the access count.

[0202] Specifically, the update module 54 is further configured to determine the larger access count among the access counts corresponding to any two metadata items, and determine the ratio of the co-occurrence count to the larger access count; and update the spatial correlation in the three-dimensional correlation using the ratio and the hierarchical distance.

[0203] Furthermore, the update module 54 is also used to determine the access time of the i-th access of the target metadata in any two metadata within the first time interval, wherein each node is allowed to obtain the first information group from the target storage area at intervals of the first time interval; and to update the time decay degree in the three-dimensional correlation degree by the access frequency corresponding to the i-th access, the second time interval and the decay coefficient, wherein the second time interval is the time interval between the current time and the access time.

[0204] Optionally, the update module 54 is further configured to determine the contribution ratio of the time decay degree in the prefetch priority; if the contribution ratio is higher than the preset ratio for m consecutive periods, the decay coefficient is reduced, where m is a positive integer.

[0205] Furthermore, the update module 54 is also used to determine the process sets that access the two metadata respectively through the first information group; and to update the process correlation degree in the three-dimensional correlation degree through the process sets corresponding to the two metadata respectively.

[0206] Specifically, the update module 54 is also used to determine the intersection and union between the process sets corresponding to any two metadata; and to update the process correlation degree in the three-dimensional correlation degree through the intersection and the union.

[0207] In an exemplary embodiment, the determining module 56 is further configured to determine the dynamic weights corresponding to the spatial correlation, the time decay, and the process correlation, respectively; and to perform a weighted calculation on the updated spatial correlation, the updated time decay, and the updated process correlation using the dynamic weights to obtain the prefetch priority corresponding to any two metadata.

[0208] In an exemplary embodiment, the instruction module 58 is further configured to filter multiple metadata pairs in descending order of prefetch priority, wherein the prefetch priority of each metadata pair is higher than the prefetch priority of other metadata pairs in each node besides the multiple metadata pairs; generate the hot association list through the multiple metadata pairs, wherein the hot association list includes: a third identifier of the metadata in each metadata pair, the prefetch priority, and the validity period, wherein the validity period is determined by the prefetch priority.

[0209] In an exemplary embodiment, the instruction module 58 is further configured to instruct the client to filter the hot-related lists pushed by multiple nodes respectively through a metadata index to obtain a list to be prefetched, wherein the metadata index is an index of metadata cached locally by the client; and to instruct the client to send a batch prefetch request to the multiple nodes according to the correspondence between the fourth identifier of the metadata included in the list to be prefetched and the multiple nodes, so as to prefetch the target metadata in the multiple nodes, wherein the batch prefetch request is encoded through a predefined binary protocol.

[0210] In an exemplary embodiment, the instruction module 58 is further configured to instruct the client to group the metadata included in the list to be prefetched based on the correspondence, to obtain multiple groups, wherein each group uniquely corresponds to a target identifier in the node identifiers of the multiple nodes; and to instruct the client to send a batch prefetch request to the node corresponding to the target identifier, wherein the batch prefetch request is a batch prefetch request for the metadata in each group.

[0211] In an exemplary embodiment, the apparatus further includes: a subtree migration module, configured to, after determining the prefetch priority corresponding to any two metadata items through the updated three-dimensional correlation degree, receive a first acquisition request sent by a subtree migration coordinator when subtree migration is triggered at each node, wherein the first acquisition request is used to request acquisition of an association summary table, wherein the association summary table includes: association entries of all metadata of the subtree; and send the association summary table to the subtree migration coordinator to instruct the subtree migration coordinator to send the association summary table to the target node of the subtree migration.

[0212] In an exemplary embodiment, the subtree migration module is further configured to, after determining the prefetch priority corresponding to any two metadata items through the updated three-dimensional association degree, parse the association summary table sent by the source node of the subtree migration through the subtree migration coordinator when each node is the target node of the subtree migration; generate a preload instruction for the first n metadata pairs in the parsed association summary table, wherein the first n metadata pairs are the first n metadata pairs in the parsed association summary table sorted from high to low prefetch priority, and n is a positive integer; and preload the first n metadata pairs into memory through the preload instruction.

[0213] In an exemplary embodiment, the subtree migration module is further configured to generate a metadata location mapping table corresponding to the first n metadata pairs; and broadcast the metadata location mapping table to the client through a preset broadcast mechanism.

[0214] For a description of the features in the embodiment corresponding to the metadata prefetching device, please refer to the relevant description in the embodiment corresponding to the metadata prefetching method, which will not be repeated here.

[0215] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described metadata prefetching method embodiments.

[0216] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described metadata prefetching method embodiments at runtime.

[0217] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0218] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described metadata prefetching method embodiments.

[0219] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described metadata prefetching method embodiments.

[0220] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0221] The foregoing has provided a detailed description of a metadata prefetching method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for prefetching metadata, characterized in that, include: Obtain the first information group corresponding to the metadata operation event in the target storage area of ​​each node, wherein each node is a metadata server node; The three-dimensional correlation between any two metadata items in each node is updated based on the first information group; The prefetch priority corresponding to any two metadata items is determined by the updated three-dimensional correlation degree. Based on the prefetch priority, a hot association list of metadata in each node is generated. These hot association lists for multiple nodes are then sent to the client, instructing the client to prefetch the target metadata from the multiple nodes according to these hot association lists. Update the three-dimensional correlation between any two metadata items in each node based on the first information group, including: The spatial correlation in the three-dimensional correlation is updated based on the co-occurrence frequency of the first information group and any two metadata items; and, The time decay factor in the three-dimensional correlation is updated based on the first information group and the decay coefficient; and... The process correlation in the three-dimensional correlation is updated based on the first information group.

2. The metadata prefetching method according to claim 1, characterized in that, Before obtaining the first information group corresponding to the metadata operation event in the target storage area of ​​each node, the method further includes: The kernel-mode hook function intercepts the metadata operation event and extracts the first information group corresponding to the metadata operation event. The first information group includes: a first identifier of the metadata operated by the metadata operation event, a second identifier of the access process that executed the metadata operation event, and a timestamp of the metadata operation event. The first information group is stored in the target storage area of ​​each node.

3. The metadata prefetching method according to claim 1, characterized in that, The spatial correlation in the three-dimensional correlation is updated based on the co-occurrence frequency of the first information group and any two metadata, including: The hierarchical distance between any two metadata items is determined by the directory tree of the metadata corresponding to each node; and the number of accesses corresponding to each of the two metadata items is determined by the first information group. The spatial correlation in the three-dimensional correlation is updated using the co-occurrence count, the hierarchical distance, and the number of visits.

4. The metadata prefetching method according to claim 3, characterized in that, The spatial correlation in the three-dimensional correlation is updated using the co-occurrence count, the hierarchical distance, and the number of visits, including: Determine the larger access count among the access counts corresponding to any two metadata entries, and determine the ratio of the co-occurrence count to the larger access count; The spatial correlation in the three-dimensional correlation is updated using the ratio and the hierarchical distance.

5. The metadata prefetching method according to claim 1, characterized in that, The time decay factor in the three-dimensional correlation is updated based on the first information group and the decay coefficient, including: Determine the access time of the i-th access to the target metadata in any two metadata within a first time interval, wherein each node is allowed to retrieve the first information group from the target storage area at intervals of the first time interval; The time decay factor in the three-dimensional correlation is updated by using the access frequency corresponding to the i-th access, the second time interval, and the decay coefficient, wherein the second time interval is the time interval between the current time and the access time.

6. The metadata prefetching method according to claim 5, characterized in that, After updating the time decay factor in the three-dimensional correlation degree based on the first information group and the decay coefficient, the method further includes: Determine the contribution percentage of the time decay rate in the prefetch priority; If the contribution ratio is higher than the preset ratio for m consecutive periods, the attenuation coefficient is reduced, where m is a positive integer.

7. The metadata prefetching method according to claim 1, characterized in that, Update the process correlation degree in the three-dimensional correlation degree according to the first information group, including: The first information group determines the set of processes that access any two metadata entries respectively; The process correlation in the three-dimensional correlation is updated by using the process sets corresponding to any two metadata items.

8. The metadata prefetching method according to claim 7, characterized in that, The process correlation in the three-dimensional correlation is updated by using the process sets corresponding to any two metadata items, including: Determine the intersection and union of the process sets corresponding to any two metadata sets; The process correlation degree in the three-dimensional correlation degree is updated by the intersection and the union.

9. The metadata prefetching method according to claim 1, characterized in that, The prefetch priority corresponding to any two metadata items is determined by the updated three-dimensional correlation, including: Determine the dynamic weights corresponding to the spatial correlation, the temporal decay, and the process correlation, respectively; The updated spatial correlation, the updated temporal decay, and the updated process correlation are weighted by the dynamic weights to obtain the prefetch priority corresponding to any two metadata.

10. The metadata prefetching method according to claim 1, characterized in that, Based on the prefetch priority, a hot-association list of metadata in each node is generated, including: Multiple metadata pairs are selected in descending order of prefetch priority, wherein the prefetch priority of each metadata pair is higher than the prefetch priority of other metadata pairs in each node besides the multiple metadata pairs. The hot association list is generated from the multiple metadata pairs, wherein the hot association list includes: a third identifier of the metadata in each metadata pair, the prefetch priority, and the validity period, wherein the validity period is determined by the prefetch priority.

11. The metadata prefetching method according to claim 1, characterized in that, The client is instructed to prefetch target metadata from the plurality of nodes according to the plurality of hot-association lists, including: The client is instructed to filter the hot-related lists pushed by multiple nodes through a metadata index to obtain a list to be prefetched, wherein the metadata index is the index of the metadata cached locally by the client; The client is instructed to send a batch prefetch request to the multiple nodes according to the correspondence between the fourth identifier of the metadata included in the prefetch list and the multiple nodes, so as to prefetch the target metadata in the multiple nodes, wherein the batch prefetch request is encoded through a predefined binary protocol.

12. The metadata prefetching method according to claim 11, characterized in that, The client is instructed to send a batch prefetch request to the multiple nodes according to the correspondence between the fourth identifier of the metadata included in the prefetch list and the multiple nodes, including: The client is instructed to group the metadata included in the list to be prefetched based on the correspondence, resulting in multiple groups, wherein each group uniquely corresponds to the target identifier in the node identifier of the multiple nodes; The client is instructed to send a bulk prefetch request to the node corresponding to the target identifier, wherein the bulk prefetch request is a bulk prefetch request for the metadata in each group.

13. The metadata prefetching method according to claim 1, characterized in that, After determining the prefetch priority corresponding to any two metadata items through the updated three-dimensional correlation, the method further includes: In the event that each node triggers a subtree migration, a first retrieval request is received from the subtree migration coordinator, wherein the first retrieval request is used to request the retrieval of an association summary table, wherein the association summary table includes: association entries of all metadata of the subtree; Send the association summary table to the subtree migration coordinator to instruct the subtree migration coordinator to send the association summary table to the target node of the subtree migration.

14. The metadata prefetching method according to claim 1, characterized in that, After determining the prefetch priority corresponding to any two metadata items through the updated three-dimensional correlation, the method further includes: In the case where each node is the target node of the subtree migration, the association summary table sent by the source node of the subtree migration through the subtree migration coordinator is parsed. Generate a preload instruction for the first n metadata pairs in the parsed association summary table, wherein the first n metadata pairs are the first n metadata pairs in the parsed association summary table after prefetching priority is sorted from high to low, and n is a positive integer; The first n metadata pairs are preloaded into memory using the preload instruction.

15. The metadata prefetching method according to claim 14, characterized in that, After preloading the first n metadata pairs into memory via the preloading instruction, the method further includes: Generate a metadata location mapping table corresponding to the first n metadata pairs; The metadata location mapping table is then broadcast to the client via a preset broadcast mechanism.

16. A metadata prefetching device, characterized in that, include: The acquisition module is used to acquire the first information group corresponding to the metadata operation event in the target storage area of ​​each node, wherein each node is a metadata server node; The update module is used to update the three-dimensional correlation between any two metadata in each node based on the first information group; The determination module is used to determine the prefetch priority corresponding to any two metadata items based on the updated three-dimensional correlation degree. The instruction module is used to generate a hot association list of metadata in each node according to the prefetch priority, and send the hot association lists corresponding to multiple nodes to the client respectively, so as to instruct the client to prefetch the target metadata in the multiple nodes according to the multiple hot association lists. The update module is further used to update the spatial correlation degree in the three-dimensional correlation degree according to the first information group and the co-occurrence frequency of any two metadata; and to update the temporal decay degree in the three-dimensional correlation degree according to the first information group and the decay coefficient; and to update the process correlation degree in the three-dimensional correlation degree according to the first information group.

17. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the metadata prefetching method as described in any one of claims 1 to 15 when executing the computer program.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the metadata prefetching method as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the metadata prefetching method as described in any one of claims 1 to 15.

Images