Data management method and device, electronic equipment and computer readable storage medium

Abstract

A data management method, a data management device, an electronic device and a computer readable storage medium. The data management method comprises: in the metadata management node, in response to the directory operation for the first directory segment, determining the current hotness value of the first directory segment, wherein the first directory segment is any one of the plurality of directory segments in the directory tree managed by the metadata management node; based on the current hotness value of the first directory segment, updating the corresponding hot spot directory set of the metadata management node, and the hot spot directory set comprises N directory entries corresponding to the top N directory segments with the highest hotness in the plurality of directory segments respectively.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a data management method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In the era of big data, distributed storage systems have become the core infrastructure supporting petabyte (PB) and even exabyte (EB) level data storage, processing, and access. Ceph distributed storage system, as a highly scalable and reliable unified storage solution, and its derivative CephFS (Ceph File System), with its excellent compatibility with the POSIX standard, effectively address the shortcomings of traditional distributed file systems in data consistency and storage scalability, becoming one of the important technological directions in the current distributed storage field.

[0003] The CephFS distributed file system is built on the Ceph distributed storage system and mainly consists of three core components: a client, a metadata server (MDS), and an object storage daemon (OSD). These components work together to achieve efficient storage and access of file data. The metadata server (MDS) is the key to ensuring the normal operation of the file system. It is mainly responsible for managing the file system's namespace (i.e., directory tree structure) and handling various metadata operations, including but not limited to file search, directory reading, file creation, and file deletion. To cope with the metadata access pressure in large-scale storage scenarios and improve the system's concurrent processing capabilities, CephFS supports a multi-active MDS deployment mode. In this mode, the file system's directory tree is divided into multiple independent subtrees. The metadata for each subtree is managed by different MDS Ranks (MDS management nodes), thereby achieving load balancing for metadata management and further improving the system's scalability and response performance. Summary of the Invention

[0004] At least one embodiment of this disclosure provides a data management method for a metadata management node. The method includes: in the metadata management node, in response to a directory operation on a first directory segment, determining the current popularity value of the first directory segment, wherein the first directory segment is any one of a plurality of directory segments in a directory tree managed by the metadata management node; and updating a hotspot directory set corresponding to the metadata management node based on the current popularity value of the first directory segment, wherein the hotspot directory set includes N directory entries corresponding to the top N most popular directory segments among the plurality of directory segments, where N is a positive integer.

[0005] For example, in at least one example of the data management method provided in the above embodiments of this disclosure, the method further includes: in response to a file operation on a first file, determining the current popularity value of the first file, wherein the first file is any one of a plurality of files in the directory tree; and updating the hot file set corresponding to the metadata management node based on the current popularity value of the first file, wherein the hot file set includes N file entries corresponding to the top N most popular files in the plurality of files.

[0006] For example, in a data management method provided by at least one example of the above embodiments of this disclosure, determining the current popularity value of the first directory segment in response to a directory operation on the first directory segment includes: updating the count value of the first directory segment using a decay counter in response to a directory operation on the first directory segment, wherein the decay counter is used to count the number of times the first directory segment is operated on and decays the count value over time; and determining the current popularity value of the first directory segment based on the updated count value.

[0007] For example, in a data management method provided in at least one example of the above embodiments of this disclosure, updating the count value of the first directory segment using a decay counter in response to a directory operation on the first directory segment includes: updating the count value of the first directory segment in response to any one of a plurality of directory operations, wherein the plurality of directory operations includes at least one of the following: a read operation on file metadata managed by the first directory segment, a write operation on file metadata managed by the first directory segment, a read operation on the first directory segment, loading metadata of the first directory segment from a disk, and writing metadata of the first directory segment to the disk.

[0008] For example, in the data management method provided in at least one example of the above embodiments of this disclosure, multiple decay counters are set for the various directory operations, and the multiple decay counters are used to record the count values ​​of the various directory operations. Updating the count value of the first directory segment in response to any one of the various directory operations includes: updating the count value of the first directory operation in response to the first directory operation, and obtaining the current count value of each of the other directory operations besides the first directory operation. Determining the popularity value of the first directory segment based on the updated count value includes: calculating the weighted sum of the current count values ​​of the various directory operations to obtain the popularity value of the first directory segment.

[0009] For example, in at least one example of the data management method provided in the above embodiments of this disclosure, the method further includes: storing the hotspot directory set using a first data structure and a second data structure, wherein the first data structure is a forward ordered mapping structure that sorts directory segments according to their popularity values, and the second data structure is a reverse hash mapping structure that points from directory segments to their corresponding popularity values.

[0010] For example, in a data management method provided by at least one example of the above embodiments of this disclosure, updating the hotspot directory set corresponding to the metadata management node based on the current popularity value of the first directory segment includes: if the hotspot directory set that has not been updated already contains a first entry corresponding to the first directory segment, then updating the popularity value of the first entry to the current popularity value, and updating the sorting position of the first directory segment based on the current popularity value; and if the hotspot directory set that has not been updated does not contain an entry corresponding to the first directory segment, then inserting the entry corresponding to the first directory segment into the hotspot directory set, and deleting the entry corresponding to the lowest popularity value from the hotspot directory set when the number of entries contained in the hotspot directory set is greater than N.

[0011] For example, in at least one example of the data management method provided in the above embodiments of this disclosure, the method further includes: in response to a second directory segment among the plurality of directory segments being deleted from the cache, deleting a second entry corresponding to the second directory segment from the hotspot directory set, wherein the second directory segment is any one of the plurality of directory segments.

[0012] For example, in the data management method provided in at least one example of the above embodiments of this disclosure, each of the N directory entries includes one or more of the following: the popularity value of the directory segment, the complete path of the directory segment, the identifier of the directory segment, the operational load component directly borne by the directory segment, the nested load of the directory segment, and the subtree load.

[0013] For example, in the data management method provided in at least one example of the above embodiments of this disclosure, the method further includes: updating the popularity value of the directory segment corresponding to the second file in response to the popularity value update of the second file in the plurality of files, wherein the second file is any one of the plurality of files; and updating the hotspot directory set corresponding to the metadata management node based on the updated popularity value of the directory segment corresponding to the second file.

[0014] For example, in a data management method provided in at least one example of the above embodiments of this disclosure, determining the current popularity value of the first file in response to a file operation on the first file includes: updating the count value of the first file using a decay counter in response to a file operation on the first file, wherein the decay counter is used to count the number of times the first file is operated on and decays the count value over time; and determining the popularity value of the first file based on the updated count value.

[0015] For example, in a data management method provided in at least one example of the above embodiments of this disclosure, updating the count value of the first file using a decay counter in response to a file operation on the first file includes: updating the count value of the first file in response to any one of a plurality of file operations, wherein the plurality of file operations includes: a read operation on the metadata of the first file and a write operation on the metadata of the first file; wherein multiple decay counters are set for each of the plurality of file operations, and the multiple decay counters are used to record the count values ​​of the plurality of file operations respectively. Updating the count value of the first file in response to any one of the plurality of file operations includes: updating the count value of the first file operation in response to the first file operation among the plurality of file operations, and obtaining the current count value of each of the remaining file operations besides the first file operation among the plurality of file operations. Determining the popularity value of the first file based on the updated count value includes: calculating a weighted sum of the current count values ​​of the plurality of file operations to obtain the popularity value of the first file.

[0016] For example, in the data management method provided by at least one example of the above embodiments of this disclosure, each of the N file entries includes one or more of the following: file popularity value, file number, file full path, read operation decay count value, and write operation decay count value.

[0017] At least one embodiment of this disclosure provides another data management method for managing a management system with multiple metadata management nodes. The method includes: in response to a cross-node aggregation request, determining multiple target metadata management nodes to be aggregated; sending a query instruction to each of the multiple target metadata management nodes, and receiving multiple hotspot directory sets respectively fed back by the multiple nodes, wherein the hotspot directory set includes N directory entries corresponding to the top N most popular directory segments in a directory tree managed by the node; and generating an aggregation result based on the multiple hotspot directory sets.

[0018] For example, in a data management method provided by at least one example of the above embodiments of this disclosure, generating an aggregation result based on the plurality of hotspot directory sets includes: merging the entries in the plurality of hotspot directory sets to obtain a merge result; and determining a global hotspot directory set based on the merge result, wherein the global hotspot directory set includes N directory entries corresponding to the N most popular directory segments in the merge result.

[0019] For example, in the data management method provided in at least one example of the above embodiments of this disclosure, merging entries in the plurality of hotspot directory sets to obtain a merging result includes: for a fourth directory fragment appearing in a node, determining the entry corresponding to the fourth directory fragment in the merging result based on the popularity value of the fourth directory fragment in its own node; for a fifth directory fragment appearing in at least two nodes, determining the entry corresponding to the fifth directory fragment in the merging result based on the highest popularity value of the fifth directory fragment in the at least two nodes.

[0020] At least one embodiment of this disclosure provides a data management apparatus for a metadata management node. The apparatus includes a popularity determination module and a set update module. The popularity determination module is configured to determine the current popularity value of a first directory segment in response to a directory operation on the first directory segment in the metadata management node, wherein the first directory segment is any one of a plurality of directory segments in a directory tree managed by the metadata management node. The set update module is configured to update the hotspot directory set corresponding to the metadata management node based on the current popularity value of the first directory segment. The hotspot directory set includes N directory entries corresponding to the N most popular directory segments among the plurality of directory segments, where N is a positive integer.

[0021] At least one embodiment of this disclosure provides another information processing apparatus for managing a management system with multiple metadata management nodes. The apparatus includes a node determination module, a sending and receiving module, and an aggregation module. The node determination module is configured to determine multiple target metadata management nodes to be aggregated in response to a cross-node aggregation request. The sending and receiving module is configured to send a query instruction to each of the multiple target metadata management nodes and receive multiple hotspot directory sets respectively fed back by the multiple nodes. The hotspot directory set includes N directory entries corresponding to the top N most popular directory segments in the directory tree managed by the node. The aggregation module is configured to generate an aggregation result based on the multiple hotspot directory sets.

[0022] At least one embodiment of this disclosure provides an electronic device, including a processor; a memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the processor to implement the data management method provided in any embodiment of this disclosure.

[0023] At least one embodiment of this disclosure provides a computer-readable storage medium storing non-transitory computer-readable instructions, which, when executed by a computer, can implement the data management method provided in any embodiment of this disclosure. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0025] Figure 1 A flowchart is shown illustrating a data management method for a metadata management node according to at least one embodiment of this disclosure;

[0026] Figure 2 The diagram illustrates two data structures provided in at least one embodiment of this disclosure;

[0027] Figure 3 A flowchart illustrating an update hotspot directory set provided in at least one embodiment of this disclosure is shown;

[0028] Figure 4 A flowchart of another data management method provided by at least one embodiment of this disclosure is shown;

[0029] Figure 5 A flowchart of another data management method provided by at least one embodiment of this disclosure is shown;

[0030] Figure 6A schematic block diagram of a data management apparatus provided in at least one embodiment of the present disclosure is shown;

[0031] Figure 7 A schematic block diagram of another data management apparatus provided in at least one embodiment of the present disclosure is shown;

[0032] Figure 8A A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure is shown;

[0033] Figure 8B A schematic block diagram of another electronic device provided in at least one embodiment of the present disclosure is shown; and

[0034] Figure 9 A schematic diagram of a computer-readable storage medium provided in at least one embodiment of the present disclosure is shown. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0036] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0037] To monitor the operational status of the CephFS distributed file system, some implementations use a Perf Counter to count key parameters. This counter serves as a global performance counter at the Metadata Server (MDS) level, collecting real-time global performance metrics such as total requests, total latency, and cache hit rate. This data can be collected and displayed through monitoring systems like Prometheus. In other implementations, the `dump loads` command can be used to view the load vectors of all cache directories within the system; however, the output data is raw and unprocessed, requiring manual analysis and filtering by the administrator. Still other implementations use the `fs perf stats` (File System Performance Statistics) command, which aggregates client-level performance metrics, including read / write latency and cache hit rate, through the MGR Stats module (Manager Statistics). However, this command does not involve directory-level performance monitoring and cannot reflect the access load of individual directories.

[0038] Based on the above implementation methods, access popularity tracking can be achieved per-directory within the MDS, and access load-related data for each directory can be collected and stored. However, this data is only used for load balancing logic scheduling within the MDS and is not directly queried by administrators and operations personnel. This makes it difficult for relevant personnel to quickly locate hot directories in the current system, which is detrimental to operation and maintenance management and system optimization. Although the `dump loads` command can obtain load-related data for all directories in the system, the information obtained in this way has obvious limitations: the output directory load data lacks effective filtering and sorting, and contains a large amount of information about directories with low access frequency, requiring administrators to invest a lot of effort in manual identification; in a production environment with millions of directories, the amount of data output by this command is extremely large, causing great inconvenience to the administrator's data analysis work; at the same time, there is a lack of a comprehensive evaluation and ranking mechanism for the access popularity of each directory, making it difficult for relevant personnel to quickly identify the "hottest" directories with the highest access volume and the most concentrated load, and making it impossible to take timely optimization measures for hot directories, thus affecting the overall operating efficiency of the system and the timeliness of operation and maintenance management.

[0039] At least one embodiment of this disclosure provides a data management method and apparatus for a metadata management node, a data management method and apparatus for a management system that manages multiple metadata management nodes, an electronic device, and a computer-readable storage medium. The data management method for a metadata management node includes: in the metadata management node, in response to a directory operation on a first directory segment, determining the current popularity value of the first directory segment, wherein the first directory segment is any one of multiple directory segments in a directory tree managed by the metadata management node; and updating a hotspot directory set corresponding to the metadata management node based on the current popularity value of the first directory segment, the hotspot directory set including N directory entries corresponding to the top N most popular directory segments among the multiple directory segments, where N is an integer greater than 1.

[0040] In this data management method, whenever a directory segment in the metadata management node undergoes a directory operation, its popularity value can be updated, further updating the hot directory set. Based on this scheme, on the one hand, the top-N popular directory segments in the metadata management node can be selected, supporting quantitative sorting and rapid filtering of hot directories, automatically eliminating invalid and low-load data, eliminating the need for manual sorting and filtering, and quickly and accurately locating core access hot directories; on the other hand, when the first directory segment is operated on while the other directory segments are not currently operated on, the count value of the first directory segment increases, while the count values ​​of the other directory segments naturally decrease, causing a change in the ranking of the hot directory set. The new score of the first directory segment is used to re-rank the hot directory set, achieving incremental updates to the hot directory set without recalculating the popularity value of all directory segments, resulting in lower computational overhead.

[0041] As described above, in Multi-Active MDS mode, the directory tree is divided into multiple subtrees. Each MDS Rank is responsible for managing the metadata of a portion of the subtrees. Each MDS Rank is a metadata management node, and each metadata management node corresponds to an active MDS (optionally, each metadata management node can also correspond to at least one standby MDS). When a client accesses different directories, it is automatically routed to the corresponding responsible MDS Rank.

[0042] Figure 1 A flowchart is shown of a data management method for a metadata management node provided in at least one embodiment of the present disclosure.

[0043] like Figure 1 As shown, the data management method includes steps S110 to S130.

[0044] Step S110: In the metadata management node, in response to the directory operation for the first directory fragment, determine the current popularity value of the first directory fragment, where the first directory fragment is any one of multiple directory fragments in the directory tree managed by the metadata management node.

[0045] Step S120: Based on the current popularity value of the first directory fragment, update the hot directory set corresponding to the metadata management node. The hot directory set includes the N directory entries corresponding to the top N most popular directory fragments among multiple directory fragments.

[0046] For example, in some embodiments, the data management method further includes: in response to a popularity query request, the metadata management node outputs a corresponding set of hotspot directories. The popularity query request is, for example, a request initiated by a client.

[0047] For example, the metadata management node in step S110 is the MDS Rank mentioned above. The directory tree managed by the metadata management node is a subtree (or subdirectory tree) within the overall directory tree. The overall directory tree is a hierarchical structure diagram of the entire storage system, presenting a nested tree structure. At the top is the root directory (root), below which are first-level folders, second-level folders, and so on (branches), and at the bottom are files / innermost folders (leaves). A subtree is a small branch tree formed by starting from an intermediate folder and including the intermediate folder itself, all its subordinate folders, and all files. An intermediate folder is a folder located between the root directory and the innermost folder.

[0048] For example, the subtree managed by the metadata management node includes multi-level folders, such as second-level folders, third-level folders, etc., and any folder in each level can be considered a directory fragment. For example, the subtree managed by the metadata management node includes a / b / c and d, where a is a first-level folder, b is a second-level folder, and c and d are third-level folders. Any one of a, b, c, and d can be considered a directory fragment. In some of the following embodiments, directory fragments are also referred to simply as directories.

[0049] For example, a metadata management node corresponds to an active metadata server. The metadata server has a memory cache used to cache directory objects and inode objects (index nodes that store file metadata). Directory objects include folders, file names, parent-child directory relationships, etc., while inode objects include file size, permissions, timestamps, locations, etc. For example, the multiple directory fragments in step S110 are all or part of the directory fragments (directory fragments can be represented as CDirs) located in the cache of the metadata server in the subtree managed by the metadata management node.

[0050] For example, a set of hot directories is maintained within the metadata management node. This set is established through the following process: The popularity value of each directory segment among multiple directory segments in the metadata management node is determined. Based on the popularity value, the multiple directory segments are sorted, and the N most popular directory segments are selected. The directory entries corresponding to these N directory segments are used as the hot directory set. Each directory entry records relevant information for that directory segment. After establishing the hot directory set, whenever any directory segment among these multiple directory segments is operated on, its popularity value is updated, and the hot directory set is further updated based on the updated popularity value.

[0051] For example, in step S110, in response to a directory operation on any directory segment, the current popularity value of that directory segment is determined within the metadata management node. Taking the first directory segment among multiple directory segments as an example, in response to a directory operation on the first directory segment, a decay counter is used to update the count value of the first directory segment. The decay counter is used to count the number of times the first directory segment has been operated on, and the count value decays over time. Based on the updated count value, the current popularity value of the first directory segment is determined. The first directory segment can be any one of multiple directory segments; the method for obtaining the current popularity value of other directory segments can refer to the first directory segment. In some of the following embodiments, the popularity value is also referred to as a popularity score or simply a score.

[0052] For example, in step S110, a decay counter is used to count the number of times the first directory segment is operated on within the current time window. The current time window can be a period of time prior to the current moment, such as the time window from the moment the popularity value was last determined to the current moment. Within this time window, the decay counter is used to record the number of times the first directory segment is operated on. In response to each operation, the count value increases by a predetermined value (e.g., 1). The operation here refers to operations such as reading the directory and reading file metadata, as detailed in the following embodiments. The counter has a decay function, decreasing the count value over time. For example, if no new operation is added within a predetermined duration (e.g., 5 seconds), the count value is halved. In this way, by increasing the count value when the directory segment is operated on and decreasing the count value over time, the popularity value can reflect the true operation frequency of the directory segment, and the determined popularity value is more consistent with the recent actual popularity situation. Furthermore, the count value reflects the access popularity within the recent time window rather than the historical cumulative total.

[0053] For example, for each directory segment cached in the MDS, the following four sets of popularity vectors are maintained: pop_me (the operational load directly borne by this directory itself), pop_nested (the nested load including this directory itself and all its subdirectories), pop_auth_subtree (the load within the authentication subtree of this MDS), and pop_auth_subtree_nested (the load of the nested authentication subtree). For each set of popularity vectors, the popularity of the directory segment is determined using a DecayCounter (exponential decay counter). For example, the popularity value of the first directory segment mentioned above refers to the popularity value corresponding to pop_me, that is, the popularity value corresponding to the operational load borne by the first directory segment itself, reflecting the popularity of the first directory segment itself, but excluding the popularity of its subdirectories. For example, in response to a read operation on the first directory segment itself, the count value of the first directory segment is increased, while for a read operation on any subdirectory under the first directory segment, the count value of the first directory segment is not increased. In some embodiments, for other popularity vectors, corresponding popularity values ​​can also be determined, but the popularity values ​​of other popularity vectors are not used for sorting, but can be used to simultaneously display to the user for reference. In the following embodiments, the directory segments are sorted by the popularity vector pop_me's heat value. Therefore, pop_me is used as an example for illustration. For other popularity vectors, please refer to the description for pop_me.

[0054] For example, in response to a directory operation on a first directory segment, updating the count value of the first directory segment using a decay counter includes: updating the count value of the first directory segment in response to any one of a plurality of directory operations, wherein the plurality of directory operations includes at least one of the following: a read operation on the file metadata managed by the first directory segment (IRD), a write operation on the file metadata managed by the first directory segment (IWR), a read operation on the first directory segment (READDIR), loading the metadata of the first directory segment from disk (FETCH), and writing the metadata of the first directory segment to disk (STORE). For example, the file metadata here can be understood as an inode storing the file metadata, and the read / write operation on the file metadata can be understood as an inode read operation / inode write operation. An inode read operation includes finding a file and querying file attributes, and an inode write operation includes creating a file, deleting a file, and renaming a file. The metadata of the first directory segment includes information such as folder permissions, owner, folder size, creation / modification time, and all filenames in the folder.

[0055] For example, multiple decay counters are set for different directory operations, and these multiple decay counters are used to record the count values ​​of different directory operations. For pop_me in the above four sets of popularity vectors, five independent decay counters are set for each directory segment, which are used to count the above five directory operations. For each decay counter, the count value is increased by a predetermined value each time a corresponding operation is detected. These counters are updated by the hit_dir() method of the load balancer (MDBalancer) for each operation, and the historical popularity value is automatically reduced using the following exponential decay formula (1).

[0056] (1)

[0057] In the formula, val(t) represents the heat value at the current time t, which is the decayed counter value; val(t0) represents the heat value at the last recorded time t0, which is the historical heat baseline value; t represents the current time; t0 represents the time when the heat was last updated or calculated; Δt represents the time difference, usually equal to t - t0; e represents the natural constant, approximately equal to 2.71828; k represents the exponential decay coefficient, which determines the rate of heat decrease. Since ln(0.5) is negative, k is also negative, and the heat will decrease over time; T half This indicates the half-life, which represents the time required for the calorific value to decrease to half its original value. For example, T... half = 5 seconds, so after a directory segment is no longer accessed, its popularity value will be halved every 5 seconds.

[0058] For example, while updating the count value of the first directory operation, the current count value of each of the other directory operations besides the first directory operation is obtained to get the current count value of each of the five directory operations mentioned above. The weighted sum of the count values ​​of these five directory operations is calculated to obtain the popularity value of the first directory segment. For example, the comprehensive popularity value of the first directory segment is calculated using the following formula (2).

[0059] (2)

[0060] In the formula, the five components IRD, IWR, READDIR, FETCH, and STORE represent the decayed count values ​​of the five directory operations mentioned above. Each component is the current value after exponential decay and reflects the access popularity in the recent time window rather than the historical cumulative total.

[0061] For example, for the popularity vector `pop_me`, the five operations mentioned above on the first directory segment refer to operations on the first directory segment itself, excluding operations on its subdirectories. For instance, read / write operations on file metadata managed by the first directory segment refer to read / write operations on file metadata directly managed by the first directory segment, excluding read / write operations on file metadata managed by subdirectories of the first directory segment; read operations on the first directory segment refer to read operations on the first directory segment itself, excluding read operations on subdirectories of the first directory segment.

[0062] For example, for the remaining popularity vectors pop_nested, pop_auth_subtree, and pop_auth_subtree_nested, the counting ranges for the five operations are different and different from the counting range of pop_me. For example, for pop_nested, its counting range is the directory itself and all subdirectories under the directory. In response to read or write operations on the directory itself or any subdirectory under the directory, its corresponding count value is increased. For pop_auth_subtree, its counting range is the scope of the authentication subtree of the MDS. In response to operations within the authentication subtree, its corresponding count value is increased. For example, for each of the four popularity vectors mentioned above, five independent decay counters are set. These five independent decay counters are used to count the five directory operations, and the count values ​​of the five decay counters are weighted and summed to obtain the overall popularity of each popularity vector. The overall popularity of pop_me is used for sorting in subsequent operation S120.

[0063] For example, after determining the current popularity value (pop_me popularity value) of the first directory segment in step S110, in step S120, the hot directory set corresponding to the metadata management node is updated based on the current popularity value of the first directory segment. When a client queries popularity information, the current hot directory set can be output to the client. Furthermore, when any directory segment among multiple directory segments is operated on (such as the five directory operations mentioned above), the information of the hot directory set is updated again using steps S110~S120, and the updated hot directory set can be fed back to the client again. The value of N can be set according to actual needs, for example, it can be 100.

[0064] For example, any one of the N directory entries includes: the score of this directory segment, the full path of this directory segment, the identifier of this directory segment, the operation load component directly borne by this directory segment, and the nested load and subtree load of this directory segment (pop_nested, pop_auth_subtree, pop_auth_subtree_nested).

[0065] According to at least one embodiment of this disclosure, whenever a directory segment in the metadata management node is subjected to a directory operation, the popularity value of the directory segment can be updated, and the hot directory set can be further updated. Based on this scheme, on the one hand, the directory segments with a popularity of TOP-N in the metadata management node can be selected, supporting quantitative sorting and TOP-N fast filtering of hot directories, automatically eliminating invalid low-load data, eliminating the need for manual sorting and filtering, and quickly and accurately locating core access hot directories; on the other hand, when the first directory segment is operated on while the other directory segments are not currently operated on, the count value of the first directory segment increases, while the count value of the other directory segments naturally decreases (for example, automatically reducing the count value according to the above exponential decay formula (1), which will cause the sorting of the hot directory set to change. The new score of the first directory segment is used to re-sort the hot directory set, realizing incremental updates to the hot directory set without recalculating the popularity value of all directory segments, resulting in lower computational overhead.

[0066] For example, in at least one embodiment, a query interface is opened to the client or other ports to output a set of TOP-N directories in response to a popularity query request, thus providing a query interface for the access popularity of the MDS directory.

[0067] For example, in some embodiments, the data management method further includes: storing a set of hotspot directories using a first data structure and a second data structure, wherein the first data structure is a forward ordered mapping structure that sorts directory fragments according to their popularity values, and the second data structure is a reverse hash mapping structure that points from directory fragments to their corresponding popularity values.

[0068] Figure 2 A schematic diagram of two data structures provided in at least one embodiment of this disclosure is shown.

[0069] like Figure 2As shown, the MDBalancer class adds the illustrated data structures. In the forward ordered mapping structure (e.g., `hot_dirs_by_score`), multiple directory segments (dir_A~dir_N) are sorted in descending order by their popularity score (i.e., popularity value). The forward ordered mapping structure uses a multimap structure (multi-key mapping) instead of a map structure (single-key mapping) because different directory segments may have the same score. In the forward ordered mapping structure, the directory segments can be sorted in descending order of popularity score, with the highest-scoring segment at the beginning. In the reverse hash mapping structure (e.g., `hot_dirs_map`), the data structure points from the directory pointer to the current popularity score. This data structure is used to check if a directory is already in the set and its old score during update operations. The reverse hash mapping structure can use an unordered_map structure (unordered multi-hash mapping). The two data structures are always synchronized; any insertion, update, or deletion operation operates on both simultaneously. Based on this approach, a dual-index structure is used to store the entry information of the hot directory set, each addressing a different problem. The forward ordered mapping is used for sorting (entry insertion / update), and the reverse hash mapping is used for fast location. When the two are combined, it is easy to implement insertion / update operations with a time complexity of O(log N) and search operations with a time complexity of O(1). Each metadata operation only performs a small-scale incremental update on the current object, so it does not introduce high-cost operations such as full scan or full sort. Under high-concurrency metadata operations, it does not introduce significant performance overhead.

[0070] For example, in step S120, for the first directory segment, if the current unupdated hotspot directory set already contains a first entry corresponding to the first directory segment, then the popularity value of the first entry is updated to the new popularity value, and the sorting position of the first directory segment is updated based on the new popularity value. If the current unupdated hotspot directory set does not contain an entry corresponding to the first directory segment, then the entry corresponding to the first directory segment is inserted into the hotspot directory set, and if the number of entries contained in the hotspot directory set is greater than N, the entry corresponding to the lowest popularity value is deleted from the hotspot directory set.

[0071] Figure 3 A flowchart illustrating an update hotspot directory set provided by at least one embodiment of this disclosure is shown.

[0072] like Figure 3As shown, each directory operation can update the hotspot directory set (including a forward ordered mapping structure and a reverse hash mapping structure) once. Whenever a directory operation occurs, the comprehensive popularity score of the directory segment is calculated using the existing exponential decay counter (step S301); the entry for the directory segment is searched in the reverse hash mapping structure (step S302); if found, the entry corresponding to the old score is deleted from the forward ordered mapping structure, and the score in the reverse hash mapping structure is updated (step S303); if not found, a new entry corresponding to the directory segment is inserted into the reverse hash mapping structure (step S304); then, a new entry is inserted into the forward ordered mapping structure for the directory segment (step S305), and the popularity score in the new entry is the new score of the directory segment; it is determined whether the number of entries contained in the set is greater than N (step S306); if yes, the entry with the lowest score is deleted from the end of the forward ordered mapping structure, and the entry with the lowest score in the reverse hash mapping structure is deleted simultaneously (step S307); if no, the process is completed and ends. The algorithm used in this embodiment has excellent computational efficiency. Specifically, the target entry retrieval operation is based on a reverse hash mapping structure, with a time complexity of O(1) (constant level). The removal of expired entries and the writing of new entries are based on an ordered forward mapping structure, both with a time complexity of O(log N) (logarithmic level). Automatic elimination of over-limit entries is achieved through a queue tail-elimination mechanism, with an elimination operation time complexity of O(1). Therefore, when the number of directory entries is in the hundreds, the performance overhead of the algorithm is minimal and has almost no impact on the overall operating efficiency of the metadata server, ensuring efficient and stable execution of the hotspot directory statistics and update process.

[0073] For example, in some embodiments, the data management method further includes: in response to a second directory segment being removed from the cache, deleting a second entry corresponding to the second directory segment from a set of hot directories, wherein the second directory segment is any one of the multiple directory segments.

[0074] For example, directory objects (CDir) and inode objects (CInode) in the MDS cache have dynamic lifecycles and are released when the cache is evicted. Holding pointers to released objects in external data structures can lead to dangling pointers and memory safety issues. To address this, in this embodiment, after a directory segment is evicted from the metadata server's cache, the corresponding directory segment is also removed from the TOP-N sort. Specifically, the entries corresponding to the directory segment are simultaneously deleted from both the forward ordered mapping structure and the reverse hash mapping structure, ensuring that no references to released directory objects remain. A cleanup callback mechanism is added to the system when the directory object's lifecycle ends, preventing dangling pointers from forming in the hotspot set after the directory object is evicted from the cache.

[0075] Similar to directory-level hotspots, the inode read / write operation counters maintained by MDS are internally closed, preventing administrators from directly querying which files are currently access hotspots. In real-world production environments, file-level hotspots (such as frequently read / written large files, log files, and temporary files) are also significant sources of performance bottlenecks, and the lack of file-level hotspot detection creates blind spots for operational diagnostics. To address this issue, this disclosure provides an alternative data management method.

[0076] Figure 4 A flowchart of another data management method provided by at least one embodiment of the present disclosure is shown.

[0077] like Figure 4 As shown, in some embodiments, the data management method includes steps S410 to S420 in addition to the steps S110 to S120 described above (not shown).

[0078] Step S410: In response to a file operation on the first file, determine the current popularity value of the first file, which is any one of multiple files in the directory tree.

[0079] Step S420: Based on the current popularity value of the first file, update the hot file set corresponding to the metadata management node. The hot file set includes N file entries corresponding to the N most popular files in the multiple files.

[0080] For example, as mentioned above, the directory tree managed by the metadata management node refers to a subtree within the overall directory tree. The directory tree contains only file metadata, not the actual files themselves. Therefore, files in the directory tree can be understood as files corresponding to the inode data managed by the directory tree, or files corresponding to filenames under any level of folder within the directory tree. These multiple files can be multiple files corresponding to inode data cached in the metadata server's cache. The metadata management node does not store the actual files, but rather the file metadata; therefore, operations on files described here can be understood as operations on the file's metadata.

[0081] For example, in step S410, in response to a file operation, the current popularity value of the file being operated on is determined within the metadata management node. Taking the first file among multiple files as an example, in response to a file operation on the first file, the count value of the first file is updated using a decay counter. The decay counter is used to count the number of times the first file has been operated on, and the count value decays over time. Based on the updated count value, the popularity value of the first file is determined.

[0082] For example, the first file can be any one of the multiple files. The method for obtaining the current popularity value of other files can refer to the first file. Within the current time window, a decay counter is used to record the number of operations performed on the first file. In response to each operation, the decay counter value increases by a predetermined value (e.g., 1). Here, operations refer to reading file metadata (inode read operation) and writing file metadata (inode write operation). This counter has a decay function, decreasing the count value over time. For example, if no new operations are performed within a predetermined duration (e.g., 5 seconds), the count value is halved. This method of increasing the count value when the file's metadata is manipulated and decreasing it over time allows the popularity value to reflect the true frequency of operations on the file's metadata, resulting in a popularity value that more accurately reflects recent real-world popularity.

[0083] For example, in response to a file operation on the first file, the count value of the first file is updated using a decay counter, including: in response to any one of a variety of file operations, the count value of the first file is updated, wherein the variety of file operations includes: a read operation on the metadata of the first file (inode read operation, abbreviated as RD) and a write operation on the metadata of the first file (inode write operation, abbreviated as WR).

[0084] For example, multiple decay counters are set up for different file operations, each used to record the count value of a different file operation. In response to the first file operation being performed on a first file, the count value corresponding to the first file operation is updated, and the current count value of each of the other file operations is obtained. The weighted sum of the current count values ​​of the various file operations is calculated to obtain the popularity value of the first file.

[0085] For example, unlike the five directory operations, file-level operations include two file operations. Two corresponding decay counters can be set for each file, used to count inode read and write operations respectively. For each decay counter, the count value increases by a predetermined value each time a corresponding operation is detected, and decreases over time according to the decay formula (1) mentioned above. File-level popularity can be updated by the hit_inode() method of the load balancer (MDBalancer), which is the entry point for updating the popularity of all inode operations. After updating the file's own counter, the hit_dir() method can be called to update the directory-level counter of its parent directory.

[0086] For example, for the first file, determine the count value for each of the two file operations mentioned above, where the count value for each operation is a count value that decays over time. Calculate the sum of the count values ​​for these two file operations to obtain the popularity value of the first file.

[0087] For example, after determining the current popularity value of the first file in step S410, in step S420, the hot file set corresponding to the metadata management node is updated based on the current popularity value of the first file. Furthermore, if any file among the multiple files is subjected to another operation (such as the two file operations mentioned above), the information of the hot file set is updated again using steps S410-S420. Thus, each file operation updates the hot file set. Each file operation updates the popularity value of one file, while the popularity of the remaining unoperated files naturally decays, causing the overall popularity ranking of the files to change. Inserting the new popularity value of the operated file into the ranking yields a new order, achieving an incremental update of the hot file set. When a popularity query request is received, the current hot file set can be output. The value of N can be set according to actual needs, for example, it can be 100.

[0088] For example, each of the N file entries includes: the file's score, the file's inode number, the file's full path, the read operation decay count (pop_rd), and the write operation decay count (pop_wr).

[0089] According to at least one embodiment of this disclosure, not only can directory fragments with a popularity of TOP-N in the metadata server be obtained, but also files with a popularity of TOP-N can be obtained. This supports quantitative sorting of hot files and rapid TOP-N filtering, automatically eliminating invalid and low-load data, eliminating the need for manual sorting and filtering, and enabling quick and accurate location of core access hot files. Furthermore, by re-sorting the set of hot files using the new score of the first file, incremental updates to hot files can be achieved without recalculating the popularity value for all files, resulting in lower computational overhead.

[0090] For example, in some embodiments, the data management method further includes: storing the set of hot files using a first data structure and a second data structure, wherein the first data structure is a forward ordered mapping structure that sorts files according to their popularity values, and the second data structure is a reverse hash mapping structure that points from a file to its corresponding popularity value. For details regarding the first and second data structures, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0091] For example, in some embodiments, updating the hot file set includes: if there is already a first file entry corresponding to the first file in the currently unupdated hot file set, then update the popularity value of the first file entry to a new popularity value, and update the sorting position of the first file based on the new popularity value; and if there is no entry corresponding to the first file in the currently unupdated hot file set, then insert the entry corresponding to the first file into the hot file set, and delete the entry corresponding to the lowest popularity value from the hot file set when the number of entries contained in the hot file set is greater than N.

[0092] For example, the hot file set (including a forward ordered mapping structure and a reverse hash mapping structure) can be updated once in response to each file operation. Whenever a file operation occurs, the overall popularity score of the file is calculated using an existing exponential decay counter; the entry for the file is searched in the reverse hash mapping structure; if found, the entry corresponding to the old score is deleted from the forward ordered mapping structure, and the score in the reverse hash mapping structure is updated; if not found, a new entry corresponding to the file is inserted into the reverse hash mapping structure; then, a new entry is inserted into the forward ordered mapping structure, with the popularity score in the new entry being the file's new score; it is determined whether the number of entries in the set is greater than N; if so, the entry with the lowest score is deleted from the end of the forward ordered mapping structure, and the lowest-scoring entry in the reverse hash mapping structure is also deleted simultaneously; if the number of entries is not greater than N, the process is complete and ends. For details on this process, please refer to the description of the update process for the hot directory set in the above embodiments.

[0093] For example, in some embodiments, the data management method further includes: updating the popularity value of the directory segment corresponding to the second file in response to an update of the popularity value of the second file in the plurality of files, wherein the second file is any one of the plurality of files; and updating the hotspot directory set corresponding to the metadata management node based on the updated popularity value of the directory segment corresponding to the second file.

[0094] For example, when any one of these files is operated on, in addition to updating the popularity value of that file itself, the popularity value of the directory segment corresponding to that file is also updated, and the set of hot directories is further updated. The directory segment corresponding to the file can be understood as the parent directory where the file is located.

[0095] For example, in some embodiments, the data management method further includes: in response to the deletion of file metadata (inode data) corresponding to a third file among the plurality of files from the cache, deleting the third file entry corresponding to the third file from the hot file set, wherein the third file is any one of the plurality of files.

[0096] For example, inode objects (CInodes) in the MDS cache have a dynamic lifecycle and are released when the cache is evicted. Holding pointers to released objects in external data structures can lead to dangling pointers and memory safety issues. To address this, in this embodiment of the disclosure, after the metadata server cache evicts an inode object for a file, the file is also removed from the TOP-N sort. Specifically, the entry corresponding to the file is simultaneously deleted from both the forward ordered mapping structure and the reverse hash mapping structure, ensuring that no references to released inode objects remain. A cleanup callback mechanism is added to the system when the inode object's lifecycle ends, preventing dangling pointers from forming in the hotspot set after the inode object is evicted from the cache.

[0097] In a multi-active MDS deployment environment, different subtrees are distributed across different MDS Ranks. Observation methods are limited to a single MDS node, requiring administrators to connect to each MDS node individually for querying. This makes it impossible to obtain a unified hotspot ranking across the entire cluster. Furthermore, the same directory segment may temporarily exist on multiple MDSs during subtree migration, lacking a merging mechanism. This fragmented observation approach poses a significant obstacle to hotspot diagnosis in multi-MDS environments. To address these issues, at least one embodiment of this disclosure provides an alternative data management method for managing a system with multiple metadata management nodes.

[0098] Figure 5 A flowchart of another data management method provided by at least one embodiment of the present disclosure is shown.

[0099] like Figure 5 As shown, the data management method includes steps S510 to S530.

[0100] Step S510: In response to the cross-node aggregation request, identify the multiple target metadata management nodes to be aggregated.

[0101] Step S520: Send a query command to each of the multiple target metadata management nodes, and receive multiple hot directory sets from the multiple target metadata management nodes respectively. The hot directory set includes N directory entries corresponding to the N most popular directory segments in the directory tree managed by this node.

[0102] Step S530: Generate directory aggregation results based on multiple hotspot directory sets.

[0103] For example, in step S510, the cross-node aggregation request contains the file system identifier and the RANK identifier. By parsing these two identifiers, it can be determined which file system corresponding to the MDS RANK needs to be queried and which MDS RANK corresponding to the file system needs to be queried. In turn, it can be determined which RANKs need to be queried. These RANKs serve as the multiple target metadata management nodes to be aggregated mentioned above.

[0104] For example, in step S520, a query command is sent to each of the multiple target RANKs, and multiple hotspot directory sets are received from each of the multiple target RANKs. The process for determining the hotspot directory set for each RANK can be found in the data management method for the metadata management node described above, and will not be repeated here.

[0105] For example, in step S530, a directory aggregation result is generated based on multiple hot directory sets, including: merging the entries in the multiple hot directory sets to obtain a merge result; and determining a global hot directory set based on the merge result, wherein the global hot directory set includes the N directory entries corresponding to the N most popular directory segments in the merge result.

[0106] For example, if multiple target RANKs are RANK0 to RANK3, the four hot directory sets fed back by RANK0 to RANK3 are merged to obtain a comprehensive ranking (e.g., including 4*N entries). The N directory segments with the highest popularity are determined from the comprehensive ranking (as the directory aggregation result mentioned above) to obtain the global TOP-N. This global TOP-N can be output to the client. The output information includes the N directory entries and the time when the aggregation result was generated.

[0107] For example, merging entries from multiple hot directory sets yields a merged result, including: for a third directory fragment appearing in one node, determining the entry corresponding to the third directory fragment in the merged result based on the popularity value of the third directory fragment in its own node; for a fourth directory fragment appearing in at least two nodes, determining the entry corresponding to the fourth directory fragment in the merged result based on the highest popularity value of the fourth directory fragment in at least two nodes.

[0108] For example, when a directory fragment appears only in the set of hot directories in a single Rank, its popularity score in the merged result is the same as its popularity score in that Rank. During the subtree migration transition, the same directory fragment may appear in multiple Ranks. In this case, the highest score of the directory fragment across those multiple Ranks can be used as its popularity score in the merged result.

[0109] For example, if a target RANK does not return a hotspot directory set, an error is reported for that target RANK, and the hotspot directory sets of other target RANKs that returned data normally are merged.

[0110] For example, in some embodiments, the data management method further includes: receiving multiple sets of hot files from multiple target metadata management nodes. The sets of hot files include N file entries corresponding to the N most popular files in the directory tree managed by the current node; and generating a file aggregation result based on the multiple sets of hot files.

[0111] According to at least one embodiment of this disclosure, an aggregation module (the aforementioned management system) is introduced into the Ceph Management Service (MGR) layer. When a user queries, this module sends instructions to each active MDS Rank to collect hotspot data (directories or files) on demand. Then, the results returned by multiple nodes are merged according to path (if the same path, the highest score is taken), sorted, and the aggregated result for the entire cluster is returned. This MGR aggregation module provides a unified hotspot ranking across the entire cluster (directories and files are aggregated separately), eliminating the need for administrators to query node by node and manually summarize. With a configuration of 3 active MDSs, a single query takes approximately 3 round trips of the admin socket (the local management socket built into the Ceph component) (typically <100ms), significantly faster than manual node-by-node operations. Furthermore, by using on-demand querying of MDSs instead of continuous push, additional continuous communication overhead between MDSs is avoided.

[0112] For example, in the data management method provided in at least one embodiment of this disclosure, directory access popularity detection and file access popularity detection of the metadata server in the distributed file system CephFS can be realized, and Top-N hot directories and Top-N hot files can be tracked.

[0113] For example, in the data management method provided in at least one embodiment of this disclosure, incremental Top-N hotspot tracking is performed based on an exponentially decaying counter. In the metadata load balancer (MDBalancer) of each MDS node, two capacity-constrained Top-N ordered sets are maintained: one for hotspot directory tracking and one for hotspot file tracking. For hotspot directories: whenever a directory operation occurs (hit_dir is called), the hotspot score of the directory is calculated using the existing exponentially decaying counter, and the directory Top-N set is incrementally updated. For hotspot files: whenever a non-directory type inode operation occurs (hit_inode is called), the file hotspot score is calculated using the inode-level read / write operation counter, and the file Top-N set is incrementally updated.

[0114] For example, in the data management method provided in at least one embodiment of this disclosure, when a directory object or inode object is evicted from the cache, it is synchronously removed from the corresponding Top-N set of response entries through a lifecycle callback mechanism to ensure memory safety.

[0115] For example, in at least one embodiment of the data management method provided in this disclosure, hot data can be aggregated and uniformly displayed across multiple metadata service nodes.

[0116] For example, in at least one embodiment of the data management method provided in this disclosure, the data management method is jointly implemented by three subsystems: an MDS-side Top-N Hotspot Tracking Subsystem, an MDS-side Admin Socket Reporting Subsystem, and an MGR-side Cross-Node Aggregation Subsystem. The MDS-side Top-N Hotspot Tracking Subsystem incrementally maintains the hotspot directory set and hotspot file set in the MDBalancer of each MDS, for example, by executing steps S110~S130 and S410~S430 as described above. The MDS-side Admin Socket Reporting Subsystem exports the hotspot directory set and hotspot file set in JSON (JavaScript Object Notation) format through management socket commands. The MGR-side Cross-Node Aggregation Subsystem aggregates data from multiple MDSs and provides a unified query interface, supporting both directory and file hotspot queries.

[0117] For example, in the MDS-side Admin Socket reporting subsystem, two new commands can be registered in the management socket of the MDS daemon: the command to get the directory hotspot set and the command to get the file hotspot set. The responses to these two commands will get the directory hotspot set and the file hotspot set, respectively.

[0118] For example, a performance counter can be added to MDS. The count value of this performance counter can be used as an observable metric to record how many directory fragments are in the Top-N set of hot directories in the current MDS Rank.

[0119] For example, in the cross-node aggregation subsystem on the MGR side, cross-node aggregation can be implemented using Python modules at the MGR (manager) layer. Commands can be sent to each active MDS Rank on demand to collect hot data (directories or files), instead of adding new message types between MDSs. Compared to the MDS-to-MDS message push scheme, the MGR on-demand query scheme adopted in this embodiment requires no additional modifications to the C++ code, has no continuous communication overhead (communication only occurs during queries), and hot reloading of Python modules reduces deployment complexity. Query-time snapshots improve data freshness, and single-rank failures automatically degrade the system. Hot directory data is diagnostic / observational in nature and does not require continuous push; therefore, the on-demand query model is more suitable.

[0120] For example, in the MGR-side cross-node aggregation subsystem, the following two commands can be registered: a hotspot directory query command and a hotspot file query command. Both share the same data collection and aggregation framework within the MGR module, differing only in the commands and aggregation functions on the MDS side. In response to these two commands, hotspot directory query requests and hotspot file query requests can be sent to the target MDS, and the corresponding aggregation operations can be performed.

[0121] For example, directory hotspot data and file hotspot data can also be used for two decisions within the MDS: directory fragmentation and subtree migration. Directory fragmentation refers to triggering directory sharding when the access frequency of a directory exceeds a threshold, splitting an excessively large directory into multiple fragments to improve concurrency. Subtree migration refers to migrating high-load subtrees from busy MDSs to idle MDSs based on the aggregate load of each MDS, achieving cross-node load balancing.

[0122] For example, the data management method provided in at least one embodiment of this disclosure offers high real-time performance and zero additional overhead. Specifically, the hotspot tracking logic is directly embedded in the existing hit_dir() and hit_inode() call paths, and is incrementally updated with each metadata operation. The time complexity of updating the Top-N set is O(log N), requiring approximately 7 comparison operations under the default configuration of N=100. The additional latency for a single metadata operation (typically in the millisecond range) is negligible. No additional background scanning thread or timer is required.

[0123] For example, the data management method provided in at least one embodiment of this disclosure has a very small memory footprint. Specifically, each MDS node maintains a directory set of at most N (e.g., 100) entries and a file set of N entries. Taking N=100 as an example, each entry consumes approximately 24 bytes (8-byte double + 8-byte pointer + 8-byte hash table overhead), with a total memory footprint of approximately 4.8KB per MDS node (2.4KB for directories + 2.4KB for files). Compared to the millions of cached entries typically managed by an MDS, the memory overhead is negligible.

[0124] For example, in the data management method provided in at least one embodiment of this disclosure, the popularity value decays naturally without manual cleanup. Specifically, relying on the characteristics of an exponential decay counter, the score of a directory or file that is no longer accessed will automatically decrease over time, eventually being eliminated from the Top-N set by new popular entries. Administrators do not need to manually reset or clean up historical data. Taking a default half-life of 5 seconds as an example, after a directory or file stops being accessed, its score will decay to 1 / 64 of its original value in about 30 seconds (6 half-lives), which is sufficient to eliminate it from the Top-N in most scenarios.

[0125] For example, the data management method provided in at least one embodiment of this disclosure ensures lifecycle safety and eliminates memory risks. Specifically, by injecting cleanup callbacks into the directory cache eviction path and inode deletion path, it is guaranteed that no dangling pointers will appear in the Top-N sets of directories and files. This mechanism seamlessly integrates with the existing cache management framework of MDS, without requiring the introduction of additional reference counting or garbage collection mechanisms.

[0126] For example, in the data management method provided in at least one embodiment of this disclosure, both directory and file dimensions are covered. Specifically, both directory-level and file-level hotspot detection are supported simultaneously, providing comprehensive hotspot visibility. Directory-level hotspots help locate namespace regions with concentrated load, while file-level hotspots further pinpoint specific high-frequency access files (such as frequently written log files, frequently read data files, etc.). The two dimensions complement each other, enabling administrators to progressively troubleshoot performance bottlenecks from coarse-grained to fine-grained.

[0127] For example, in the data management method provided in at least one embodiment of this disclosure, multi-MDS aggregation can be implemented, and a unified view can be used to display global results. Specifically, the MGR aggregation module provides a unified hotspot ranking across the entire cluster (aggregating directories and files separately), eliminating the need for administrators to query and manually summarize node by node.

[0128] For example, the data management method provided in at least one embodiment of this disclosure can gracefully tolerate faults and maintain availability even during degradation. Specifically, when some MDS Ranks are unreachable (e.g., during a failure or restart), the aggregation module automatically skips unavailable nodes, uses data from available nodes to return partial results, and clearly indicates in the output which Ranks are missing data. This ensures the availability of the tool under certain failure scenarios.

[0129] For example, at least one embodiment of the data management method provided in this disclosure achieves zero continuous communication overhead. Specifically, it adopts an on-demand query mode, where communication between the MGR and MDS only occurs when the administrator executes a query command. This eliminates the introduction of any periodic message pushes and does not increase network load or message processing overhead between MDSs or between MDS and MGR.

[0130] Figure 6 A schematic block diagram of a data management device 600 provided in at least one embodiment of the present disclosure is shown. The data management device 600 is used for a metadata management node.

[0131] For example, such as Figure 6 As shown, the data management device 600 includes a heat determination module 610 and a collection update module 620.

[0132] The popularity determination module 610 is configured in the metadata management node to determine the current popularity value of the first directory segment in response to a directory operation on the first directory segment, wherein the first directory segment is any one of a plurality of directory segments in the directory tree managed by the metadata management node. The popularity determination module 610 may, for example, execute... Figure 1 Step S110 is described.

[0133] The set update module 620 is configured to update the hot directory set corresponding to the metadata management node based on the current popularity value of the first directory fragment. The hot directory set includes N directory entries corresponding to the top N most popular directory fragments among the plurality of directory fragments. The sorting module 620 can, for example, execute... Figure 1 Step S120 is described.

[0134] Figure 7 A schematic block diagram of another data management device 700 provided in at least one embodiment of the present disclosure is shown. This data management device 700 is used in a management system for managing multiple metadata management nodes.

[0135] For example, such as Figure 7 As shown, the data management device 700 includes a node determination module 710, a sending and receiving module 720, and an aggregation module 730.

[0136] The node determination module 710 is configured to determine multiple target metadata management nodes to be aggregated in response to a cross-node aggregation request. For example, the node determination module 710 can perform... Figure 4 Step S410 is described.

[0137] The sending and receiving module 720 is configured to send a query command to each of the plurality of target metadata management nodes and receive multiple sets of hot directories respectively from the plurality of nodes. The hot directory set includes N directory entries corresponding to the top N most popular directory segments in the directory tree managed by the node. The sending and receiving module 720 can, for example, execute... Figure 4 Step S420 is described.

[0138] The aggregation module 730 is configured to generate an aggregation result based on the multiple sets of hotspot directories. For example, the aggregation module 730 can execute... Figure 4 Step S430 is described.

[0139] For example, the above modules can be hardware, software, firmware, or any feasible combination thereof. For example, the above modules can be dedicated or general-purpose circuits, chips, or devices, or a combination of a processor and memory. The embodiments of this disclosure do not limit the specific implementation of the above units.

[0140] For example, each of the above modules may include code and programs stored in memory; the processor may execute the code and programs to implement some or all of the functions of the above modules as described above. For example, each of the above modules may be a dedicated hardware device used to implement some or all of the functions of the above modules as described above. For example, each of the above modules may be a circuit board or a combination of multiple circuit boards used to implement the functions described above. In embodiments of this disclosure, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-transitory memories connected to the processor; and (3) processor-executable firmware stored in memory.

[0141] It should be noted that in the embodiments of this disclosure, the data management device 600 corresponds to each step of the aforementioned data management method for metadata management nodes. For the specific functions of the data management device 600, please refer to the relevant description of the data management method for metadata management nodes, which will not be repeated here.

[0142] The data management device 700 corresponds to each step of the data management method of the aforementioned management system for managing multiple metadata management nodes. For details on the specific functions of the data management device 700, please refer to the relevant description of the data management method for the management system, which will not be repeated here.

[0143] Figure 6 The data management device 600 shown and Figure 7 The components and structure of the data management device 700 shown are exemplary and not limiting. The data management device 600 and data management device 700 may include other components and structures as needed. The data management device 600 and data management device 700 may include more or fewer circuits or units, and the connection relationships between the various circuits or units are not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0144] At least one embodiment of this disclosure also provides an electronic device including a processor and a memory, the memory storing one or more computer program modules. The one or more computer program modules are configured to be executed by the processor to implement the data management method of any of the above embodiments.

[0145] Figure 8A This is a schematic block diagram of an electronic device provided for some embodiments of this disclosure. For example... Figure 8A As shown, the electronic device 800 includes a processor 810 and a memory 820. The memory 820 stores non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 810 is used to execute the non-transitory computer-readable instructions, which, when executed by the processor 810, perform one or more steps of the data management method described above. The memory 820 and the processor 810 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown). For specific implementations and explanations of the various steps of this data management method, please refer to the embodiments of the data management method described above; repetitions will not be repeated here.

[0146] It should be noted that Figure 8A The components of the electronic device 800 shown are merely exemplary and not limiting. The electronic device 800 may have other components as needed for the actual application.

[0147] For example, the processor 810 and the memory 820 can communicate with each other directly or indirectly.

[0148] For example, processor 810 and memory 820 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. Processor 810 and memory 820 can also communicate with each other via a system bus, and this disclosure is not limiting in this regard.

[0149] For example, the processor 810 and memory 820 can be located on the server side (or in the cloud).

[0150] For example, processor 810 can control other components in electronic device 800 to perform desired functions. For example, processor 810 can be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) can be an x86 or ARM architecture. Processor 810 can be a general-purpose processor or a special-purpose processor, and can control other components in electronic device 800 to perform desired functions.

[0151] For example, memory 820 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 810 may run one or more computer program modules to implement various functions of electronic device 800. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0152] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 800 can be referred to the description of the data management method above, and will not be repeated here.

[0153] Figure 8B This is a schematic block diagram of another electronic device provided in some embodiments of this disclosure. The electronic device 900 is, for example, suitable for implementing the data management method provided in the embodiments of this disclosure. The electronic device 900 may be a terminal device, etc. It should be noted that... Figure 8B The illustrated electronic device 900 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0154] like Figure 8BAs shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 910, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 920 or a program loaded from a storage device 980 into a random access memory (RAM) 930. The RAM 930 also stores various programs and data required for the operation of the electronic device 900. The processing device 910, the ROM 920, and the RAM 930 are interconnected via a bus 940. An input / output (I / O) interface 950 is also connected to the bus 940.

[0155] Typically, the following devices can be connected to I / O interface 950: input devices 960 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 970 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 980 including, for example, magnetic tapes, hard disks, etc.; and communication devices 990. Communication device 990 allows electronic device 900 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 8B An electronic device 900 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 900 may alternatively implement or have more or fewer devices.

[0156] For example, according to embodiments of this disclosure, the data management method described above can be implemented as a computer software program. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the data management method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 990, or installed from a storage device 980, or installed from a ROM 920. When the computer program is executed by a processing device 910, the functions defined in the data management method provided by embodiments of this disclosure can be implemented.

[0157] At least one embodiment of this disclosure also provides a computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, can implement the data management method described above.

[0158] Figure 9 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. For example... Figure 9 As shown, the storage medium 1000 stores non-transitory computer-readable instructions 1010. For example, when the non-transitory computer-readable instructions 1010 are executed by a computer, one or more steps in the data management method described above are performed.

[0159] For example, the storage medium 1000 can be used in the aforementioned electronic device 800. For example, the storage medium 1000 can be... Figure 8A The memory 820 in the illustrated electronic device 800. For example, a description of the storage medium 1000 can be found here. Figure 8A The corresponding description of the memory 820 in the illustrated electronic device 800 will not be repeated here.

[0160] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0161] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0162] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0163] The following points should be noted regarding this disclosure:

[0164] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0165] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0166] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A data management method for a metadata management node, the method comprising: In the metadata management node, in response to a directory operation on the first directory segment, the current popularity value of the first directory segment is determined, wherein the first directory segment is any one of a plurality of directory segments in the directory tree managed by the metadata management node; Based on the current popularity value of the first directory fragment, the hotspot directory set corresponding to the metadata management node is updated. The hotspot directory set includes N directory entries corresponding to the N most popular directory fragments among the plurality of directory fragments. Where N is a positive integer.

2. The method according to claim 1, further comprising: In response to a file operation on a first file, the current popularity value of the first file is determined, wherein the first file is any one of a plurality of files in the directory tree; Based on the current popularity value of the first file, the hot file set corresponding to the metadata management node is updated, wherein the hot file set includes N file entries corresponding to the N most popular files among the multiple files.

3. The method according to claim 1, wherein, In response to a directory operation on a first directory segment, the current popularity value of the first directory segment is determined, including: In response to a directory operation on the first directory segment, a decay counter is used to update the count value of the first directory segment, wherein the decay counter is used to count the number of times the first directory segment has been operated on, and the count value decays over time; and Based on the updated count value, the current popularity value of the first directory segment is determined.

4. The method according to claim 3, wherein, In response to a directory operation on the first directory segment, the count value of the first directory segment is updated using a decay counter, including: In response to any one of a plurality of directory operations, the count value of the first directory segment is updated, wherein the plurality of directory operations include at least one of the following: a read operation on the file metadata managed by the first directory segment, a write operation on the file metadata managed by the first directory segment, a read operation on the first directory segment, loading the metadata of the first directory segment from the disk, and writing the metadata of the first directory segment to the disk.

5. The method according to claim 4, wherein, Multiple decay counters are set for each of the various directory operations, and the multiple decay counters are used to record the count values ​​of the various directory operations respectively; The process of updating the count value of the first directory segment in response to any one of a variety of directory operations includes: In response to the first directory operation among the multiple directory operations, update the count value of the first directory operation, and obtain the current count value of each of the other directory operations besides the first directory operation. The process of determining the popularity value of the first directory segment based on the updated count value includes: The weighted sum of the current count values ​​of the various directory operations is calculated to obtain the popularity value of the first directory segment.

6. The method according to claim 1, further comprising: The hotspot directory set is stored using a first data structure and a second data structure, wherein the first data structure is a forward ordered mapping structure that sorts directory segments according to their popularity value, and the second data structure is a reverse hash mapping structure that points from directory segments to their corresponding popularity values.

7. The method according to claim 1, in, Based on the current popularity value of the first directory fragment, update the hotspot directory set corresponding to the metadata management node, including: If the current set of unupdated hot directories already contains a first entry corresponding to the first directory segment, then the popularity value of the first entry is updated to the current popularity value, and the sorting position of the first directory segment is updated based on the current popularity value; and If the entry corresponding to the first directory fragment does not exist in the currently unupdated hotspot directory set, then the entry corresponding to the first directory fragment is inserted into the hotspot directory set, and the entry corresponding to the lowest popularity value is deleted from the hotspot directory set when the number of entries contained in the hotspot directory set is greater than N.

8. The method according to claim 1, further comprising: In response to the removal of a second directory segment from the plurality of directory segments from the cache, a second entry corresponding to the second directory segment is removed from the set of hot directories, wherein the second directory segment is any one of the plurality of directory segments.

9. The method according to claim 1, wherein, Each of the N directory entries includes one or more of the following: the popularity value of the directory segment, the full path of the directory segment, the identifier of the directory segment, the operational load component directly borne by the directory segment, the nested load of the directory segment, and the subtree load.

10. The method according to claim 2, further comprising: In response to an update of the popularity value of the second file in the plurality of files, the popularity value of the directory segment corresponding to the second file is updated, wherein the second file is any one of the plurality of files; Based on the updated popularity value of the directory segment corresponding to the second file, the hotspot directory set corresponding to the metadata management node is updated.

11. The method according to claim 2, wherein, In response to a file operation on the first file, determining the current popularity value of the first file includes: In response to a file operation on a first file, a decay counter is used to update the count value of the first file, wherein the decay counter is used to count the number of times the first file has been operated on, and the count value decays over time; and Based on the updated count value, the popularity value of the first file is determined.

12. The method according to claim 11, wherein, In response to a file operation on the first file, the count value of the first file is updated using a decay counter, including: In response to any of a variety of file operations, the count value of the first file is updated, wherein the variety of file operations includes: a read operation on the metadata of the first file and a write operation on the metadata of the first file; Among them, multiple decay counters are set for each of the various file operations, and the multiple decay counters are used to record the count values ​​of the various file operations respectively; Among them, updating the count value of the first file in response to any one of the multiple file operations includes: In response to the first file operation among the multiple file operations, update the count value of the first file operation, and obtain the current count value of each of the other file operations besides the first file operation. The process of determining the popularity value of the first file based on the updated count value includes: The popularity value of the first file is obtained by calculating the weighted sum of the current count values ​​of the various file operations.

13. The method according to claim 2, wherein, Each of the N file entries includes one or more of the following: file popularity value, file number, file full path, read operation decay count value, and write operation decay count value.

14. A data management method for managing a management system with multiple metadata management nodes, the method comprising: In response to a cross-node aggregation request, identify the multiple target metadata management nodes to be aggregated; Send a query command to each of the multiple target metadata management nodes, and receive multiple hot directory sets fed back by the multiple nodes respectively, wherein the hot directory set includes N directory entries corresponding to the top N most popular directory segments in the directory tree managed by this node. Based on the multiple hotspot directory sets, an aggregated result is generated.

15. The method according to claim 14, wherein, Based on the aforementioned collections of multiple hotspot directories, an aggregation result is generated, including: The entries in the multiple hotspot directory sets are merged to obtain the merged result; Based on the merging results, a global hotspot directory set is determined, wherein the global hotspot directory set includes N directory entries corresponding to the N most popular directory segments in the merging results.

16. The method according to claim 15, wherein, The entries in the multiple hotspot directory sets are merged to obtain the merged result, including: For a third directory fragment appearing in a node, the entry corresponding to the third directory fragment in the merge result is determined based on the popularity value of the third directory fragment in its own node; For a fourth directory fragment that appears in at least two nodes, the entry corresponding to the fourth directory fragment in the merge result is determined based on the highest popularity value of the fourth directory fragment in the at least two nodes.

17. A data management apparatus for a metadata management node, the apparatus comprising: The popularity determination module is configured to determine the current popularity value of the first directory segment in response to a directory operation on the first directory segment in the metadata management node, wherein the first directory segment is any one of a plurality of directory segments in the directory tree managed by the metadata management node; The set update module is configured to update the hot directory set corresponding to the metadata management node based on the current popularity value of the first directory fragment. The hot directory set includes N directory entries corresponding to the N most popular directory fragments among the plurality of directory fragments. Where N is a positive integer.

18. An information processing apparatus for managing a management system with multiple metadata management nodes, the apparatus comprising: The node determination module is configured to determine multiple target metadata management nodes to be aggregated in response to cross-node aggregation requests. The sending and receiving module is configured to send a query instruction to each of the multiple target metadata management nodes and receive multiple hot directory sets fed back by the multiple nodes respectively, wherein the hot directory set includes N directory entries corresponding to the top N most popular directory segments in the directory tree managed by this node. The aggregation module is configured to generate aggregation results based on the multiple hotspot directory sets.

19. An electronic device comprising: processor; Memory, which stores one or more computer program modules; The one or more computer program modules are configured to be executed by the processor to implement the data management method according to any one of claims 1-16.

20. A computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, can implement the data management method according to any one of claims 1-16.

Images