Directory management method, directory management device, electronic device and storage medium

Abstract

The present application discloses a directory management method, a directory management device, an electronic device and a storage medium, and relates to the field of directory management technology, including: in response to a distribution node receiving a directory update task, building a directory tree based on the directory information stored in the database, and dividing the directory tree to obtain multiple independent directory trees; identifying an idle microservice node as a first target node; generating a corresponding directory statistics task and assigning it to the first target node; in response to the first target node receiving the directory statistics task, determining whether the corresponding task directory tree needs to be split, and if so, extracting a sub-directory tree from the task directory tree and generating a corresponding statistics subtask, selecting a second target node to receive and execute the statistics subtask. The present application can divide the directory tree based on path independence, and dynamically assign directory statistics tasks in combination with the status of cluster microservice nodes, thereby realizing distributed concurrent statistics of the directory, improving statistical efficiency, reducing resource consumption and reducing latency.

Description

Technical Field

[0001] The present application relates to the technical field of directory management, and in particular to a directory management method, a directory management device, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of the artificial intelligence industry, the demand for high-performance computing capabilities from research institutions and enterprises continues to rise. AI cluster platforms have emerged as a crucial infrastructure for supporting model training and inference tasks. A core function of AI clusters is to support the efficient storage and management of massive amounts of data. Platform users often need to process petabytes or even larger amounts of file data, including user home directories, public directories, training dataset directories, and model output directories. In this high-frequency access and highly concurrent computing environment, the platform places higher demands on the storage system's directory information update efficiency.

[0003] Currently, mainstream AI platforms typically perform a full traversal of all directories and files in the storage system to collect and update directory information. While this approach is logically simple, it has significant drawbacks. First, the full directory traversal frequently triggers underlying storage I / O operations, significantly consuming CPU and memory resources on business nodes and impacting computational efficiency. Second, when dealing with file architectures exceeding petabytes, the overall latency of traversal statistics is high, making it difficult to meet the real-time requirements of scenarios such as model training. Summary of the Invention

[0004] The present application provides a directory management method that can divide the directory tree based on path independence, dynamically allocate directory statistics tasks in combination with the status of cluster microservice nodes, and realize distributed concurrent statistics of the directory, so as to at least solve the problems of low statistical efficiency, high resource consumption and high latency in related technologies.

[0005] This application provides a directory management method, including:

[0006] In response to the distribution node receiving the directory update task, a directory tree is constructed based on the directory information stored in the database, and the directory tree is divided according to path independence to obtain multiple independent directory trees;

[0007] Obtain the current running status of the cluster microservice node and identify the idle microservice node as the first target node;

[0008] Based on the independent directory tree, generate the corresponding directory statistics task and assign it to the first target node;

[0009] In response to the first target node receiving the directory statistics task, it determines whether the corresponding task directory tree needs to be split. If so, it extracts the sub-directory tree from the task directory tree and generates the corresponding statistics sub-task. It selects the second target node from the cluster microservice node to receive and execute the statistics sub-task. If not, it executes the directory statistics task itself.

[0010] In response to the allocation node receiving the task completion information fed back by the first target node, the allocation node parses the task completion information, obtains the directory update information, and writes the updated information into the database to update the directory information.

[0011] This application also provides a directory management device, including:

[0012] A directory tree construction and division module is configured to, in response to the distribution node receiving the directory update task, construct a directory tree based on the directory information stored in the database, and divide the directory tree according to path independence to obtain multiple independent directory trees;

[0013] An idle node identification module is used to obtain the current running status of the cluster microservice node and identify the idle microservice node as the first target node;

[0014] A task allocation module, configured to generate corresponding directory statistics tasks based on the independent directory tree and allocate the tasks to the first target node;

[0015] The task splitting and statistics module is used to determine whether the corresponding task directory tree needs to be split in response to the first target node receiving the directory statistics task. If so, it extracts the sub-directory tree from the task directory tree and generates the corresponding statistics sub-task, selects the second target node from the cluster microservice node to receive and execute the statistics sub-task, and if not, automatically executes the directory statistics task;

[0016] The directory update module is used to parse the task completion information fed back by the first target node in response to the allocation node receiving the task completion information, obtain the directory update information and write it into the database to update the directory information.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned directory management methods when executing the computer program.

[0018] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned directory management methods are implemented.

[0019] The present invention divides the directory tree by path independence, reduces task dependence and repeated traversal, can realize parallel processing of multiple independent directories, and improve the concurrency of statistical tasks; at the same time, the present invention uses the microservice node status to dynamically identify idle nodes and perform task allocation, which can make full use of computing resources and avoid further congestion of high-load nodes, thereby improving the intelligence and computing efficiency of overall resource scheduling; furthermore, the present invention makes the task granularity dynamically scalable by judging whether the directory tree can be split into sub-directory tree tasks, avoids blocking caused by a single node processing an oversized task, and improves the controllability and stability of system processing; in addition, the present invention avoids frequent database write operations and reduces the write pressure of the database by feeding back the statistical results from each task node layer by layer and centrally updating them to the database, while improving the centralization and consistency of data updates. Therefore, the present invention ensures the concurrency and flexibility of task execution through directory tree division and distributed concurrent statistics, while effectively controlling the data scale and IO pressure of the database, thereby improving the speed of directory information update and system response efficiency in a large-scale AI cluster storage environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A schematic diagram of an application environment provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a directory management method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a directory traversal process provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a task allocation and execution process provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of node state switching and fault handling provided in an embodiment of the present application;

[0026] Figure 6 An example diagram of an independent directory tree provided in an embodiment of the present application;

[0027] Figure 7 An example diagram of a subdirectory tree provided in an embodiment of the present application;

[0028] Figure 8A schematic diagram of a directory update information storage and aggregation process provided in an embodiment of the present application;

[0029] Figure 9 A structural block diagram of a directory management device provided in an embodiment of the present application;

[0030] Figure 10 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0033] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0034] This application provides a directory management method that can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The user performs a change operation on the data stored in the database through the terminal 102, and the terminal 102 sends the database directory update task to the server 104. The server 104 uses several microservice nodes of the AI ​​cluster set therein to read and traverse the directory information in the database in a distributed manner, generate directory update information to update the directory information in the database. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets and portable wearable devices, and the server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0035] like Figure 2 As shown, an embodiment of the present application provides a directory management method, including:

[0036] Step 201: In response to the distribution node receiving the directory update task, a directory tree is constructed based on the directory information stored in the database, and the directory tree is divided according to path independence to obtain multiple independent directory trees;

[0037] Step 202: Obtain the current running status of the cluster microservice node and identify the idle microservice node as the first target node;

[0038] Step 203: Generate a corresponding directory statistics task based on the independent directory tree and assign it to the first target node;

[0039] Step 204: In response to the first target node receiving the directory statistics task, it is determined whether the corresponding task directory tree needs to be split. If so, a sub-directory tree is extracted from the task directory tree and a corresponding statistics sub-task is generated. A second target node is selected from the cluster microservice node to receive and execute the statistics sub-task. If not, the directory statistics task is automatically executed.

[0040] Step 205 : In response to the allocation node receiving the task completion information fed back by the first target node, the allocation node parses the task completion information, obtains directory update information, and writes the updated information into the database to update the directory information.

[0041] Specifically, the present invention divides the directory tree by path independence, reduces task dependency and repeated traversal, can realize parallel processing of multiple independent directories, and improve the concurrency of statistical tasks; at the same time, the present invention uses the microservice node status to dynamically identify idle nodes and perform task allocation, which can make full use of computing resources and avoid further congestion of high-load nodes, thereby improving the intelligence and computing efficiency of overall resource scheduling; furthermore, the present invention determines whether the directory tree can be split into sub-directory tree tasks, so that the task granularity has dynamic scalability, avoids blocking caused by a single node processing an oversized task, and improves the controllability and stability of system processing; in addition, the present invention avoids frequent database write operations and reduces the write pressure of the database by feeding back the statistical results from each task node layer by layer and centrally updating them to the database, while improving the centralization and consistency of data updates. Therefore, the present invention ensures the concurrency and flexibility of task execution through directory tree division and distributed concurrent statistics, while effectively controlling the data scale and IO pressure of the database, thereby improving the speed of directory information update and system response efficiency in a large-scale AI cluster storage environment.

[0042] It's worth noting that in traditional non-distributed directory statistics solutions, as the amount of stored data continues to increase, the time required for directory traversal and statistics increases exponentially, leading to slower system response and significant performance bottlenecks. A distributed partitioning solution can divide a large directory tree into multiple independent sub-trees, each processed in parallel by different nodes. This effectively avoids the performance bottleneck of single-point processing and significantly improves the efficiency of directory statistics and the overall throughput of the system.

[0043] It should be further explained that the purpose of dividing the directory tree in this solution is not only to split the task into smaller units, but more importantly to reduce the amount of data stored in the database through division, so as to achieve refined management of the data in the database. By quickly counting and updating smaller data blocks, the read and write efficiency of the database can be improved, the storage pressure can be reduced, and the data access speed can be accelerated, ensuring that the system can still achieve efficient and stable directory statistics services in a large-scale storage environment.

[0044] In one embodiment, Figure 6 As shown, the directory tree is divided according to path independence to obtain multiple independent directory trees, including:

[0045] Identify multiple independent directory sets in the directory tree whose root paths do not contain each other, and each of them constitutes multiple independent directory trees.

[0046] Specifically, in this embodiment, by dividing the directory tree according to path independence, the task can be effectively split into multiple independent sub-tasks that do not interfere with each other, avoiding cross-dependencies between directory trees and improving the efficiency and accuracy of parallel task processing; at the same time, it provides clear task boundaries for subsequent directory statistics and scheduling, which is beneficial to the load balancing and task allocation of microservice nodes.

[0047] In one embodiment, the directory information stored in the database is stored through a preset directory storage data structure, which includes at least one of the following: ID number, storage directory or index, last modification time, directory size, last statistical directory size, directory owner, directory hierarchy, parent directory of the storage directory, and table record update time.

[0048] Specifically, this embodiment adopts a structured directory storage data structure with rich attributes to facilitate accurate management and rapid query of directory information, especially fields such as modification time and directory hierarchy, which is conducive to efficient detection of directory changes, updates and maintenance of directory tree status, reducing repeated scanning, and improving system response speed and storage consistency.

[0049] In one embodiment, obtaining the current running status of a cluster microservice node and identifying an idle microservice node as a first target node includes:

[0050] Based on the preset evaluation dimensions, the cluster microservice node operation information is obtained and parsed to obtain the node load evaluation parameters. The node load evaluation parameters include at least one of the following: node CPU utilization, memory usage, and the length of the pending task queue;

[0051] According to the node load evaluation parameters, combined with the corresponding preset node load weights, a weighted calculation is performed to obtain the node load evaluation value;

[0052] In response to a node load evaluation value of one or more microservice nodes being less than or equal to a first load threshold, determining that the microservice node is idle and setting it as a first target node, wherein the first load threshold is preferably 0.2-0.4;

[0053] In response to a node load evaluation value of one or more microservice nodes being greater than a first load threshold, the microservice node is determined to be busy.

[0054] Specifically, in this embodiment, through multi-dimensional load parameters and weighted evaluation, idle and busy nodes can be identified more accurately, avoiding misjudgment caused by a single indicator, thereby rationally utilizing cluster resources, achieving node load balancing, and improving the overall system processing capability and response efficiency.

[0055] In one embodiment, based on the independent directory tree, generating a corresponding directory statistics task and assigning it to the first target node includes:

[0056] Construct task description information according to multiple independent directory trees respectively, and generate corresponding multiple directory statistics tasks according to the task description information;

[0057] Obtain the number of independent directory trees and the number of first target nodes respectively, and determine whether the independent directory trees are more than the first target nodes;

[0058] In response to no, assigning less than or equal to one directory statistics task to the first target node;

[0059] In response to the above, the plurality of directory statistics tasks are divided into a first task group whose number is less than or equal to the first target node, and the first task group is allocated to the first target node.

[0060] Specifically, in this embodiment, reasonable task allocation is performed based on the relationship between the number of independent directory trees and target nodes, ensuring that the task load of each node is moderate, avoiding node overload or resource waste, improving the system's parallel processing efficiency, and ensuring the fairness of task allocation and the timeliness of execution.

[0061] In one embodiment, dividing a plurality of directory statistics tasks into a plurality of first task groups and allocating the first task groups to a first target node includes:

[0062] Parse the task description information of multiple directory statistics tasks to obtain the directory size, directory hierarchy depth, and directory last modification time;

[0063] Based on the directory size, directory hierarchy depth, and directory last modification time, combined with the preset load factor, complexity factor, and urgency factor, a weighted sum is performed to obtain the load estimate of multiple directory statistics tasks.

[0064] Initially arrange and group multiple directory statistics tasks to obtain multiple initial task groups;

[0065] Intra-group task migration is performed between multiple initial task groups so that the total load estimation deviation of the multiple initial task groups is less than or equal to the preset deviation threshold, and multiple first task groups are generated, wherein the preset deviation threshold is preferably 0.2-0.3, so that the total load of each first task group is similar.

[0066] Specifically, in this embodiment, tasks are grouped by integrating multiple task characteristics and load parameters, and the load balance between task groups is dynamically adjusted to effectively avoid the problem of some nodes being overloaded or underloaded, thereby improving the overall balance of task execution and system throughput.

[0067] In one embodiment, in response to the first target node receiving the directory statistics task, determining whether the corresponding task directory tree needs to be split includes:

[0068] Parse the task description information of the directory statistics task to obtain the maximum hierarchical depth of the corresponding task directory tree;

[0069] Setting a split threshold based on the number of first target nodes and the average level of the plurality of independent directory trees;

[0070] In response to a maximum hierarchical depth of the task directory tree being greater than a split threshold, determining that the task directory tree needs to be split;

[0071] In response to the maximum hierarchical depth of the task directory tree being less than or equal to the split threshold, it is determined that the task directory tree does not need to be split.

[0072] Specifically, in this embodiment, by setting a reasonable splitting threshold for the depth of the directory tree hierarchy, it is possible to dynamically identify directory trees with complex structures and large task volumes and perform reasonable splitting. This not only prevents a single task from being too large, resulting in low processing efficiency or node resource exhaustion, but also avoids operational redundancy caused by excessive splitting, thereby improving the manageability and execution efficiency of tasks.

[0073] In one embodiment, the split threshold is set based on the number of first target nodes and the average level of multiple independent directory trees, including:

[0074] Obtain the number of first target nodes and the average levels of multiple independent directory trees, combine them with the preset adjustment coefficient, and dynamically calculate the split threshold using the following formula:

[0075] T splkit = L avg + α ·log ( N node +1), where T splkit represents the split threshold, L avg Represents the average level of multiple independent directory trees, α represents the preset adjustment coefficient, N node Indicates the number of the first target node.

[0076] Specifically, this embodiment enhances the system's adaptability to directory trees of different sizes and structures by dynamically adjusting the split threshold, ensuring that the split strategy can be flexibly adjusted as node resources and directory complexity change, thereby improving the rationality of task splitting and the overall adaptability of the system.

[0077] In one embodiment, Figure 4 and Figure 7 As shown, if it is, a sub-directory tree is extracted from the task directory tree and a corresponding statistical sub-task is generated, and a second target node is selected from the cluster microservice node to receive and execute the statistical sub-task, including:

[0078] The first target node determines whether there is a microservice node in the cluster that is currently idle or a first target node that has not been assigned a directory statistics task as an assignable microservice node;

[0079] In response to "yes", the first target node sets the assignable microservice node as the second target node, traverses one or more sub-paths under the task directory tree, and selects sub-directories that are non-empty and can independently form a tree structure as candidate sub-directories;

[0080] The first target node evaluates and sorts the extraction priorities of the candidate subdirectories, selects the top-ranked subdirectories as split subdirectories based on the number of second target nodes, constructs a subdirectory tree, and generates corresponding statistical subtasks to be assigned to the second target node for execution;

[0081] The first target node deletes the path corresponding to the split subdirectory from the task directory tree, obtains the retained subdirectory tree and constructs the corresponding retained statistics subtask executed by the first target node;

[0082] In response to the first target node receiving completion information of the plurality of statistical subtasks, obtaining completion information of the retained statistical subtask, generating task completion information and sending the task completion information to the allocation node;

[0083] If the response is no, the first target node suspends splitting the task directory tree, directly executes the directory statistics task, generates task completion information and sends it to the allocation node.

[0084] It is worth noting that each microservice node in the cluster can serve as the first target node or the second target node, and so on. It can serve as an execution node to process received tasks and as a local allocation node to distribute tasks.

[0085] It is further explained that if Figure 4 As shown, taking microservice node 3 as the third target node receiving a statistics subtask as an example, if microservice node 1 or microservice node 2 has completed its assigned directory statistics task or statistics subtask, the assigned node is determined to be an idle microservice node, and the assigned microservice node can be assigned. Microservice node 3 can then split the statistics subtask again and assign it to microservice node 1 and microservice node 2 respectively. Therefore, in this embodiment, the assignment node does not uniformly assign and issue tasks. Instead, before any microservice node executes the assigned task, it checks whether there are idle nodes in the current cluster. If so, the assigned task can be split again to ensure that all microservice nodes in the cluster are fully utilized.

[0086] Specifically, in this embodiment, by adopting a multi-level task splitting and allocation mechanism, the directory statistics tasks can be further refined and parallelized, the concurrent processing capabilities of the cluster microservice nodes can be fully utilized, the system throughput and task processing efficiency can be improved, and at the same time, the complex and changing directory structure can be flexibly responded to.

[0087] In one embodiment, to improve the concurrent execution efficiency of the system and reduce network delays during the task result transmission process, the allocation node uses a same-source priority task allocation strategy after splitting the task directory tree to generate multiple statistical subtasks, specifically including:

[0088] In response to the presence of allocable microservice nodes being greater than the number of statistical subtasks, constructing source directory path information to which multiple statistical subtask records belong;

[0089] Based on the source directory path information, multiple subtasks with the same or similar source paths are preferentially scheduled to microservice nodes with adjacent physical topology structures or consistent resource domains. Resource domains can include the same rack node, the same network exchange area, the same logical resource pool, or a shared data channel.

[0090] The execution results of subtasks are obtained in real time. If some subtask nodes fail to execute, the path affinity is retained during rescheduling, that is, the nodes in the same group to which the atomic task is attached are given priority for redistribution.

[0091] It is worth noting that this same-origin priority task allocation strategy is also applicable to directory tree tasks that have not been split. In the initial scheduling stage, it attempts to aggregate multiple independent directory trees with path affinity and allocate them to microservice nodes with physically adjacent data to further enhance resource utilization. At the same time, it is not limited to the allocation node and can also be used for any microservice node to which a task is assigned.

[0092] Specifically, this embodiment effectively reduces the network delay and relay hops of data transmission during task execution by adopting a same-source priority allocation strategy, improves the efficiency of local data aggregation, reduces network load, increases task execution speed and system resource utilization, and ensures efficient and stable operation of the distributed system.

[0093] In one embodiment, constructing a subdirectory tree and generating corresponding statistical subtasks to be assigned to the second target node for execution further includes:

[0094] In response to the second target node receiving the statistical subtask, it is again determined whether there is an allocable microservice node in the cluster microservice nodes;

[0095] In response to "yes", the second target node will allocate the microservice node as the third target node, parse the statistical subtask to obtain a subdirectory tree, split the subdirectory tree again, generate the corresponding statistical subtask and send it to the third target node for execution;

[0096] The second target node executes the corresponding sub-task of retaining statistics after the sub-directory tree is split;

[0097] In response to the second target node receiving completion information of multiple statistical grandchild tasks, the second target node obtains completion information of the retained statistical grandchild task, generates completion information of the statistical subtask, and sends it to the first target node;

[0098] The third target node repeatedly determines whether there is an allocable microservice node in the cluster microservice node and repeats the corresponding execution steps until the judgment result is no or the maximum level value of the subdirectory tree is less than the preset minimum split threshold;

[0099] In response to no or the maximum level value of the sub-directory tree is less than the preset minimum splitting threshold, the second target node suspends splitting the sub-directory tree, directly executes the statistical subtask, generates completion information of the statistical subtask and sends it to the first target node.

[0100] Specifically, in this embodiment, a multi-level task structure is formed by recursively splitting the sub-directory tree, and the task granularity is further subdivided, so that complex directories can be processed in a multi-level decentralized manner, improving the parallelism and flexibility of processing, enhancing the system's ability to handle large-scale directory statistical tasks, and preventing single-point overload.

[0101] In one embodiment, after the first target node receives the directory statistics task, the method further includes:

[0102] Based on the preset bottom-up traversal strategy, set the traversal stack and traversal queue and combine them to obtain the traversal channel;

[0103] In response to the first target node not splitting the task directory tree, the first target node processes the task directory tree through the traversal channel, completes the leaf node traversal of the task directory tree, generates directory update information accordingly, and sends it to the allocation node;

[0104] In response to the first target node having split the task directory tree, the second target node processes the sub-directory tree by traversing the channel, generates sub-directory update information, and sends it to the first target node;

[0105] The first target node generates reserved sub-directory update information by traversing the channel processing reserved sub-directory tree, and generates directory update information by combining the sub-directory update information.

[0106] Specifically, in this embodiment, a bottom-up preset traversal strategy is used to flexibly combine the traversal stack and the traversal queue to form a traversal channel, thereby realizing efficient traversal and change information generation of the task directory tree and its split sub-directory trees. This not only ensures the complete statistics of the unsplit directory, but also supports the hierarchical collaborative processing of the split directory, thereby improving the concurrent execution efficiency of the directory statistics and the accuracy of data synchronization, and enhancing the scalability and responsiveness of the system.

[0107] In one embodiment, based on a preset bottom-up traversal strategy, a traversal stack and a traversal queue are set and combined to obtain a traversal channel, including:

[0108] Set up a traversal stack, perform depth-first traversal on the received directory tree to be traversed, push the directories corresponding to the directory tree to be traversed into the traversal stack in sequence, and gradually identify several leaf nodes;

[0109] Compare the directories corresponding to several leaf nodes with the corresponding directory information in the underlying storage and the directory information stored in the database to determine whether there are structural differences or attribute changes, and determine several changed leaf nodes;

[0110] Set up a traversal queue, add several changed leaf nodes to the traversal queue in sequence, process them in the first-in-first-out order, and generate corresponding directory change information;

[0111] The directory change information is merged upward to generate the directory update information corresponding to the directory tree to be traversed.

[0112] Specifically, this embodiment combines a depth-first traversal stack and a first-in-first-out traversal queue to accurately identify leaf nodes that have changed in the directory tree, and merges the change information layer by layer to generate complete directory update data, which significantly improves the accuracy and processing efficiency of directory change detection, ensures real-time synchronization and consistency of directory status, and is suitable for efficient management of large-scale complex directories.

[0113] In one embodiment, Figure 3 As shown, the first target node processes the task directory tree through the traversal channel to complete the leaf node traversal of the task directory tree, including:

[0114] In response to receiving the task directory tree through the traversal stack, the root directory of the task directory tree is pushed into the stack, and the directory at the top of the stack is popped out in sequence and used as the current popped directory;

[0115] Respectively obtain information of the current pop directory in the underlying storage and the database, compare them, and perform corresponding processing according to the comparison results, including: in response to the information of the current pop directory being stored in both the underlying storage and the database, determine whether the modification times of the two are consistent; if they are consistent, obtain a layer of subdirectories of the current pop directory from the database and add them to the stack; if they are inconsistent, obtain a layer of subdirectories of the current pop directory from the underlying storage and add them to the stack; in response to the information of the current pop directory not existing in the underlying storage but existing in the database, delete the current pop directory and all corresponding subdirectories in the database; in response to the information of the current pop directory existing in the underlying storage but not existing in the database, obtain a layer of subdirectories of the current pop directory from the underlying storage and add them to the stack;

[0116] Determine whether the current pop directory is a leaf node of the task directory tree and whether it has changed, and perform an enqueue operation based on the judgment result, including: if it is a leaf node and has changed, enqueue the current pop directory; if it is not a leaf node and all child nodes under the current pop directory have been enqueued, then determine whether the current pop directory and its child nodes have changed, if so, enqueue the current pop directory, if not, do not enqueue the current pop directory, where the current directory or node is considered to have changed if any of the following items is met: the underlying storage exists but the database does not exist, the underlying storage does not exist but the database exists, both exist but the modification time is inconsistent, and the directory summary information is inconsistent;

[0117] The current out-of-queue directory at the head of the queue is sequentially dequeued as the current out-of-queue directory. In response to the current out-of-queue directory being a leaf node, corresponding addition, deletion, and modification operations are performed according to the records of the database and the underlying storage, including: when the information of the current out-of-queue directory does not exist in the database but exists in the underlying storage, an insert operation is performed; when the information of the current out-of-queue directory exists in the database and is inconsistent with the information in the underlying storage, an update operation is performed; when the information of the current out-of-queue directory does not exist in the underlying storage, a delete operation is performed;

[0118] In response to the current dequeued directory information being a non-leaf node, a directory size merge operation is performed, and it is determined whether the merge result is consistent with the directory size stored in the database. If so, no update is performed; otherwise, the directory size stored in the database is updated based on the merge result.

[0119] Specifically, this embodiment achieves dynamic update and synchronization of the directory by comparing the underlying storage and database directory information, ensuring the consistency and integrity of the database and underlying storage data, reflecting directory changes in a timely manner, and improving system accuracy and data reliability.

[0120] In one embodiment, after generating a corresponding directory statistics task based on the independent directory tree and assigning it to the first target node, the method further includes:

[0121] In response to the allocation node not receiving the task completion information fed back by the first target node within a preset period or the allocation node receiving the fault information fed back by the first target node, the status of the first target node is determined to be abnormal, and the corresponding directory statistics task is reallocated to other idle microservice nodes for execution.

[0122] Specifically, in this embodiment, through real-time monitoring of task execution status and abnormality judgment, rapid recovery and rescheduling of faulty tasks are achieved, the fault tolerance and stability of the system are improved, long-term task blocking is avoided, and the continuity and reliability of directory statistics work are ensured.

[0123] In one embodiment, Figure 5 As shown, the corresponding directory statistics tasks are reallocated to other idle microservice nodes for execution, and also include:

[0124] Any microservice node registers at startup and marks its initial running state as idle. When it receives the assigned directory statistics task as the first target node, it marks its own running state as busy.

[0125] When the microservice node finds that there is an allocatable microservice node in the cluster, it assigns the corresponding statistical subtask to the allocatable microservice node, changes its status to busy, and records the corresponding node information;

[0126] When the microservice node finds a microservice node in the cluster that is continuously busy, it will prioritize completing its own running tasks, and then continue to execute the tasks assigned to the microservice node that is continuously busy and mark its status as abnormal.

[0127] After the microservice node in an abnormal state is repaired and returns to normal state, it will mark the state as idle by itself.

[0128] Specifically, this embodiment effectively ensures the healthy operation of nodes in the cluster and the continuous execution of tasks through dynamic marking of node status and abnormal repair mechanism, reduces the impact of node failures on the overall performance of the system, and improves the reliability and resource utilization efficiency of the cluster.

[0129] In one embodiment, Figure 8 As shown, cluster microservice nodes communicate with each other through JSON-RPC to obtain the status and feedback information of each microservice node in real time; cluster microservice nodes receive and execute tasks issued by the allocation node according to the database, generate directory update information and store it in their respective microservice node databases; set up a summary synchronization database to merge the directory update information stored in each microservice node database and merge the directory size at the same time. After the merger is completed, the summary is fed back to the database and synchronized to each microservice node database.

[0130] Specifically, this embodiment uses the JSON-RPC communication mechanism to achieve efficient information exchange and status synchronization between nodes, and combines it with a summary synchronization database for centralized management, thereby improving the system's information consistency, coordination and overall management efficiency, and realizing efficient collaborative work of distributed systems.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0132] The embodiment of the present application also provides a directory management device, such as Figure 9 As shown, it includes: directory tree construction and division module, idle node identification module, task allocation module, task splitting and statistics module and directory update module, among which,

[0133] A directory tree construction and division module is configured to, in response to the distribution node receiving the directory update task, construct a directory tree based on the directory information stored in the database, and divide the directory tree according to path independence to obtain multiple independent directory trees;

[0134] An idle node identification module is used to obtain the current running status of the cluster microservice node and identify the idle microservice node as the first target node;

[0135] A task allocation module, configured to generate corresponding directory statistics tasks based on the independent directory tree and allocate the tasks to the first target node;

[0136] The task splitting and statistics module is used to determine whether the corresponding task directory tree needs to be split in response to the first target node receiving the directory statistics task. If so, it extracts the sub-directory tree from the task directory tree and generates the corresponding statistics sub-task, selects the second target node from the cluster microservice node to receive and execute the statistics sub-task, and if not, automatically executes the directory statistics task;

[0137] The directory update module is used to parse the task completion information fed back by the first target node in response to the allocation node receiving the task completion information, obtain the directory update information and write it into the database to update the directory information.

[0138] The directory tree construction and division module is also used to identify multiple independent directory sets in the directory tree whose root paths do not contain each other, and respectively form multiple independent directory trees.

[0139] The idle node identification module is also used to obtain and parse the cluster microservice node operation information based on preset evaluation dimensions to obtain node load evaluation parameters; perform weighted calculation based on the node load evaluation parameters in combination with the corresponding preset node load weights to obtain a node load evaluation value; in response to the node load evaluation value of one or more microservice nodes being less than or equal to a first load threshold, the microservice node is determined to be idle and set as the first target node; in response to the node load evaluation value of one or more microservice nodes being greater than the first load threshold, the microservice node is determined to be busy.

[0140] The task assignment module is also used to construct task description information according to multiple independent directory trees respectively, and generate corresponding multiple directory statistical tasks according to the task description information; obtain the number of independent directory trees and the number of first target nodes respectively, and judge whether the independent directory trees are more than the first target nodes; in response to no, less than or equal to one directory statistical task is assigned to the first target node; in response to yes, multiple directory statistical tasks are divided into a first task group with a number less than or equal to the first target node, and the first task group is assigned to the first target node.

[0141] The task allocation module is also used to parse the task description information of multiple directory statistics tasks to obtain the directory size, directory hierarchy depth and directory last modification time; perform weighted summation based on the directory size, directory hierarchy depth and directory last modification time, combined with the preset load coefficient, complexity coefficient and urgency coefficient, to obtain the load estimation of multiple directory statistics tasks; initially arrange and group the multiple directory statistics tasks to obtain multiple initial task groups; perform intra-group task migration between the multiple initial task groups so that the total load estimation deviation of the multiple initial task groups is less than or equal to the preset deviation threshold, and generate multiple first task groups.

[0142] The task allocation module is also used to determine that the status of the first target node is abnormal in response to the allocation node not receiving task completion information fed back by the first target node within a preset period or the allocation node receiving fault information fed back by the first target node, and reallocate the corresponding directory statistics task to other idle microservice nodes for execution.

[0143] The task splitting and statistics module is also used to parse the task description information of the directory statistics task to obtain the maximum hierarchical depth of the corresponding task directory tree; set a splitting threshold based on the number of first target nodes and the average level of multiple independent directory trees; in response to the maximum hierarchical depth of the task directory tree being greater than the splitting threshold, it is determined that the task directory tree needs to be split; in response to the maximum hierarchical depth of the task directory tree being less than or equal to the splitting threshold, it is determined that the task directory tree does not need to be split.

[0144] The task splitting and statistics module is also used to obtain the number of first target nodes and the average level of multiple independent directory trees, and dynamically calculate the splitting threshold in combination with the preset adjustment coefficient.

[0145] The task splitting and statistics module is also used for the first target node to determine whether there is a microservice node in the cluster microservice node that is currently idle or a first target node that has not been assigned a directory statistics task as an assignable microservice node; in response to yes, the first target node sets the assignable microservice node as the second target node, traverses the first or multi-level subpath under the task directory tree, and filters out non-empty subdirectories that can independently constitute a tree structure as candidate subdirectories; the first target node extracts priority evaluation and sorts from the candidate subdirectories, selects the top-ranked subdirectories as split subdirectories based on the number of second target nodes, constructs a subdirectory tree and generates corresponding statistics subtasks to be assigned to the second target node for execution; the first target node deletes the path corresponding to the split subdirectory from the task directory tree, obtains a retained subdirectory tree and constructs a corresponding retained statistics subtask to be executed by the first target node; in response to the first target node receiving completion information of multiple statistics subtasks, the completion information of the retained statistics subtask is obtained, task completion information is generated and sent to the allocation node; in response to no, the first target node suspends splitting the task directory tree, directly executes the directory statistics task, generates task completion information and sends it to the allocation node.

[0146] The task splitting and statistics module is also used to, in response to the second target node receiving the statistics subtask, determine again whether there is an assignable microservice node in the cluster microservice node; in response to yes, the second target node uses the assignable microservice node as the third target node, parses the statistics subtask to obtain the sub-directory tree, splits the sub-directory tree again, generates the corresponding statistics grandson task and sends it to the third target node for execution; the second target node executes the corresponding retained statistics grandson task after the sub-directory tree is split; in response to the second target node receiving the completion information of multiple statistics grandson tasks, obtains the completion information of the retained statistics grandson task, generates the completion information of the statistics subtask and sends it to the first target node; the third target node repeatedly determines whether there is an assignable microservice node in the cluster microservice node and repeats the corresponding execution steps until the judgment result is no or the maximum level value of the sub-directory tree is less than the preset minimum splitting threshold; in response to no or the maximum level value of the sub-directory tree is less than the preset minimum splitting threshold, the second target node suspends splitting the sub-directory tree, directly executes the statistics subtask, generates the completion information of the statistics subtask and sends it to the first target node.

[0147] The task splitting and statistics module is also used to set a traversal stack and a traversal queue based on a bottom-up preset traversal strategy and combine them to obtain a traversal channel; in response to the first target node not splitting the task directory tree, the first target node processes the task directory tree through the traversal channel, completes the leaf node traversal of the task directory tree, generates directory update information accordingly and sends it to the allocation node; in response to the first target node having split the task directory tree, the second target node processes the sub-directory tree through the traversal channel, generates sub-directory update information and sends it to the first target node; the first target node generates retained sub-directory update information by processing the retained sub-directory tree through the traversal channel, and generates directory update information in combination with the sub-directory update information.

[0148] The task splitting and statistics module also sets up a traversal stack to perform a depth-first traversal on the received directory tree to be traversed, and pushes the directories corresponding to the directory tree to be traversed into the traversal stack in turn, and gradually identifies several leaf nodes; compares the directories corresponding to several leaf nodes with the corresponding directory information in the underlying storage and the directory information stored in the database to determine whether there are structural differences or attribute changes, and determines several changed leaf nodes; sets up a traversal queue, and adds several changed leaf nodes to the traversal queue in turn, and processes them in sequence according to the first-in-first-out order to generate corresponding directory change information; merges the directory change information upward to generate directory update information corresponding to the directory tree to be traversed.

[0149] For the description of the features in the embodiment corresponding to the directory management device, please refer to the relevant description of the embodiment corresponding to the directory management method, and no further details will be given here.

[0150] The embodiment of the present application also provides an electronic device, such as Figure 10 As shown, it includes a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above directory management method embodiments.

[0151] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above directory management method embodiments when running.

[0152] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0153] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above directory management method embodiments are implemented.

[0154] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned directory management method embodiments are implemented.

[0155] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] The above is a detailed introduction to the directory management method, device, equipment, storage medium and product provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A directory management method, characterized in that: include: In response to the distribution node receiving the directory update task, the distribution node constructs a directory tree based on the directory information stored in the database, and divides the directory tree according to path independence to obtain multiple independent directory trees; Obtain the current running status of the cluster microservice node and identify the idle microservice node as the first target node; Based on the independent directory trees, corresponding directory statistics tasks are generated and assigned to the first target node, including: constructing task description information according to the plurality of independent directory trees respectively, and generating the corresponding plurality of directory statistics tasks according to the task description information; obtaining the number of the independent directory trees and the number of the first target nodes respectively, and determining whether the independent directory trees are more than the first target nodes; in response to no, assigning less than or equal to one directory statistics task to the first target node; in response to yes, dividing the plurality of directory statistics tasks into a first task group whose number is less than or equal to the first target node, and assigning the first task group to the first target node; In response to the first target node receiving the directory statistics task, it is determined whether the corresponding task directory tree needs to be split. If so, a sub-directory tree is extracted from the task directory tree and a corresponding statistics sub-task is generated. A second target node is selected from the cluster microservice node to receive and execute the statistics sub-task. If not, the directory statistics task is automatically executed. In response to the allocation node receiving the task completion information fed back by the first target node, the allocation node parses the task completion information, obtains directory update information, and writes the updated information into the database to update the directory information.

2. The directory management method according to claim 1, characterized in that: The directory tree is divided according to path independence to obtain multiple independent directory trees, including: A plurality of independent directory sets whose root paths do not contain each other in the directory tree are identified, and each set constitutes a plurality of the independent directory trees.

3. The directory management method according to claim 1, wherein: The obtaining of the current operating status of the cluster microservice node and identifying the idle microservice node as the first target node includes: Based on the preset evaluation dimensions, obtain and analyze the cluster microservice node operation information to obtain the node load evaluation parameters; A weighted calculation is performed based on the node load evaluation parameters and the corresponding preset node load weights to obtain a node load evaluation value; In response to a node load evaluation value of one or more microservice nodes being less than or equal to a first load threshold, determining that the microservice node is idle and setting it as the first target node; In response to the node load evaluation value of one or more microservice nodes being greater than the first load threshold, the microservice nodes are determined to be busy.

4. The directory management method according to claim 1, wherein: The dividing the plurality of directory statistics tasks into a plurality of first task groups, and allocating the first task groups to the first target node, comprises: Parsing the task description information of the plurality of directory statistics tasks to obtain the directory size, directory level depth and directory last modification time; According to the directory size, the directory level depth and the last modification time of the directory, combined with the preset load factor, complexity factor and urgency factor, a weighted sum is performed to obtain the load estimation of the plurality of directory statistics tasks; Initially arranging and grouping the plurality of directory statistics tasks to obtain a plurality of initial task groups; Intra-group task migration is performed between the multiple initial task groups so that a total load estimation deviation of the multiple initial task groups is less than or equal to a preset deviation threshold, thereby generating a plurality of first task groups.

5. The directory management method according to claim 1, wherein: In response to the first target node receiving the directory statistics task, determining whether the corresponding task directory tree needs to be split includes: Parsing the task description information of the directory statistics task to obtain the maximum hierarchical depth of the corresponding task directory tree; Setting a splitting threshold based on the number of the first target nodes and the average level of the plurality of the independent directory trees; In response to the maximum hierarchical depth of the task directory tree being greater than the splitting threshold, determining that the task directory tree needs to be split; In response to the maximum hierarchical depth of the task directory tree being less than or equal to the splitting threshold, it is determined that the task directory tree does not need to be split.

6. The directory management method according to claim 5, characterized in that: The step of setting the splitting threshold based on the number of the first target nodes and the average level of the plurality of independent directory trees includes: The number of the first target nodes and the average level of the plurality of independent directory trees are obtained, and the split threshold is dynamically calculated using the following formula in combination with a preset adjustment coefficient: T splkit = L avg + α ·log ( N node +1), where T splkit represents the split threshold, L avg Represents the average level of multiple independent directory trees, α represents the preset adjustment coefficient, N node Indicates the number of the first target node.

7. The directory management method according to claim 1, characterized in that: If yes, extract the sub-directory tree from the task directory tree and generate the corresponding statistical sub-task, select the second target node from the cluster microservice node to receive and execute the statistical sub-task, including: The first target node determines whether there is a microservice node in the cluster microservice node that is currently idle or a first target node that is not assigned a directory statistics task as an assignable microservice node; In response to "yes", the first target node sets the assignable microservice node as the second target node, traverses one or more sub-paths under the task directory tree, and selects sub-directories that are non-empty and can independently form a tree structure as candidate sub-directories; The first target node evaluates and sorts the extraction priorities of the candidate subdirectories, selects the top-ranked subdirectories as split subdirectories based on the number of the second target nodes, constructs the subdirectory tree, and generates corresponding statistical subtasks, which are assigned to the second target node for execution; The first target node deletes the path corresponding to the split subdirectory from the task directory tree, obtains a retained subdirectory tree, and constructs a corresponding retained statistics subtask executed by the first target node; In response to the first target node receiving the completion information of the plurality of statistical subtasks, obtaining the completion information of the retaining statistical subtask, generating the task completion information and sending it to the allocation node; In response to no, the first target node suspends splitting the task directory tree, directly executes the directory statistics task, generates the task completion information and sends it to the allocation node.

8. The directory management method according to claim 7, characterized in that: The step of constructing the subdirectory tree and generating corresponding statistical subtasks to be assigned to the second target node for execution includes: In response to the second target node receiving the statistical subtask, determining again whether the allocable microservice node currently exists in the cluster microservice nodes; In response to "yes", the second target node uses the assignable microservice node as the third target node, parses the statistical subtask to obtain the subdirectory tree, splits the subdirectory tree again, generates corresponding statistical subtasks, and sends them to the third target node for execution; The second target node executes the corresponding sub-task of retaining statistics after the sub-directory tree is split; In response to the second target node receiving the completion information of the plurality of statistical grandchild tasks, the second target node obtains the completion information of the retain statistical grandchild task, generates the completion information of the statistical subtask, and sends it to the first target node; The third target node repeatedly determines whether the allocable microservice node currently exists in the cluster microservice node and repeats the corresponding execution steps until the determination result is no or the maximum level value of the subdirectory tree is less than a preset minimum split threshold; In response to no or the maximum level value of the sub-directory tree is less than the preset minimum splitting threshold, the second target node suspends splitting the sub-directory tree, directly executes the statistical subtask, generates completion information of the statistical subtask and sends it to the first target node.

9. The directory management method according to claim 7, characterized in that: After the first target node receives the directory statistics task in response, the method further includes: Based on the preset bottom-up traversal strategy, set the traversal stack and traversal queue and combine them to obtain the traversal channel; In response to the first target node not splitting the task directory tree, the first target node processes the task directory tree through the traversal channel, completes the leaf node traversal of the task directory tree, generates the directory update information accordingly, and sends it to the allocation node; In response to the first target node having split the task directory tree, the second target node processes the sub-directory tree through the traversal channel, generates sub-directory update information and sends it to the first target node; The first target node processes the reserved sub-directory tree through the traversal channel to generate reserved sub-directory update information, and generates the directory update information by combining the sub-directory update information.

10. The directory management method according to claim 9, characterized in that: The bottom-up preset traversal strategy is based on setting the traversal stack and traversal queue and combining them to obtain the traversal channel, including: Setting the traversal stack, performing depth-first traversal on the received directory tree to be traversed, pushing the directories corresponding to the directory tree to be traversed into the traversal stack in sequence, and gradually identifying a number of leaf nodes; Comparing the directories corresponding to the leaf nodes with the corresponding directory information in the underlying storage and the directory information stored in the database to determine whether there are structural differences or attribute changes, and determining the number of changed leaf nodes; Setting the traversal queue, adding the plurality of changed leaf nodes to the traversal queue in sequence, processing them in sequence according to the first-in-first-out order, and generating corresponding directory change information; The directory change information is merged upward to generate directory update information corresponding to the directory tree to be traversed.

11. The directory management method according to claim 1, wherein: After generating a corresponding directory statistics task based on the independent directory tree and allocating it to the first target node, the method further includes: In response to the allocation node not receiving the task completion information fed back by the first target node within a preset period or the allocation node receiving the fault information fed back by the first target node, the state of the first target node is determined to be abnormal, and the corresponding directory statistics task is reallocated to other idle microservice nodes for execution.

12. A directory management device, characterized in that: include: a directory tree construction and division module, configured to, in response to a distribution node receiving a directory update task, construct a directory tree based on directory information stored in a database, and divide the directory tree into multiple independent directory trees based on path independence; An idle node identification module is used to obtain the current running status of the cluster microservice node and identify the idle microservice node as the first target node; The task assignment module is configured to generate corresponding directory statistics tasks based on the independent directory trees and assign the tasks to the first target node, comprising: constructing task description information based on the plurality of independent directory trees, and generating the corresponding plurality of directory statistics tasks based on the task description information; obtaining the number of the independent directory trees and the number of the first target nodes, and determining whether the number of the independent directory trees exceeds the first target nodes; in response to a negative result, assigning less than or equal to one directory statistics task to the first target node; in response to a positive result, dividing the plurality of directory statistics tasks into a first task group whose number is less than or equal to the first target node, and assigning the first task group to the first target node; The task splitting and statistics module is used to determine whether the corresponding task directory tree needs to be split in response to the first target node receiving the directory statistics task. If so, it extracts a sub-directory tree from the task directory tree and generates a corresponding statistics sub-task, selects a second target node from the cluster microservice node to receive and execute the statistics sub-task, and if not, automatically executes the directory statistics task; The directory update module is configured to, in response to the distribution node receiving the task completion information fed back by the first target node, parse the task completion information, obtain directory update information, and write the updated directory information into the database to update the directory information.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the directory management method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the directory management method according to any one of claims 1 to 11.

Images