Directory management method, directory management device, electronic equipment and storage medium

Abstract

The invention discloses a directory management method, a directory management device, electronic equipment and a storage medium, and relates to the technical field of directory management.The directory management method comprises the steps that in response to a directory updating task received by a distribution node, a directory tree is built based on directory information stored in a database, and the directory tree is divided to obtain a plurality of independent directory trees; identifying the idle micro-service node as a first target node; generating a corresponding catalogue statistical task and distributing the catalogue statistical task to the first target node; and in response to the directory statistical task received by the first target node, judging whether a corresponding task directory tree needs to be split, if so, extracting a sub-directory tree from the task directory tree and generating a corresponding statistical sub-task, and selecting a second target node to receive and execute the statistical sub-task. According to the method, the directory tree can be divided based on path independence, and the directory statistics task is dynamically allocated in combination with the cluster micro-service node state, so that distributed concurrent statistics of the directory is realized, the statistics efficiency is improved, the resource consumption is reduced, and the delay is reduced.

Description

Technical Field

[0001] This application relates to the technical field of directory management, and in particular, to a directory management method, a directory management device, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of the artificial intelligence industry, the demand for high-performance computing power from research institutions and enterprises has been continuously increasing. The AI cluster platform has emerged as an important infrastructure to support model training and inference tasks. A core function of an AI cluster is to support users in efficiently storing and managing massive amounts of data. Platform users often need to process file data at the PB level and above, including user home directories, public directories, training dataset directories, and model output directories, etc. In such a high-frequency access and high-concurrency computing environment, the platform has put forward higher requirements for the update efficiency of the directory information of the storage system.

[0003] Currently, mainstream AI platforms mostly adopt the method of full traversal of all directories and files in the storage system for directory information statistics and update. Although this method has a simple implementation logic, it has obvious drawbacks: First, frequent triggering of underlying storage I / O operations during the full traversal of directories will consume a large amount of CPU and memory resources of business nodes, affecting the execution efficiency of computing tasks; Second, when facing a file system structure above the PB level, the overall latency of traversal statistics is relatively high, making it difficult to meet the real-time requirements of scenarios such as model training. Summary of the Invention

[0004] This application provides a directory management method that can partition a directory tree based on path independence, dynamically allocate directory statistics tasks in combination with the status of cluster microservice nodes, and achieve distributed concurrent statistics of directories, so as to at least solve the problems of low statistical efficiency, high resource consumption, and high latency in related technologies.

[0005] This application provides a directory management method, including: In response to a directory update task received by an allocation node, construct a directory tree based on the directory information stored in the database, and partition the directory tree according to path independence to obtain multiple independent directory trees; Obtain the current running status of cluster microservice nodes, and identify idle microservice nodes as first target nodes; Generate corresponding directory statistics tasks based on the independent directory trees and allocate them to the first target nodes; In response to a first target node receiving a directory statistics task, determine whether the corresponding task directory tree needs to be split. If so, extract sub-directory trees from the task directory tree and generate corresponding statistical subtasks, select second target nodes from the cluster microservice nodes to receive and execute the statistical subtasks, and if not, execute the directory statistics task itself; Upon receiving the task completion information fed back by the first target node by the allocation node, the task completion information is parsed to obtain directory update information and written into the database to update the directory information.

[0006] This application also provides a directory management device, including: A directory tree construction and division module, configured to, upon receiving a directory update task by the allocation node, construct a directory tree based on the directory information stored in the database, and divide the directory tree according to path independence to obtain a plurality of independent directory trees; An idle node identification module, configured to obtain the current running status of the cluster microservice nodes and identify the idle microservice nodes as the first target nodes; A task allocation module, configured to generate corresponding directory statistics tasks based on the independent directory trees and allocate them to the first target nodes; A task splitting and statistics module, configured to, upon receiving a directory statistics task by the first target node, determine whether the corresponding task directory tree needs to be split. If so, extract a sub-directory tree from the task directory tree and generate corresponding statistical subtasks, select second target nodes from the cluster microservice nodes to receive and execute the statistical subtasks, and if not, execute the directory statistics task itself; A directory update module, configured to, upon receiving the task completion information fed back by the first target node by the allocation node, parse the task completion information to obtain directory update information and write it into the database to update the directory information.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above directory management methods when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of any of the above directory management methods when executed by a processor.

[0009] The present invention divides the directory tree through path independence, reduces task dependencies and repeated traversals, and can achieve parallel processing of multiple independent directories, improving the concurrency ability of statistical tasks. At the same time, the present invention dynamically identifies idle nodes and assigns tasks using the status of microservice nodes, which can make full use of computing resources, avoid further congestion of high-load nodes, and thus enhance the intelligence and computing efficiency of overall resource scheduling. Moreover, the present invention makes the task granularity have dynamic scalability by determining whether the directory tree can be split into sub-directory tree tasks, avoiding blocking caused by a single node processing an oversized task, and enhancing the controllability and stability of system processing. In addition, the present invention avoids frequent database write operations, reduces the write pressure on the database, and at the same time enhances the centralization and consistency of data updates by feeding back the statistical results layer by layer from each task node and centrally updating them to the database. Therefore, the present invention effectively controls the data scale and IO pressure of the database while ensuring the concurrency and flexibility of task execution through directory tree division and distributed concurrent statistics, thereby improving the speed of directory information update and the system response efficiency in a large-scale AI cluster storage environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0011] Figure 1 It is a schematic diagram of an application environment provided by an embodiment of the present application; Figure 2 It is a flowchart of a directory management method provided by an embodiment of the present application; Figure 3 It is a schematic diagram of a directory traversal process provided by an embodiment of the present application; Figure 4 It is a schematic diagram of a task allocation and execution process provided by an embodiment of the present application; Figure 5 It is a schematic diagram of node status switching and fault handling provided by an embodiment of the present application; Figure 6 It is an example diagram of an independent directory tree provided by an embodiment of the present application; Figure 7 It is an example diagram of a sub-directory tree provided by an embodiment of the present application; Figure 8 It is a schematic diagram of a directory update information storage and summary process provided by an embodiment of the present application; Figure 9 It is a structural block diagram of a directory management device provided by an embodiment of the present application; Figure 10 Schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0012] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0013] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0015] A directory management method provided by the present application can be applied to an application environment as Figure 1 shown. Among them, the terminal 102 communicates with the server 104 through the network. The user performs a change operation on the data stored in the database through the terminal 102, and the terminal 102 sends a directory update task of the database to the server 104. The server 104 reads and traverses the directory information in the database distributively through a number of microservice nodes of the AI cluster set therein to generate directory update information to update the directory information in the database. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0016] As Figure 2 shown, an embodiment of the present application provides a directory management method, including: Step 201, in response to the allocation node receiving a directory update task, a directory tree is constructed based on the directory information stored in the database, and the directory tree is divided according to path independence to obtain multiple independent directory trees; Step 202, obtain the current running state of the cluster microservice nodes, and identify the idle microservice nodes as the first target nodes; Step 203: Generate corresponding directory statistics tasks based on the independent directory tree and allocate them to the first target node; Step 204: In response to the first target node receiving the directory statistics task, determine whether the corresponding task directory tree needs to be split. If so, extract the sub-directory tree from the task directory tree and generate corresponding statistical sub-tasks, select the second target node from the cluster microservice nodes to receive and execute the statistical sub-tasks. If not, execute the directory statistics task by itself; Step 205: In response to the assigned node receiving the task completion information fed back by the first target node, parse the task completion information to obtain directory update information and write it into the database to update the directory information.

[0017] Specifically, the present invention divides the directory tree through path independence, reduces task dependencies and repeated traversals, and can achieve parallel processing of multiple independent directories, improving the concurrency of statistical tasks. At the same time, the present invention dynamically identifies idle nodes based on the microservice node status and allocates tasks, which can make full use of computing resources, avoid further congestion of high-load nodes, and thus enhance the intelligence and computing efficiency of overall resource scheduling. Moreover, the present invention determines whether the directory tree can be split into sub-directory tree tasks, making the task granularity have dynamic scalability, avoiding blocking caused by a single node processing an oversized task, and enhancing the controllability and stability of system processing. In addition, the present invention feeds back the statistical results layer by layer from each task node and centrally updates them to the database, which can avoid frequent database write operations, reduce the write pressure on the database, and at the same time enhance the centralization and consistency of data updates. Therefore, the present invention ensures the concurrency and flexibility of task execution through directory tree division and distributed concurrent statistics, effectively controls the data scale and IO pressure of the database, and thus improves the speed of directory information update and system response efficiency in a large-scale AI cluster storage environment.

[0018] It should be noted that in traditional non-distributed directory statistics solutions, as the amount of stored data continues to increase, the time required for directory traversal and statistics will increase exponentially, resulting in a slowdown in system response and obvious performance bottlenecks. Adopting a distributed division solution can divide a huge directory tree into multiple independent sub-directory trees, which are processed in parallel by different nodes, effectively avoiding the performance bottleneck of single-point processing and significantly improving the efficiency of directory statistics and the overall throughput capacity of the system.

[0019] Furthermore, the purpose of dividing the directory tree in this solution is not only to split the task into smaller units, but more importantly, to reduce the amount of data stored in the database through division, achieve refined management of the data in the database, and improve the read and write efficiency of the database and reduce the storage pressure by quickly counting and updating smaller data blocks, thereby accelerating the data access speed and ensuring that the system can still provide efficient and stable directory statistics services in a large-scale storage environment.

[0020] In one embodiment, as Figure 6 shown, the directory tree is divided according to path independence to obtain multiple independent directory trees, including: Identifying an independent directory set in which multiple root paths in the directory tree do not contain each other, and respectively constituting multiple independent directory trees.

[0021] Specifically, in this embodiment, by dividing the directory tree according to path independence, tasks can be effectively split into multiple independent subtasks that do not interfere with each other, avoiding cross-dependencies between directory trees, and improving the efficiency and accuracy of task parallel processing; at the same time, it provides a clear task boundary for subsequent directory statistics and scheduling, which is conducive to load balancing and task allocation of microservice nodes.

[0022] In one embodiment, the directory information stored in the database is stored through a preset directory storage data structure, and the preset directory storage data structure includes at least one of the following: id number, storage directory or index, last modified time, directory size, last statistics directory size, directory owner, directory level, parent directory of the storage directory, table record update time.

[0023] Specifically, in this embodiment, a structured directory storage data structure with rich attributes is adopted, which is convenient for precise management and quick query of directory information. Especially fields such as modification time and directory level are beneficial to efficiently detecting directory changes, updating and maintaining the status of the directory tree, reducing repeated scans, and improving system response speed and storage consistency.

[0024] In one embodiment, obtaining the current running state of the cluster microservice nodes and identifying the idle microservice nodes as the first target nodes includes: Based on a preset evaluation dimension, obtaining and parsing the running information of the cluster microservice nodes to obtain node load evaluation parameters, and the node load evaluation parameters include at least one of the following: node CPU utilization rate, memory usage rate, length of the task queue to be processed; According to the node load evaluation parameters, combined with the corresponding preset node load weights for weighted calculation, obtaining a node load evaluation value; In response to the node load evaluation value of one or more microservice nodes being less than or equal to the first load threshold, it is determined that the microservice node is idle and set as the first target node, where the first load threshold is preferably 0.2 - 0.4; In response to the node load evaluation value of one or more microservice nodes being greater than the first load threshold, it is determined that the microservice node is busy.

[0025] Specifically, in this embodiment, through multi-dimensional load parameters and weighted evaluation, idle and busy nodes can be identified more accurately, avoiding misjudgment caused by a single indicator, thereby rationally utilizing cluster resources, achieving node load balancing, and improving the overall processing capacity and response efficiency of the system.

[0026] In one embodiment, generating corresponding directory statistics tasks based on independent directory trees and allocating them to a first target node includes: Constructing task description information according to multiple independent directory trees respectively, and generating corresponding multiple directory statistics tasks according to the task description information; Obtaining the number of independent directory trees and the number of first target nodes respectively, and determining whether the number of independent directory trees is more than that of the first target nodes; In response to "no", allocating less than or equal to one directory statistics task to the first target node; In response to "yes", dividing multiple directory statistics tasks into a first task group with a quantity less than or equal to that of the first target nodes, and allocating the first task group to the first target node.

[0027] Specifically, in this embodiment, reasonable task allocation is performed according to the quantity relationship between the independent directory trees and the target nodes, ensuring that the task load of each node is moderate, avoiding node overload or resource waste, improving the parallel processing efficiency of the system, and ensuring the fairness of task allocation and the timeliness of execution.

[0028] In one embodiment, dividing multiple directory statistics tasks into multiple first task groups and allocating the first task groups to a first target node includes: Analyzing the task description information of multiple directory statistics tasks to obtain the directory size, directory level depth, and directory last modification time; According to the directory size, directory level depth, and directory last modification time, combined with preset load coefficients, complexity coefficients, and emergency coefficients, performing weighted summation to obtain load estimates of multiple directory statistics tasks; Performing initial arrangement and grouping on multiple directory statistics tasks to obtain multiple initial task groups; Performing intra-group task migration among multiple initial task groups so that the total load estimate deviation of multiple initial task groups is less than or equal to a preset deviation threshold, generating multiple first task groups, where the preset deviation threshold is preferably 0.2 - 0.3 to make the total load of each first task group similar.

[0029] Specifically, in this embodiment, tasks are grouped by comprehensively considering multiple task characteristics and load parameters, and the load balance between task groups is dynamically adjusted, effectively avoiding the problem of some nodes being overloaded or underloaded, and improving the overall balance of task execution and the system throughput capacity.

[0030] In one embodiment, in response to the first target node receiving a directory statistics task, it is determined whether the corresponding task directory tree needs to be split, including: Parse the task description information of the directory statistics task to obtain the maximum hierarchical depth of the corresponding task directory tree; Based on the number of the first target nodes and the average level of multiple independent directory trees, set a splitting threshold; If the maximum hierarchical depth of the task directory tree is greater than the splitting threshold, it is determined that the task directory tree needs to be split; If the maximum hierarchical depth of the task directory tree is less than or equal to the splitting threshold, it is determined that the task directory tree does not need to be split.

[0031] Specifically, in this embodiment, by setting a reasonable splitting threshold for the hierarchical depth of the directory tree, it is possible to dynamically identify directory trees with complex structures and large amounts of tasks, and perform reasonable splitting. This not only prevents a single task from being too large, resulting in low processing efficiency or node resource exhaustion, but also avoids operational redundancy caused by excessive splitting, improving the manageability and execution efficiency of the task.

[0032] In one embodiment, based on the number of the first target nodes and the average level of multiple independent directory trees, setting a splitting threshold includes: Obtain the number of the first target nodes and the average level of multiple independent directory trees, and combine with a preset adjustment coefficient to dynamically calculate the splitting threshold through the following formula: T splkit = L avg + α ·log ( N node +1), where T splkit represents the splitting threshold, L avg represents the average level of multiple independent directory trees, α represents the preset adjustment coefficient, N node represents the number of the first target nodes.

[0033] Specifically, in this embodiment, by dynamically adjusting the splitting threshold, the adaptability of the system to directory trees of different scales and structures is enhanced, ensuring that the splitting strategy can be flexibly adjusted as the node resources and directory complexity change, improving the rationality of task splitting and the overall adaptive ability of the system.

[0034] In one embodiment, as shown in Figure 4 and Figure 7 if so, extract the sub-directory tree from the task directory tree and generate the corresponding statistical subtask, and select the second target node from the cluster microservice nodes to receive and execute the statistical subtask, including: The first target node determines whether there is currently a microservice node with an idle current state or a first target node that has not been assigned a directory statistics task in the cluster microservice nodes as assignable microservice nodes; In response, the first target node sets the assignable microservice node as the second target node, traverses one or more levels of sub - paths under the task directory tree, and filters out non - empty sub - directories that can independently form a tree structure as candidate sub - directories; The first target node performs extraction, priority evaluation, and sorting on the candidate sub - directories, selects the sub - directories with higher rankings based on the number of second target nodes as split sub - directories, constructs a sub - directory tree, and generates corresponding statistical subtasks for the second target nodes to execute; The first target node deletes the path corresponding to the split sub - directory from the task directory tree, obtains the remaining sub - directory tree, and constructs the corresponding remaining statistical subtask for the first target node to execute; In response to the first target node receiving the completion information of multiple statistical subtasks, it obtains the completion information of the remaining statistical subtask, generates task completion information, and sends it to the assignment node; In response to no, the first target node pauses the splitting of the task directory tree, directly executes the directory statistics task, generates task completion information, and sends it to the assignment node.

[0035] It should be noted that each microservice node in the cluster can be used as the first target node or the second target node. By analogy, it acts both as an execution node to process the received tasks and as a local assignment node to perform task assignment.

[0036] Further explanation is as follows. Figure 4 As shown, taking microservice node 3 as the third target node receiving a statistical grand - task as an example, if microservice node 1 or microservice node 2 has completed its assigned directory statistics task or statistical subtask at this time, it will be determined by the assignment node as an idle microservice node, that is, an assignable microservice node. Microservice node 3 can then split the statistical grand - task again and assign it to microservice node 1 and microservice node 2 respectively. Therefore, in this embodiment, it is not the assignment node that uniformly performs task assignment and distribution once. Instead, before any microservice node executes the assigned task, it queries whether there are idle nodes in the current cluster. If there are, the assigned task can be split again to ensure that the microservice nodes in the cluster are fully utilized.

[0037] Specifically, in this embodiment, by adopting a multi - level task splitting and assignment mechanism, it is possible to further refine and parallelize the directory statistics task, give full play to the concurrent processing capabilities of the cluster microservice nodes, improve system throughput and task processing efficiency, and at the same time flexibly handle the complex and changeable directory structure.

[0038] In one embodiment, to improve the concurrent execution efficiency of the system and reduce the network latency during the task result feedback process, after the allocation node splits the task directory tree to generate multiple statistical subtasks, it adopts a task allocation strategy with priority for the same source, which specifically includes: When there are assignable microservice nodes with a quantity greater than the number of statistical subtasks, construct the source directory path information to which multiple statistical subtask records belong; Based on the source directory path information to which they belong, preferentially schedule multiple subtasks with the same or similar source paths to microservice nodes adjacent in the physical topology structure or with the same resource domain for execution. The resource domain can include the same rack nodes, the same network switching area, the same logical resource pool, or shared data channels, etc.; Obtain the execution results of the subtasks in real time. If some subtask nodes fail to execute, retain the path affinity during re-scheduling for allocation, that is, preferentially select the nodes in the same group to which the atomic task is attached for re-allocation.

[0039] It should be noted that this task allocation strategy with priority for the same source is also applicable to directory tree tasks that have not been split. At the initial scheduling stage, attempt to aggregate and allocate multiple independent directory trees with path affinity to microservice nodes physically adjacent in data, further enhancing resource utilization; at the same time, it is not limited to the allocation node and can also be used for any microservice node to which a task is allocated.

[0040] Specifically, in this embodiment, by adopting the allocation strategy with priority for the same source, the network latency and relay hops of data transmission during task execution are effectively reduced, the local data aggregation efficiency is improved, the network load is reduced, the task execution speed and the utilization rate of system resources are increased, and the efficient and stable operation of the distributed system is ensured.

[0041] In one embodiment, constructing a sub-directory tree and generating corresponding statistical subtasks for execution by a second target node further includes: When the second target node receives a statistical subtask, determine again whether there are currently assignable microservice nodes in the cluster microservice nodes; If so, the second target node uses the assignable microservice node as a third target node, parses the statistical subtask to obtain the sub-directory tree, splits the sub-directory tree again to generate corresponding statistical sub-subtasks, and sends them to the third target node for execution; The second target node executes the retained statistical sub-subtasks corresponding to the sub-directory tree splitting; When the second target node receives the completion information of multiple statistical sub-subtasks, obtain the completion information of the retained statistical sub-subtasks, generate the completion information of the statistical subtask, and send it to the first target node; The third target node repeatedly determines whether there are assignable microservice nodes in the cluster microservice nodes and repeats the corresponding execution steps until the judgment result is negative or the maximum hierarchical value of the sub-directory tree is less than the preset minimum splitting threshold; In response to the negative or the maximum hierarchical value of the sub-directory tree being less than the preset minimum splitting threshold, the second target node pauses splitting the sub-directory tree and directly executes the statistical subtask, generating the completion information of the statistical subtask and sending it to the first target node.

[0042] Specifically, in this embodiment, the sub-directory tree is recursively split to form a multi-level task structure, further refining the task granularity, enabling complex directories to be processed hierarchically and dispersed, improving the parallelism and flexibility of processing, enhancing the system's ability to handle large-scale directory statistical tasks, and preventing single-point overload.

[0043] In one embodiment, after the first target node receives the directory statistical task, it further includes: Based on a preset bottom-up traversal strategy, a traversal stack and a traversal queue are set up and combined to obtain a traversal channel; In response to the first target node not splitting the task directory tree, the first target node processes the task directory tree through the traversal channel, completes the traversal of the leaf nodes of the task directory tree, generates directory update information accordingly, and sends it to the assignment node; In response to the first target node having split the task directory tree, the second target node processes the sub-directory tree through the traversal channel, generates sub-directory update information, and sends it to the first target node; The first target node processes the remaining sub-directory tree through the traversal channel to generate remaining sub-directory update information, and combines the sub-directory update information to generate directory update information.

[0044] Specifically, in this embodiment, through a preset bottom-up traversal strategy, the traversal stack and the traversal queue are flexibly combined to form a traversal channel, realizing efficient traversal of the task directory tree and its split sub-directory trees and generating change information. This not only ensures the complete statistics of unsplit directories but also supports hierarchical collaborative processing of split directories, thereby improving the concurrent execution efficiency of directory statistics and the accuracy of data synchronization, and enhancing the scalability and response ability of the system.

[0045] In one embodiment, based on a preset bottom-up traversal strategy, setting up a traversal stack and a traversal queue and combining them to obtain a traversal channel includes: Set up a traversal stack, perform a depth-first traversal on the received directory tree to be traversed, push the directories corresponding to the directory tree to be traversed into the traversal stack in sequence, and gradually identify a number of leaf nodes; Compare the directories corresponding to several leaf nodes with the corresponding directory information in the underlying storage and the directory information stored in the database to determine whether there are structural differences or attribute changes, and determine several changed leaf nodes; Set up a traversal queue, add several changed leaf nodes to the traversal queue in sequence, and process them in the order of first in first out to generate corresponding directory change information; Perform upward merging on the directory change information to generate directory update information corresponding to the directory tree to be traversed.

[0046] Specifically, in this embodiment, by combining a depth-first traversal stack and a first-in first-out traversal queue, the leaf nodes in the directory tree that have changed are accurately identified, and the change information is merged layer by layer to generate complete directory update data, significantly improving the accuracy and processing efficiency of directory change detection, ensuring real-time synchronization and consistency of the directory status, and being applicable to the efficient management of large-scale complex directories.

[0047] In one embodiment, as Figure 3 shown, the first target node processes the task directory tree through the traversal channel to complete the traversal of the leaf nodes of the task directory tree, including: In response to the traversal stack receiving the task directory tree, after pushing the root directory of the task directory tree onto the stack, the directories at the top of the stack are popped out in sequence and used as the current popped directory; Obtain the information of the current popped directory in the underlying storage and the database respectively, compare them, and perform corresponding processing according to the comparison results, including: in response to both the underlying storage and the database storing the information of the current popped directory, determine whether their modification times are the same. If they are the same, obtain the first-level subdirectories of the current popped directory from the database and push them onto the stack. If they are different, obtain the first-level subdirectories of the current popped directory from the underlying storage and push them onto the stack; in response to the information of the current popped directory not existing in the underlying storage but existing in the database, delete the current popped directory and all its corresponding subdirectories in the database; in response to the information of the current popped directory existing in the underlying storage but not existing in the database, obtain the first-level subdirectories of the current popped directory from the underlying storage and push them onto the stack; Judge whether the current popped directory is a leaf node of the task directory tree and whether it has changed, and perform enqueue operations according to the judgment results, including: if it is a leaf node and has changed, enqueue the current popped directory; if it is not a leaf node and all the subnodes under the current popped directory have been enqueued, judge whether the current popped directory and its subnodes have changed. If they have changed, enqueue the current popped directory. If neither has changed, the current popped directory is not enqueued. Among them, any one of the following conditions can be considered that the current directory or node has changed: existing in the underlying storage but not existing in the database, not existing in the underlying storage but existing in the database, both existing but with different modification times, inconsistent directory summary information; Successively dequeue the current pop-up directory at the head of the queue as the current dequeued directory. In response to the current dequeued directory being a leaf node, perform corresponding addition, deletion, and modification operations based on the records in the database and the underlying storage, including: when the information of the current dequeued directory does not exist in the database but exists in the underlying storage, perform an insertion operation; when the information of the current dequeued directory exists in the database and is inconsistent with the information in the underlying storage, perform an update operation; when the information of the current dequeued directory does not exist in the underlying storage, perform a deletion operation. In response to the current dequeued directory information being a non-leaf node, perform a directory size merging operation, and determine whether the merging result is consistent with the directory size stored in the database. If so, do not update; otherwise, update the directory size stored in the database based on the merging result.

[0048] Specifically, in this embodiment, by comparing the information of the underlying storage and the database directory, the dynamic update and synchronization of the directory are realized, ensuring the consistency and integrity of the data between the database and the underlying storage, reflecting the directory changes in a timely manner, and improving the accuracy and data reliability of the system.

[0049] In one embodiment, after generating the corresponding directory statistics task based on the independent directory tree and allocating it to the first target node, it further includes: In response to the allocation node not receiving the task completion information feedback from the first target node within the preset period or the allocation node receiving the fault information feedback from the first target node, determine that the status of the first target node is abnormal, and re-allocate the corresponding directory statistics task to other idle microservice nodes for execution.

[0050] Specifically, in this embodiment, through the real-time monitoring of the task execution status and the abnormal determination, the rapid recovery and re-scheduling of the failed tasks are realized, improving the fault tolerance and stability of the system, avoiding the long-term blocking of tasks, and ensuring the continuity and reliability of the directory statistics work.

[0051] In one embodiment, as Figure 5 shown, re-allocating the corresponding directory statistics task to other idle microservice nodes for execution further includes: Any microservice node is registered when it starts, and its initial running status is marked as idle. When it receives the allocated directory statistics task as the first target node, its own running status is marked as busy; When this microservice node discovers that there are allocable microservice nodes in the cluster, it allocates the corresponding statistical subtasks to the allocable microservice nodes, modifies their status to busy, and records the corresponding node information; When this microservice node discovers that there are microservice nodes that are continuously in a busy state in the cluster, after preferentially completing the tasks it is running, it receives the tasks allocated to the continuously busy microservice nodes and continues to execute them, and marks their status as abnormal; After a microservice node in an abnormal state is repaired and returns to the normal state, it automatically marks its status as idle.

[0052] Specifically, in this embodiment, through the dynamic marking of node status and the abnormal repair mechanism, the healthy operation of nodes in the cluster and the continuous execution of tasks are effectively guaranteed, the impact of node failures on the overall system performance is reduced, and the reliability and resource utilization efficiency of the cluster are improved.

[0053] In one embodiment, as Figure 8 shown, the microservice nodes in the cluster communicate with each other via JSON-RPC to obtain the status and feedback information of each microservice node in real time; the microservice nodes in the cluster receive the tasks sent by the allocation node according to the database and execute them, generating directory update information and storing it in their respective microservice node databases; a summary synchronization database is set up to merge the directory update information stored in each microservice node database, and at the same time merge the directory sizes. After the merging is completed, a summary is fed back to the database and synchronized to each microservice node database.

[0054] Specifically, this embodiment realizes efficient information exchange and status synchronization between nodes by adopting the JSON-RPC communication mechanism, and combines the summary synchronization database for centralized management, improving the information consistency, coordination and overall management efficiency of the system, and realizing the efficient collaborative work of the distributed system.

[0055] Through the description of the above implementation manners, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation manner.

[0056] The embodiment of the present application also provides a directory management device, as Figure 9 shown, including: a directory tree construction and division module, an idle node identification module, a task allocation module, a task splitting and statistics module, and a directory update module, where The directory tree construction and division module is used to, in response to the allocation node receiving a directory update task, construct a directory tree based on the directory information stored in the database, and divide the directory tree according to path independence to obtain multiple independent directory trees; The idle node identification module is used to obtain the current running status of the microservice nodes in the cluster and identify the idle microservice nodes as the first target nodes; The task allocation module is used to generate corresponding directory statistics tasks based on the independent directory trees and allocate them to the first target nodes; The task splitting and statistics module is used to, in response to the first target node receiving a directory statistics task, determine whether the corresponding task directory tree needs to be split. If so, extract sub-directory trees from the task directory tree and generate corresponding statistical sub-tasks, select a second target node from the cluster microservice nodes to receive and execute the statistical sub-tasks. If not, execute the directory statistics task itself; The directory update module is used to, in response to the assigned node receiving the task completion information fed back by the first target node, parse the task completion information, obtain directory update information and write it into the database to update the directory information.

[0057] The directory tree construction and division module is also used to identify a set of independent directories in the directory tree where multiple root paths do not contain each other, and respectively form multiple independent directory trees.

[0058] The idle node identification module is also used to, based on a preset evaluation dimension, obtain and parse the operation information of the cluster microservice nodes to get node load evaluation parameters; according to the node load evaluation parameters, perform weighted calculation in combination with the corresponding preset node load weights to obtain a node load evaluation value; in response to the node load evaluation value of one or more microservice nodes being less than or equal to the first load threshold, determine that the microservice node is idle and set it as the first target node; in response to the node load evaluation value of one or more microservice nodes being greater than the first load threshold, determine that the microservice node is busy.

[0059] The task assignment module is also used to respectively construct task description information according to multiple independent directory trees, generate corresponding multiple directory statistics tasks according to the task description information; respectively obtain the number of independent directory trees and the number of first target nodes, and determine whether the independent directory trees are more than the first target nodes; in response to no, assign less than or equal to one directory statistics task to the first target node; in response to yes, divide the multiple directory statistics tasks into a first task group with a number less than or equal to the first target node, and assign the first task group to the first target node.

[0060] The task assignment module is also used to parse the task description information of multiple directory statistics tasks to obtain the directory size, directory level depth, and directory last modification time; according to the directory size, directory level depth, and directory last modification time, perform weighted summation in combination with the preset load coefficient, complexity coefficient, and urgency coefficient to obtain the load estimates of multiple directory statistics tasks; initially arrange and group the multiple directory statistics tasks to obtain multiple initial task groups; perform in-group task migration among the multiple initial task groups to make the deviation of the total load estimates of the multiple initial task groups less than or equal to the preset deviation threshold, and generate multiple first task groups.

[0061] The task allocation module is further configured to determine that the status of the first target node is abnormal if the allocation node does not receive the task completion information feedback by the first target node within a preset period or the allocation node receives the fault information feedback by the first target node, and re-allocate the corresponding directory statistics task to other idle microservice nodes for execution.

[0062] The task splitting and statistics module is further configured to parse the task description information of the directory statistics task to obtain the maximum hierarchical depth of the corresponding task directory tree; set a splitting threshold based on the number of first target nodes and the average hierarchy of multiple independent directory trees; determine that the task directory tree needs to be split if the maximum hierarchical depth of the task directory tree is greater than the splitting threshold; and determine that the task directory tree does not need to be split if the maximum hierarchical depth of the task directory tree is less than or equal to the splitting threshold.

[0063] The task splitting and statistics module is further configured to obtain the number of first target nodes and the average hierarchy of multiple independent directory trees, and dynamically calculate the splitting threshold in combination with a preset adjustment coefficient.

[0064] The task splitting and statistics module is further configured to determine, for the first target node, whether there are currently microservice nodes with an idle status or first target nodes that have not been assigned directory statistics tasks in the cluster microservice nodes as assignable microservice nodes; if so, the first target node sets the assignable microservice nodes as second target nodes, traverses one or more levels of sub-paths under the task directory tree, and filters out non-empty sub-directories that can independently form a tree structure as candidate sub-directories; the first target node performs priority evaluation and sorting on the candidate sub-directories, selects the sub-directories with higher rankings based on the number of second target nodes as split sub-directories, constructs a sub-directory tree and generates corresponding statistical sub-tasks to be assigned to the second target nodes for execution; the first target node deletes the path corresponding to the split sub-directory from the task directory tree, obtains a remaining sub-directory tree and constructs a corresponding remaining statistical sub-task to be executed by the first target node; if the first target node receives the completion information of multiple statistical sub-tasks, it obtains the completion information of the remaining statistical sub-task, generates task completion information and sends it to the allocation node; if not, the first target node suspends splitting the task directory tree, directly executes the directory statistics task, generates task completion information and sends it to the allocation node.

[0065] The task splitting and statistics module is further configured to, in response to the second target node receiving a statistics subtask, determine again whether there are assignable microservice nodes in the cluster microservice nodes currently; if so, the second target node uses the assignable microservice nodes as the third target nodes, parses the statistics subtask to obtain a subdirectory tree, splits the subdirectory tree again, generates corresponding statistics grandchild tasks and sends them to the third target nodes for execution; the second target node executes the remaining statistics grandchild tasks corresponding to the subdirectory tree splitting; in response to the second target node receiving the completion information of multiple statistics grandchild tasks, it obtains the completion information of the remaining statistics grandchild tasks, generates the completion information of the statistics subtask and sends it to the first target node; the third target nodes repeatedly determine whether there are assignable microservice nodes in the cluster microservice nodes currently and repeat the corresponding execution steps until the judgment result is no or the maximum level value of the subdirectory tree is less than a preset minimum splitting threshold; in response to no or the maximum level value of the subdirectory tree being less than the preset minimum splitting threshold, the second target node suspends splitting the subdirectory tree and directly executes the statistics subtask, generates the completion information of the statistics subtask and sends it to the first target node.

[0066] The task splitting and statistics module is further configured to set a traversal stack and a traversal queue based on a preset bottom-up traversal strategy and combine them to obtain a traversal channel; in response to the first target node not splitting the task directory tree, the first target node processes the task directory tree through the traversal channel, completes the traversal of the leaf nodes of the task directory tree, generates corresponding directory update information and sends it to the assignment node; in response to the first target node having split the task directory tree, the second target node processes the subdirectory tree through the traversal channel, generates subdirectory update information and sends it to the first target node; the first target node processes the remaining subdirectory tree through the traversal channel to generate remaining subdirectory update information, and combines the subdirectory update information to generate directory update information.

[0067] The task splitting and statistics module also sets a traversal stack, performs a depth-first traversal on the received directory tree to be traversed, sequentially pushes the directories corresponding to the directory tree to be traversed onto the traversal stack, and gradually identifies several leaf nodes; compares the directories corresponding to the several leaf nodes with the corresponding directory information in the underlying storage and the directory information stored in the database to determine whether there are structural differences or attribute changes, and determines several changed leaf nodes; sets a traversal queue, sequentially adds the several changed leaf nodes to the traversal queue, and processes them in the order of first in first out, generating corresponding directory change information; performs upward merging on the directory change information to generate directory update information corresponding to the directory tree to be traversed.

[0068] For the description of the features in the corresponding embodiments of the directory management device, reference can be made to the relevant descriptions in the corresponding embodiments of the directory management method, which will not be elaborated here one by one.

[0069] Embodiments of the present application also provide an electronic device, such as Figure 10 shown, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-described embodiments of the directory management method.

[0070] Embodiments of the present application also provide a computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the directory management method when running.

[0071] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disc that can store a computer program.

[0072] Embodiments of the present application also provide a computer program product, the computer program product including a computer program, and the steps in any of the above-described embodiments of the directory management method are implemented when the computer program is executed by a processor.

[0073] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, and the steps in any of the above-described embodiments of the directory management method are implemented when the computer program is executed by a processor.

[0074] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0075] The above has introduced in detail a directory management method, device, equipment, storage medium, and product provided by the present application. Specific examples are used herein to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A directory management method, characterized in that, Including: In response to the assignment node receiving a directory update task, construct a directory tree based on the directory information stored in the database, and divide the directory tree according to path independence to obtain multiple independent directory trees; Obtain the current running status of the cluster microservice nodes, and identify the idle microservice nodes as the first target nodes; Generate corresponding directory statistics tasks based on the independent directory trees and assign them to the first target nodes; In response to the first target node receiving a directory statistics task, determine whether the corresponding task directory tree needs to be split. If so, extract sub-directory trees from the task directory tree and generate corresponding statistical subtasks, and select second target nodes from the cluster microservice nodes to receive and execute the statistical subtasks. If not, execute the directory statistics task itself; In response to the assignment node receiving the task completion information fed back by the first target node, parse the task completion information to obtain directory update information and write it into the database to update the directory information.

2. The directory management method according to claim 1, wherein The dividing the directory tree according to path independence to obtain multiple independent directory trees includes: Identify independent directory sets in the directory tree where multiple root paths do not contain each other, and respectively form multiple independent directory trees.

3. The directory management method according to claim 1, wherein The obtaining the current running status of the cluster microservice nodes and identifying the idle microservice nodes as the first target nodes includes: Based on preset evaluation dimensions, obtain and parse the running information of the cluster microservice nodes to obtain node load evaluation parameters; According to the node load evaluation parameters, perform weighted calculation in combination with the corresponding preset node load weights to obtain a node load evaluation value; In response to the node load evaluation value of one or more microservice nodes being less than or equal to the first load threshold, determine that the microservice node is idle and set it as the first target node; In response to the node load evaluation value of one or more microservice nodes being greater than the first load threshold, determine that the microservice node is busy.

4. The directory management method according to claim 1, wherein The generating corresponding directory statistics tasks based on the independent directory trees and assigning them to the first target nodes includes: Construct task description information according to multiple independent directory trees respectively, and generate corresponding multiple directory statistics tasks according to the task description information; Obtain the number of independent directory trees and the number of first target nodes respectively, and determine whether the independent directory trees are more than the first target nodes; In response to no, assign less than or equal to one directory statistics task to the first target node; In response to yes, divide multiple directory statistics tasks into a first task group with a quantity less than or equal to the first target nodes, and assign the first task group to the first target nodes.

5. The directory management method according to claim 4, wherein The dividing multiple directory statistics tasks into multiple first task groups and assigning the first task group to the first target nodes includes: Parse the task description information of multiple directory statistics tasks to obtain directory size, directory level depth, and directory last modification time; Based on the directory size, the directory hierarchy depth, and the directory's most recent modification time, combined with preset load factor, complexity factor, and emergency factor, perform weighted summation to obtain the load estimations of multiple directory statistics tasks; Perform initial arrangement and grouping on multiple directory statistics tasks to obtain multiple initial task groups; Perform intra-group task migration among multiple initial task groups to make the deviation of the total load estimations of multiple initial task groups less than or equal to a preset deviation threshold, and generate multiple first task groups.

6. The directory management method according to claim 4, wherein Responding to the first target node receiving a directory statistics task, determining whether the corresponding task directory tree needs to be split includes: Parse the task description information of the directory statistics task to obtain the maximum hierarchy depth of the corresponding task directory tree; Based on the number of first target nodes and the average hierarchy of multiple independent directory trees, set a split threshold; Responding to the maximum hierarchy depth of the task directory tree being greater than the split threshold, determine that the task directory tree needs to be split; Responding to the maximum hierarchy depth of the task directory tree being less than or equal to the split threshold, determine that the task directory tree does not need to be split.

7. The directory management method according to claim 6, characterized in that, The setting of the split threshold based on the number of first target nodes and the average hierarchy of multiple independent directory trees includes: Obtain the number of first target nodes and the average hierarchy of multiple independent directory trees, and dynamically calculate the split threshold through the following formula in combination with a preset adjustment coefficient: T splkit = L avg + α ·log ( N node + 1), where T splkit represents the splitting threshold, L avg represents the average level of multiple independent directory trees, α represents a preset adjustment coefficient, N node represents the number of first target nodes.

8. The directory management method according to claim 1, characterized in that If so, extract a sub-directory tree from the task directory tree and generate a corresponding statistical sub-task, and select a second target node from the cluster microservice nodes to receive and execute the statistical sub-task, including: The first target node determines whether there is currently a microservice node with an idle current state or a first target node that has not been assigned a directory statistics task in the cluster microservice nodes as an assignable microservice node; Responding yes, the first target node sets the assignable microservice node as the second target node, traverses one or more levels of sub-paths under the task directory tree, and filters out non-empty sub-directories that can independently form a tree structure as candidate sub-directories; The first target node performs extraction priority evaluation and sorting on the candidate sub-directories, selects the sub-directories with higher rankings based on the number of second target nodes as split sub-directories, constructs the sub-directory tree and generates corresponding statistical sub-tasks to be assigned to the second target node for execution; The first target node deletes the path corresponding to the split sub-directory from the task directory tree, obtains a remaining sub-directory tree and constructs a corresponding remaining statistical task to be executed by the first target node; Responding to the first target node receiving the completion information of multiple statistical sub-tasks, obtain the completion information of the remaining statistical task, generate the task completion information and send it to the assignment node; Responding no, the first target node pauses splitting the task directory tree, directly executes the directory statistics task, generates the task completion information and sends it to the assignment node.

9. The directory management method according to claim 8, wherein Building the sub-directory tree and generating corresponding statistical subtasks for execution by the second target node includes: In response to the second target node receiving the statistical subtask, it is determined again whether there are assignable microservice nodes in the cluster microservice nodes currently; If so, the second target node uses the assignable microservice node as the third target node, parses the statistical subtask to obtain the sub-directory tree, splits the sub-directory tree again, generates corresponding statistical grandchild tasks and sends them to the third target node for execution; The second target node executes the remaining statistical grandchild tasks corresponding to the split of the sub-directory tree; In response to the second target node receiving the completion information of multiple statistical grandchild tasks, it obtains the completion information of the remaining statistical grandchild tasks, generates the completion information of the statistical subtask and sends it to the first target node; The third target node repeatedly determines whether there are assignable microservice nodes in the cluster microservice nodes currently and repeats the corresponding execution steps until the judgment result is no or the maximum level value of the sub-directory tree is less than the preset minimum split threshold; If the judgment result is no or the maximum level value of the sub-directory tree is less than the preset minimum split threshold, the second target node pauses splitting the sub-directory tree, directly executes the statistical subtask, generates the completion information of the statistical subtask and sends it to the first target node.

10. The directory management method according to claim 8, wherein After the first target node receives the directory statistics task, it further includes: Based on a preset bottom-up traversal strategy, a traversal stack and a traversal queue are set and combined to obtain a traversal channel; If the first target node does not split the task directory tree, the first target node processes the task directory tree through the traversal channel, completes the traversal of the leaf nodes of the task directory tree, generates the corresponding directory update information and sends it to the allocation node; If the first target node has split the task directory tree, the second target node processes the sub-directory tree through the traversal channel, generates sub-directory update information and sends it to the first target node; The first target node processes the remaining sub-directory tree through the traversal channel to generate remaining sub-directory update information, and combines the sub-directory update information to generate the directory update information.

11. The directory management method according to claim 10, characterized in that, The setting of the traversal stack and the traversal queue and the combination to obtain the traversal channel based on the preset bottom-up traversal strategy includes: Setting the traversal stack, performing a depth-first traversal on the received directory tree to be traversed, pushing the directories corresponding to the directory tree to be traversed into the traversal stack in sequence, and gradually identifying a number of leaf nodes; Comparing the directories corresponding to the number of leaf nodes with the corresponding directory information in the underlying storage and the directory information stored in the database to determine whether there are structural differences or attribute changes, and determining a number of changed leaf nodes; Setting the traversal queue, adding the number of changed leaf nodes to the traversal queue in sequence, and processing them in the order of first in first out to generate corresponding directory change information; Merge the directory change information upward to generate directory update information corresponding to the directory tree to be traversed.

12. The directory management method according to claim 1, wherein After generating a corresponding directory statistics task based on the independent directory tree and assigning it to the first target node, it further includes: If the assigned node does not receive the task completion information feedback from the first target node within a preset period or the assigned node receives the fault information feedback from the first target node, then determine that the status of the first target node is abnormal, and reassign the corresponding directory statistics task to other idle microservice nodes for execution.

13. A directory management device, characterized in that, It includes: A directory tree construction and division module, which is used to, in response to the assigned node receiving a directory update task, construct a directory tree based on the directory information stored in the database, and divide the directory tree according to path independence to obtain multiple independent directory trees; An idle node identification module, which is used to obtain the current running status of the cluster microservice nodes and identify the idle microservice nodes as the first target nodes; A task assignment module, which is used to generate a corresponding directory statistics task based on the independent directory tree and assign it to the first target node; A task splitting and statistics module, which is used to, in response to the first target node receiving a directory statistics task, determine whether the corresponding task directory tree needs to be split. If so, extract sub-directory trees from the task directory tree and generate corresponding statistical subtasks, select second target nodes from the cluster microservice nodes to receive and execute the statistical subtasks, and if not, execute the directory statistics task by itself; A directory update module, which is used to, in response to the assigned node receiving the task completion information feedback from the first target node, parse the task completion information to obtain directory update information and write it into the database to update the directory information.

14. An electronic device, characterized in that, It includes: A memory, which is used to store computer programs; A processor, which is used to implement the steps of the directory management method according to any one of claims 1 to 12 when executing the computer programs.

15. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of the directory management method according to any one of claims 1 to 12 when executed by a processor.

Images