Directory statistical method, product, equipment and medium

Abstract

The invention discloses a directory statistical method, a product, equipment and a medium, and relates to the technical field of distributed storage. The invention provides a directory statistics scheme based on a distributed management system. Specifically, when a statistical task for the directory size of the distributed cluster is received, the statistical task is divided into a plurality of statistical sub-tasks according to the number of main nodes; processing the corresponding statistical sub-tasks through the main nodes to obtain a plurality of statistical sub-results; and finally, adding the statistical sub-results to obtain a statistical result of the size of the directory. That is to say, the scheme splits the directory size statistical task in advance, and uses the computing resources of the multiple nodes in the distributed management system to process the divided statistical sub-tasks at the same time, thereby achieving the purpose of quickly completing the directory statistical task, reducing the overall utilization rate of resources occupied by the distributed cluster service, and improving the user experience. The method is very suitable for a business scene of distributed cluster large-scale storage management, and the use experience of the distributed cluster is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a directory statistics method, product, device and medium. Background Art

[0002] With the rapid development of the artificial intelligence (AI) industry, the construction of AI cluster platforms has solved users' high demand for computing power. However, AI cluster storage management faces challenges, especially when processing directory size statistics. Since the directory contains massive data, frequent file operations consume a lot of storage resources, affecting business functions and storage performance.

[0003] At present, mainstream AI platform manufacturers use a simple traversal method to manage the massive files in the directory, which consumes a lot of storage input and output, processor and memory resources, and the directory statistics speed is extremely slow in the case of massive files. Another method is to use the quota management (Quota) function of the storage system. Although the statistics are fast and accurate, it cannot count the size of the directory level, does not support multi-storage environments, cannot count the size of public directories and data set directories, and may also reduce storage performance.

[0004] In view of the above, how to solve the current directory size statistics method has high resource consumption, slow statistics speed, inability to count the size of all directories, and is not applicable in large-scale cluster storage usage scenarios, is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] The purpose of the present invention is to provide a directory statistics method, product, device and medium to solve the problems that the current directory size statistics method has high resource consumption, slow statistics speed, cannot count the size of all directories, and is not applicable in large-scale cluster storage usage scenarios.

[0006] In order to solve the above technical problems, the present invention provides a directory statistics method, which is applied to a distributed management system; wherein the distributed management system includes multiple master nodes; the method comprises:

[0007] When receiving a statistical task for the directory size of a distributed cluster, the statistical task is divided into multiple statistical subtasks according to the number of master nodes;

[0008] Each master node processes the corresponding statistical subtask to obtain multiple statistical sub-results;

[0009] Add up the statistical sub-results to get the directory size statistics.

[0010] On the one hand, before receiving the statistical task, it also includes:

[0011] Set up corresponding file management services and file management service agents for each master node respectively;

[0012] Initialize the state of each file management service agent to an idle state;

[0013] Set the operation mode of each file management service agent to load balancing mode.

[0014] On the other hand, corresponding file management services and file management service agents are set for each master node, including:

[0015] Determine the number of master nodes;

[0016] Set the replica number field according to the number of master nodes to set corresponding replicas for each master node.

[0017] Among them, the copies include file management service copies and file management service proxy copies.

[0018] On the other hand, the statistical task is divided into multiple statistical subtasks according to the number of master nodes, including:

[0019] Determine the directory tree whose size is to be counted according to the statistical task;

[0020] Determine a first master node that has received the statistical task, and determine a second master node that has an idle state for the corresponding file management service agent;

[0021] The first master node divides the directory tree into a plurality of sub-directory trees according to the number of the second master nodes, so as to generate a plurality of statistical sub-tasks representing the sizes of the corresponding sub-directory trees.

[0022] On the other hand, each master node processes the corresponding statistical subtasks, including:

[0023] Send each statistical subtask to the corresponding second master node through the first master node;

[0024] The state of the corresponding file management service agent is changed from an idle state to a busy state through the second master node, and the size of the subdirectory tree in the corresponding statistical subtask is counted through the file management service agent to obtain a corresponding statistical subresult;

[0025] Sending the corresponding statistical sub-results to the first master node through the second master node;

[0026] The state of the corresponding file management service agent is changed from a busy state to an idle state through the second master node.

[0027] On the other hand, after the corresponding statistical sub-result is sent to the first master node through the second master node, the method further includes:

[0028] Determine through the first master node whether there is any unfinished statistical subtask and whether there is a second master node whose corresponding file management service agent is in an idle state;

[0029] If yes, the unfinished statistical subtask is split again through the first master node to obtain a new statistical subtask;

[0030] The new statistical subtask is sent through the first main node to the second main node whose corresponding file management service agent is in an idle state, so as to process the new statistical subtask.

[0031] On the other hand, after sending each statistical subtask to the corresponding second main node through the first main node, it also includes:

[0032] Monitoring the status of the file management service agent of each second master node through the first master node, and determining whether there is a file management service agent of the second master node that is continuously busy;

[0033] If so, the second master node whose corresponding file management service agent is continuously in a busy state is determined as a faulty master node;

[0034] Obtain the statistical subtask corresponding to the faulty master node through the first master node, and reallocate the statistical subtask corresponding to the faulty master node to the remaining second master nodes that are in an idle state;

[0035] Change the status of the file management service agent of the failed master node from busy to failed.

[0036] On the other hand, after the state of the file management service agent of the failed master node is changed from a busy state to a failed state, the following is further included:

[0037] Control the restart of the faulty master node and determine whether the faulty master node has resumed normal operation;

[0038] If it is confirmed that the failed master node has resumed normal operation, the state of the file management service agent of the failed master node is changed from a failed state to an idle state.

[0039] On the other hand, the first master node sends each statistical subtask to the corresponding second master node, including:

[0040] The first master node sends each statistical subtask to the corresponding second master node through a preset remote procedure call method;

[0041] The data structure of the statistical subtask consists of an identifier number, a file management service agent name, a node where the file management service agent is located, a file management service agent status, an allocated storage directory statistical path, an allocated statistical task node, and a last modified time.

[0042] On the other hand, it also includes:

[0043] When receiving a video transcoding task, dividing the video transcoding task into a plurality of video transcoding subtasks;

[0044] Each master node processes the corresponding video transcoding subtask.

[0045] On the other hand, the video transcoding task is divided into multiple video transcoding subtasks, including:

[0046] Determine a third master node whose computing resource amount is greater than a threshold value among all the master nodes, and determine the number of the third master nodes;

[0047] Dividing the video transcoding task into a plurality of video transcoding subtasks according to the number of the third master nodes; wherein each video transcoding subtask corresponds to each third master node one by one;

[0048] Correspondingly, each master node processes the corresponding video transcoding subtask, including:

[0049] The corresponding video transcoding subtask is processed by each third master node.

[0050] On the other hand, the statistical task is a task that represents the size of the directories that have changed in each directory of the distributed cluster.

[0051] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above directory statistics method when executed by a processor.

[0052] In order to solve the above technical problems, the present invention further provides a directory statistics device, comprising:

[0053] Memory for storing computer programs;

[0054] A processor is used to implement the steps of the above-mentioned directory statistics method when executing a computer program.

[0055] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above directory statistics method are implemented.

[0056] The directory statistics method provided by the present invention is applied to a distributed management system; wherein the distributed management system includes multiple master nodes; specifically, when a statistical task for the directory size of a distributed cluster is received, the statistical task is divided into multiple statistical subtasks according to the number of master nodes; the corresponding statistical subtasks are processed by each master node to obtain multiple statistical subresults; and each statistical subresult is added to obtain a statistical result of the directory size.

[0057] The beneficial effect of the present invention is that a directory statistics scheme based on a distributed management system is proposed; specifically, when a statistical task for the directory size of a distributed cluster is received, the statistical task is divided into multiple statistical subtasks according to the number of master nodes; the corresponding statistical subtasks are processed by each master node to obtain multiple statistical subresults; finally, the statistical subresults are added to obtain the statistical result of the directory size. In other words, the scheme pre-splits the directory size statistical task, and uses the computing resources of multiple nodes in the distributed management system to simultaneously process the divided multiple statistical subtasks, thereby achieving the purpose of quickly completing the directory statistical task, reducing the overall utilization rate of resources occupied by distributed cluster services, and is very suitable for business scenarios of large-scale storage management of distributed clusters, and enhancing the user experience of distributed clusters.

[0058] On the other hand, the present invention specifically sets corresponding file management services and file management service agents for each master node respectively; initializes the state of each file management service agent to an idle state; sets the operation mode of each file management service agent to a load balancing mode, thereby ensuring that the storage auxiliary module based on K8s can perform directory size statistics on a distributed cluster. Determine the directory tree whose size is to be counted according to the statistical task; determine the first master node that receives the statistical task, and determine the second master node whose corresponding file management service agent is in an idle state; the first master node divides the directory tree into multiple sub-directory trees according to the number of second master nodes to generate multiple statistical sub-tasks that characterize the size of the corresponding sub-directory trees, thereby realizing the division of statistical sub-tasks. By handing over the divided sub-directory trees to different idle file management service agents for distributed statistics, a parallel statistical effect is achieved. At the same time, it is ensured that the statistical sub-tasks of all file management service agents end at the same time, and frequent task allocation will not consume time and network performance.

[0059] In addition, the present invention also provides a computer program product, a directory statistics device and a storage medium, the effects of which are the same as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 A flowchart of a directory statistics method provided by an embodiment of the present invention;

[0062] Figure 2A schematic diagram of a K8s cluster storage auxiliary module provided in an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of an independent directory tree and a sub-directory tree provided in an embodiment of the present invention;

[0064] Figure 4 A schematic diagram of a directory statistics device provided by an embodiment of the present invention;

[0065] Figure 5 A schematic diagram of a directory statistics device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0067] The core of the present invention is to provide a directory statistics method, product, device and medium to solve the problems that the current directory size statistics method has high resource consumption, slow statistics speed, cannot count the size of all directories, and is not applicable in large-scale cluster storage usage scenarios.

[0068] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0069] At present, mainstream AI platform manufacturers use simple traversal methods to manage massive files in directories, which consumes a lot of storage input and output, processor and memory resources, and the directory statistics speed is extremely slow in the case of massive files. Another method is to use the Quota function of the storage system. Although the statistics are fast and accurate, it cannot count the directory level size, does not support multiple storage environments, cannot count the size of public directories and data set directories, and may also reduce storage performance. Therefore, in order to solve the above problems, the present invention provides a directory statistics method.

[0070] It should be noted that the method provided by the present invention is applied to a distributed management system, specifically a Kubernetes (K8s) cluster. K8s cluster is a distributed system for automatically deploying, expanding and managing containerized applications, and is composed of a control node (Master node, also known as the main node) and a worker node (Node node). The control node is responsible for the management and control of the cluster, including scheduling containers, monitoring status and processing user requests, and runs core components such as the API Server, Scheduler, Controller Manager and distributed key-value storage (etcd); the worker node is responsible for running containerized applications and providing computing resources, and runs the node agent kubelet and container runtime. The K8s cluster has functions such as automated deployment and expansion, load balancing, fault recovery, storage management and security management. Through these functions, it can improve the deployment efficiency, scalability and reliability of applications, while reducing operation and maintenance costs and complexity. This solution is mainly based on the characteristics of K8s cluster management microservice containers, facing the large-scale storage of AI clusters, and realizing a storage directory management and fast statistical operation method to achieve the purpose of real-time acquisition of storage directory information. The technical solution of the present invention is described in detail below in conjunction with specific embodiments:

[0071] Figure 1 The following is a flow chart of a directory statistics method provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0072] S10: when receiving a statistical task for the directory size of a distributed cluster, dividing the statistical task into multiple statistical subtasks according to the number of master nodes;

[0073] File management in the K8s cluster includes a variety of file operations, such as file (folder) upload, file (folder) copy, file (folder) deletion, file editing, file (folder) renaming, file (folder) retrieval, and file decompression, etc. These conventional file operations consume less processor resources and take a short time to execute. These operations are usually placed in a storage module and an application programming interface (Application Programming Interface, API) is provided to support the visualization operation of the AI ​​platform. However, for storage directory size statistics, due to the high processor resource consumption and very long duration, it is necessary to start a separate storage auxiliary module to complete this work; at the same time, in order to speed up the completion of storage directory statistics and support more video transcoding, the storage auxiliary module needs to support distributed operation based on K8s, thereby improving the efficiency of storage size statistics. Therefore, in this embodiment, the directory size statistics of the distributed cluster are specifically performed by the storage auxiliary module based on K8s.

[0074] Specifically, a statistical task for the directory size of a distributed cluster is received. It can be understood that the statistical task is specifically a statistical task for the size of a shared storage directory of a distributed cluster, and the size of one or more shared storage directories can be counted, depending on the specific implementation situation. Furthermore, since the K8s system contains multiple master nodes, in order to be able to use multiple master nodes to process statistical tasks in parallel, it is necessary to divide the statistical task into multiple statistical subtasks according to the number of master nodes.

[0075] It should be noted that the specific division method of statistical subtasks is not restricted in this embodiment. For example, the statistical subtasks can be divided into the same number as the master nodes, so that each master node corresponds to each statistical subtask one by one. The number of statistical subtasks can also be made smaller than the number of master nodes, and the master nodes with higher computing resources can handle the statistical subtasks, depending on the specific implementation situation.

[0076] S11: Processing corresponding statistical subtasks through each master node to obtain multiple statistical sub-results;

[0077] Further, after obtaining multiple statistical subtasks, the corresponding statistical subtasks are processed by each master node to obtain multiple statistical subresults. It can be understood that since the statistical task is a statistical task for the size of the shared storage directory of the distributed cluster, the statistical subtask is a statistical task for the size of a partial directory of the shared storage of the distributed cluster, and the corresponding statistical subresult is specifically the size of a partial directory of the shared storage of the distributed cluster.

[0078] It should be noted that in this embodiment, there is no restriction on the specific process of each master node processing the corresponding statistical subtask, which depends on the specific implementation situation.

[0079] S12: Add up the statistical sub-results to obtain the statistical result of the directory size.

[0080] Finally, each statistical sub-result is added together, that is, the sizes of multiple partial directories of the shared storage of the distributed cluster are added together to obtain the statistical result of the shared storage size of the distributed cluster.

[0081] In addition, this solution is different from the directory size statistics method of directory traversal. The corresponding statistical task can be specifically the task of characterizing the size of the changed directories in the statistical distributed cluster, that is, using the incremental statistical method, only counting the changed directories, and updating all parent directories of the changed directories upward, using the stack and queue data structures, continuously pushing and popping the stack, entering and exiting the queue to complete the statistics of all changed directories, realizing a "bottom-up" statistical method, that is, updating from the leaf directory to all its parent directories, making the statistical efficiency higher.

[0082] In this embodiment, a directory statistics solution based on a distributed management system is proposed; specifically, when a statistical task for the directory size of a distributed cluster is received, the statistical task is divided into multiple statistical subtasks according to the number of master nodes; the corresponding statistical subtasks are processed by each master node to obtain multiple statistical subresults; finally, the statistical subresults are added together to obtain the statistical result of the directory size. In other words, the solution pre-splits the directory size statistical task, and uses the computing resources of multiple nodes in the distributed management system to simultaneously process the divided multiple statistical subtasks, thereby achieving the purpose of quickly completing the directory statistical task, reducing the overall utilization rate of resources occupied by distributed cluster services, and is very suitable for business scenarios of large-scale storage management of distributed clusters, and enhancing the user experience of distributed clusters.

[0083] Figure 2 A schematic diagram of a K8s cluster storage auxiliary module provided in an embodiment of the present invention. In order to ensure that the K8s-based storage auxiliary module can perform directory size statistics on a distributed cluster, based on the above embodiments, in some embodiments, before receiving a statistical task, it also includes:

[0084] S13: Setting corresponding file management services and file management service agents for each master node;

[0085] S14: Initialize the status of each file management service agent to an idle state;

[0086] S15: Setting the operation mode of each file management service agent to a load balancing mode.

[0087] like Figure 2 As shown in the figure, the corresponding file management service (istorage) and the file management service agent (istorage-agent) under the istorage are set for each master node in the K8s cluster. It is worth noting that the processing of statistical subtasks in this solution is specifically performed by the istorage-agent in the node. Furthermore, the state of the istorage-agent of each master node is initialized to an idle state, and the istorage-agent in the idle state can process statistical subtasks. Finally, the operation mode of each istorage-agent is set to the load balancing mode (LoadBalancer), so as to ensure that the pressure of each istorage-agent in the storage auxiliary module is dispersed, and different storage directory statistical tasks can be shared, so as to achieve the purpose of quickly improving the statistical efficiency of the storage module. In this way, it is ensured that the storage auxiliary module based on K8s can perform directory size statistics on distributed clusters.

[0088] In addition, in some embodiments, in order to set up corresponding file management services and file management service agents for each master node, the number of master nodes is first determined; the number of replicas field is further set according to the number of master nodes to set up corresponding replicas (pods) for each master node. It should be noted that the replicas include file management service replicas and file management service agent replicas. In this way, the setting of file management services and file management service agents for each master node is realized.

[0089] Based on the above embodiments, in some embodiments, the statistical task is divided into multiple statistical subtasks according to the number of master nodes, including:

[0090] S101: Determine the directory tree whose size is to be counted according to the statistical task;

[0091] S102: Determine a first master node that has received a statistical task, and determine a second master node that has an idle state for a corresponding file management service agent;

[0092] S103: The first master node divides the directory tree into a plurality of sub-directory trees according to the number of the second master nodes, so as to generate a plurality of statistical sub-tasks representing the size of the corresponding sub-directory trees.

[0093] In a distributed cluster, the storage directory is usually in the form of a directory tree. Therefore, in order to perform the division of statistical subtasks, in this embodiment, the directory tree whose size is to be counted is first determined according to the statistical task. It should be noted that since the statistical task is specifically a task of counting the size of one or more shared storage directories, the corresponding directory tree is also one or more.

[0094] The K8s cluster specifically receives the statistical task through one of the multiple master nodes. Therefore, in order to better divide the master nodes used to process the statistical subtasks, it is necessary to determine the first master node that receives the statistical task and determine the second master node whose corresponding istorage-agent is in an idle state, so as to process the statistical subtask through the second master node whose corresponding istorage-agent is in an idle state. It is worth noting that when the istorage-agent of the first master node is in an idle state, the first master node is also the second master node.

[0095] Finally, the first master node divides the directory tree into multiple sub-directory trees according to the number of second master nodes to generate multiple statistical sub-tasks representing the size of the corresponding sub-directory trees. It should be noted that the specific division method of the sub-directory tree is not limited in this embodiment and depends on the specific implementation situation.

[0096] In this embodiment, the directory tree whose size is to be counted is determined according to the statistical task; the first main node that receives the statistical task is determined, and the second main node whose corresponding file management service agent is in an idle state is determined; the first main node divides the directory tree into multiple sub-directory trees according to the number of second main nodes to generate multiple statistical sub-tasks that represent the size of the corresponding sub-directory trees, thereby realizing the division of the statistical sub-tasks.

[0097] In order to ensure that the istorage-agent of all secondary master nodes runs at the same time and completes the corresponding statistical subtasks at the same time, and to avoid a single subdirectory tree that is too large and takes a long time, resulting in a long overall time consumption, in the specific implementation, the following rules exist for the division of the directory tree:

[0098] (1) If the directory trees to be counted are independent of each other, the entire independent directory tree will be handed over to the idle istorage-agent for processing.

[0099] (2) If the directory tree to be counted is a single directory tree, it needs to be split into multiple sub-directory trees and handed over to multiple idle istorage-agent services for processing, and finally merged upwards into a single directory tree.

[0100] Therefore, based on the above-mentioned directory tree division rules, in some embodiments, each master node processes the corresponding statistical subtask, specifically including:

[0101] S111: Sending each statistical subtask to the corresponding second master node through the first master node;

[0102] S112: modifying the state of the corresponding file management service agent from an idle state to a busy state through the second master node, and counting the size of the subdirectory tree in the corresponding statistical subtask through the file management service agent to obtain a corresponding statistical subresult;

[0103] S113: Sending the corresponding statistical sub-result to the first master node through the second master node;

[0104] S114: Modify the state of the corresponding file management service agent from a busy state to an idle state through the second master node.

[0105] Specifically, each statistical subtask is first sent to the corresponding second master node through the first master node. It can be understood that when the istorage-agent of the first master node is in an idle state, the first master node is also the second master node, so the first master node will also retain a statistical subtask for itself and process it by itself. The second master node changes the state of the corresponding istorage-agent from an idle state to a busy state, indicating that the task processing is currently being executed, and uses the istorage-agent to count the size of the subdirectory tree in the corresponding statistical subtask to obtain the corresponding statistical sub-result.

[0106] After the second master node obtains the statistical sub-result corresponding to the statistical sub-task, the second master node sends the corresponding statistical sub-result to the first master node, and the first master node determines the final statistical result based on all the statistical sub-results. Finally, the second master node changes the state of the corresponding istorage-agent from busy to idle, indicating that the task processing is currently completed.

[0107] On this basis, in order to avoid a single sub-directory tree being too large and consuming a long time, resulting in a long overall time consumption, in some embodiments, after sending the corresponding statistical sub-result to the first main node through the second main node, the following is also included:

[0108] S115: Determine through the first master node whether there is an unfinished statistical subtask and whether there is a second master node whose corresponding file management service agent is in an idle state; if so, proceed to step S116;

[0109] S116: splitting the unfinished statistical subtask again through the first master node to obtain new statistical subtasks;

[0110] S117: Send the new statistical subtask to the second main node whose corresponding file management service agent is in an idle state through the first main node, so as to process the new statistical subtask.

[0111] Specifically, after the second master node sends the corresponding statistical sub-result to the first master node, the first master node may further determine whether there is any unfinished statistical sub-task and whether there is a second master node whose corresponding istorage-agent is in an idle state.

[0112] If the first master node confirms that there are no unfinished statistical subtasks, and / or there is no second master node whose corresponding istorage-agent is in an idle state, it is considered that no operation is required at present. If the first master node confirms that there are unfinished statistical subtasks, and there is a second master node whose corresponding istorage-agent is in an idle state, in order to avoid some sub-directory trees being too large and taking a long time, resulting in a long overall time consumption, the unfinished statistical subtasks need to be split again through the first master node to obtain new statistical subtasks; finally, the new statistical subtasks are sent to the second master node whose corresponding istorage-agent is in an idle state through the first master node to process the new statistical subtasks. The following is an example:

[0113] Figure 3 A schematic diagram of an independent directory tree and a sub-directory tree provided by an embodiment of the present invention. Figure 3 As shown in the figure, it is assumed that there are three master nodes in the K8s cluster, namely master node 1, master node 2 and master node 3, and each master node contains a corresponding istorage-agent. Among them, when the istorage-agent of master node 1 receives a statistical task, master node 1 is the first master node; when the istorage-agents of the three master nodes are all in an idle state, the three master nodes are all second master nodes.

[0114] If the statistical task indicates that it is necessary to count the sizes of all storage directories under the directory tree / a and the directory tree / m, the master node 1 will split the statistical task into three statistical subtasks: first, the directory tree / a and the directory tree / m are two independent directory trees. Because the istorage-agent of the master node 2 is in an idle state, the master node 1 can assign the directory tree / a to the istorage-agent of the master node 2 for statistical processing; because the istorage-agent of the master node 3 is in an idle state, the master node 1 can hand over the sub-directory tree / m / n1 under the directory tree / m to the istorage-agent of the master node 3 for statistical processing; the master node 1 will continue to count other sub-directory trees under / m.

[0115] If master node 2 or master node 3 is idle or has completed the assigned task, master node 1 will split the subdirectory tree being processed again to obtain a new subdirectory tree, such as / m / n2 / q1 / z1, and hand it over to the idle master node 2 or master node 3 for statistical processing. Otherwise, the istorage-agent of master node 1 will handle the remaining statistical tasks by itself. Finally, master node 1 collects the statistical results of each istorage-agent, updates the directory size upward, and finally obtains the size of directory tree / a and directory tree / m.

[0116] In addition, the first master node can also continuously monitor the processing speed of each second master node in processing the corresponding statistical subtasks. If the processing speed of a second master node is greater than a preset value, it is considered that the second master node has a higher task processing capability. Even if the second master node has not currently completed the corresponding statistical subtask, the first master node can also split the sub-directory tree currently being processed by other second master nodes again, and hand it over to the second master node with higher task processing capability for processing, thereby further improving the statistical efficiency.

[0117] In summary, by assigning the divided sub-directory trees to different idle istorage-agents for distributed statistics, the parallel statistics effect is achieved. At the same time, it is ensured that the statistical subtasks of all istorage-agents are completed at the same time, and frequent task allocation will not consume time and network performance.

[0118] In order to prevent a failed master node from being unable to process the corresponding statistical subtask, based on the above embodiment, in some embodiments, after each statistical subtask is sent to the corresponding second master node through the first master node, the method further includes:

[0119] S118: Monitor the status of the file management service agent of each second master node through the first master node, and determine whether there is a file management service agent of the second master node that is continuously busy; if so,

[0120] S119: determining the second master node whose corresponding file management service agent is continuously in a busy state as a faulty master node;

[0121] S120: Obtain the statistical subtask corresponding to the faulty master node through the first master node, and reallocate the statistical subtask corresponding to the faulty master node to the remaining second master nodes that are in an idle state;

[0122] S121: Changing the status of the file management service agent of the faulty master node from a busy status to a faulty status.

[0123] Specifically, after the first master node sends each statistical subtask to the corresponding second master node, the state of the istorage-agent of each second master node can be further monitored, and it can be determined whether there is an istorage-agent of the second master node that is continuously busy.

[0124] If it is confirmed that the istorage-agent of the second master node is continuously busy, the second master node whose corresponding istorage-agent is continuously busy is determined as the faulty master node. It should be noted that the faulty master node cannot process the corresponding statistical subtasks. Further, the statistical subtasks corresponding to the faulty master node are obtained through the first master node, and the statistical subtasks corresponding to the faulty master node are reallocated to the remaining second master nodes in the idle state, so as to ensure the normal processing of the statistical subtasks corresponding to the faulty master node. Finally, the state of the istorage-agent of the faulty master node is changed from busy to faulty, so that users can check and maintain the faults of the faulty master node in a timely manner.

[0125] In order to restore the normal operation of the failed master node, based on the above embodiment, in some embodiments, after the state of the file management service agent of the failed master node is changed from the busy state to the failed state, the method further includes:

[0126] S122: Control the failed master node to restart, and determine whether the failed master node resumes normal operation;

[0127] S123: If it is confirmed that the failed master node has resumed normal operation, the state of the file management service agent of the failed master node is changed from a failed state to an idle state.

[0128] Specifically, after confirming the faulty master node, control the faulty master node to restart, and determine whether the faulty master node has resumed normal operation. If it is confirmed that the faulty master node has resumed normal operation, the state of the istorage-agent of the faulty master node is changed from the faulty state to the idle state, so that the faulty master node is restored to an idle second master node and can participate in the processing of subsequent statistical subtasks.

[0129] In order to ensure efficient communication between the istorage-agents of each master node in the storage auxiliary module, based on the above embodiment, in some embodiments, the first master node sends each statistical subtask to the corresponding second master node respectively, including:

[0130] S124: The first master node sends each statistical subtask to the corresponding second master node through a preset remote procedure call method.

[0131] The data structure of the statistical subtask consists of an identifier number, a file management service agent name, a node where the file management service agent is located, a file management service agent status, an allocated storage directory statistical path, an allocated statistical task node, and a last modified time.

[0132] Specifically, in this embodiment, the istorage-agents communicate with each other using a preset Remote Procedure Call (RPC) method, and use the Internet protocol of the replica and the istorage-agent port for access. Since the ports of each microservice container service in the K8s cluster are not all mapped on the physical machine, in order to ensure the concealment and security of the access address, the RPC key is used to access the corresponding method at the same time to increase the security of RPC access. It should be noted that in this embodiment, there is no restriction on the specific type of the preset RPC method. For example, remote calls can be made through the JSON-RPC method. JSON-RPC is a stateless and lightweight RPC transmission protocol, which mainly transmits content through JSON.

[0133] In addition, the allocation of statistical subtasks follows the corresponding data structure. The data structure is as follows:

[0134] Table 1 istorage-agent allocation statistics subtask data structure table

[0135]

[0136] As shown in Table 1, the data structure of the statistical subtask is specifically composed of the identifier number, the file management service agent name, the node where the file management service agent is located, the file management service agent status, the allocated storage directory statistical path, the allocated statistical task node and the last modification time. Based on the above data structure, the master node can clearly know the detailed information of the corresponding statistical subtask, so as to accurately execute the statistical subtask.

[0137] In the above embodiment, the directory size statistics process is described in detail. On this basis, the K8s-based storage auxiliary module provided by this solution can also handle the task of video playback transcoding. Therefore, based on the above embodiment, in some embodiments, the method also includes:

[0138] S16: when receiving a video transcoding task, dividing the video transcoding task into a plurality of video transcoding subtasks;

[0139] S17: Processing corresponding video transcoding subtasks through each master node.

[0140] In a specific implementation, when the system receives a video transcoding task, it divides the video transcoding task into multiple video transcoding subtasks. It should be noted that the specific division method of the video transcoding subtasks is not limited in this embodiment. For example, a video transcoding task can be divided into multiple video transcoding subtasks according to the number of master nodes in a master node in the K8s cluster. The video transcoding task can also be divided into multiple video transcoding subtasks according to the computing resources of each master node, depending on the specific implementation situation. Finally, the processing of the video transcoding task is realized by processing the corresponding video transcoding subtask by each master node. Since the video transcoding task is processed jointly by multiple master nodes, it has higher processing efficiency.

[0141] In some embodiments, in order to further improve the processing efficiency of the video transcoding task, when dividing the video transcoding subtasks, a third master node whose computing resource amount is greater than a threshold is specifically determined among all the master nodes, and the number of the third master nodes is determined. In this embodiment, there is no restriction on the size of the threshold, which depends on the specific implementation situation. The video transcoding task is further divided into a plurality of video transcoding subtasks according to the number of the third master nodes, and each video transcoding subtask corresponds to each third master node one by one. Finally, the corresponding video transcoding subtask is processed by each third master node, that is, the video transcoding subtask is processed by the master node whose computing resource amount meets the requirements, which has a higher task processing efficiency.

[0142] In the above-mentioned embodiment, the directory statistics method is described in detail, and the present invention also provides a corresponding embodiment of the directory statistics device.

[0143] Figure 4 A schematic diagram of a directory statistics device provided by an embodiment of the present invention. The device is applied to a distributed management system; wherein the distributed management system includes multiple master nodes. Figure 4 As shown, the device comprises:

[0144] A division module 10 is used for, when receiving a statistical task for the directory size of a distributed cluster, dividing the statistical task into a plurality of statistical subtasks according to the number of master nodes;

[0145] The processing module 11 is used to process the corresponding statistical subtasks through each master node to obtain multiple statistical sub-results;

[0146] The summing module 12 is used to add up the statistical sub-results to obtain the statistical result of the directory size.

[0147] In some embodiments, it also includes:

[0148] A first setting module is used to set corresponding file management services and file management service agents for each master node;

[0149] An initialization module, used to initialize the state of each file management service agent to an idle state;

[0150] The second setting module is used to set the operation mode of each file management service agent to a load balancing mode.

[0151] In some embodiments, the first setting module includes:

[0152] The first determination submodule is used to determine the number of master nodes;

[0153] The first setting submodule is used to set the replica number field according to the number of master nodes, so as to set a corresponding replica for each master node respectively;

[0154] Among them, the copies include file management service copies and file management service proxy copies.

[0155] In some embodiments, the partitioning module 10 includes:

[0156] The second determination submodule is used to determine the directory tree whose size is to be counted according to the statistical task;

[0157] A third determination submodule is used to determine the first master node that has received the statistical task, and determine the second master node that has an idle state corresponding to the file management service agent;

[0158] The first division submodule is used for the first master node to divide the directory tree into multiple sub-directory trees according to the number of second master nodes, so as to generate multiple statistical subtasks representing the size of the corresponding sub-directory trees.

[0159] In some embodiments, the processing module 11 includes:

[0160] A first sending submodule, used to send each statistical subtask to the corresponding second master node through the first master node;

[0161] A first modification submodule is used to modify the state of the corresponding file management service agent from an idle state to a busy state through the second master node, and to count the size of the subdirectory tree in the corresponding statistical subtask through the file management service agent to obtain a corresponding statistical subresult;

[0162] A second sending submodule, used for sending the corresponding statistical sub-result to the first master node through the second master node;

[0163] The second modification submodule is used to modify the state of the corresponding file management service agent from a busy state to an idle state through the second master node.

[0164] In some embodiments, it also includes:

[0165] The first judgment submodule is used to judge whether there is an unfinished statistical subtask through the first main node, and whether there is a second main node whose corresponding file management service agent is in an idle state; if so, the splitting submodule is triggered;

[0166] A splitting submodule is used to split the unfinished statistical subtasks again through the first master node to obtain new statistical subtasks;

[0167] The third sending submodule is used to send the new statistical subtask to the second main node whose corresponding file management service agent is in an idle state through the first main node, so as to process the new statistical subtask.

[0168] In some embodiments, it also includes:

[0169] A second determination submodule is used to monitor the status of the file management service agent of each second master node through the first master node, and determine whether there is a file management service agent of the second master node that is continuously busy; if so, trigger a fourth determination submodule;

[0170] A fourth determination submodule is used to determine the second master node whose corresponding file management service agent is continuously in a busy state as a faulty master node;

[0171] The allocation and acquisition submodule is used to acquire the statistical subtask corresponding to the faulty master node through the first master node, and reallocate the statistical subtask corresponding to the faulty master node to the remaining second master nodes in an idle state;

[0172] The third modification submodule is used to modify the state of the file management service agent of the faulty master node from a busy state to a faulty state.

[0173] In some embodiments, it also includes:

[0174] The third judgment submodule is used to control the restart of the faulty master node and judge whether the faulty master node has resumed normal operation; if it is confirmed that the faulty master node has resumed normal operation, the fourth modification submodule is triggered;

[0175] The fourth modification submodule is used to modify the state of the file management service agent of the faulty master node from a faulty state to an idle state.

[0176] In some embodiments, the first master node sends each statistical subtask to the corresponding second master node respectively, specifically sending each statistical subtask to the corresponding second master node respectively through a preset remote procedure call method; wherein, the data structure of the statistical subtask is composed of an identifier number, a file management service agent name, a node where the file management service agent is located, a file management service agent status, an allocated storage directory statistical path, an allocated statistical task node and a last modification time.

[0177] In some embodiments, it also includes:

[0178] A second division submodule is used to divide the video transcoding task into a plurality of video transcoding subtasks when receiving the video transcoding task;

[0179] The processing submodule is used to process the corresponding video transcoding subtask through each master node.

[0180] In some embodiments, the second partitioning submodule includes:

[0181] A fifth determination submodule, configured to determine, among all the master nodes, third master nodes whose computing resource amount is greater than a threshold, and determine the number of the third master nodes;

[0182] A third division submodule, used to divide the video transcoding task into a plurality of video transcoding subtasks according to the number of the third master nodes; wherein each video transcoding subtask corresponds to each third master node one by one;

[0183] Correspondingly, the processing submodule specifically processes the corresponding video transcoding subtask through each third master node.

[0184] In some embodiments, the statistical task is a task for characterizing the size of directories that have changed in each directory of the statistical distributed cluster.

[0185] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0186] In addition, the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned directory statistics method when executed by a processor.

[0187] Figure 5 A schematic diagram of a directory statistics device provided by an embodiment of the present invention. Figure 5 As shown, the directory statistics equipment includes:

[0188] A memory 20, for storing computer programs;

[0189] The processor 21 is used to implement the steps of the directory statistics method mentioned in the above embodiment when executing the computer program.

[0190] The directory statistics device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0191] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array. The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0192] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the directory statistics method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include but is not limited to data involved in the directory statistics method.

[0193] In some embodiments, the directory statistics device may also include a display screen 22 , an input and output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0194] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation of the directory statistics device, and may include more or fewer components than those shown in the figure.

[0195] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0196] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0197] The above is a detailed introduction to a directory statistics method, product, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0198] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A directory statistics method, characterized in that: Applied to a distributed management system; wherein the distributed management system includes multiple master nodes; the method comprises: When receiving a statistical task for the directory size of a distributed cluster, dividing the statistical task into a plurality of statistical subtasks according to the number of the master nodes; Processing the corresponding statistical subtasks by each of the master nodes to obtain a plurality of statistical subresults; Each of the statistical sub-results is added together to obtain a statistical result of the directory size.

2. The directory statistics method according to claim 1, characterized in that: Before receiving the statistical task, the method further includes: Set up corresponding file management services and file management service agents for each of the master nodes respectively; Initializing the state of each of the file management service agents to an idle state; The operation mode of each of the file management service agents is set to a load balancing mode.

3. The directory statistics method according to claim 2, characterized in that: The corresponding file management service and file management service agent are respectively set for each of the master nodes, including: Determining the number of the master nodes; According to the number of the master nodes, a replica number field is set to respectively set a corresponding replica for each of the master nodes; The copies include file management service copies and file management service proxy copies.

4. The directory statistics method according to claim 2, characterized in that: The statistical task is divided into a plurality of statistical subtasks according to the number of the master nodes, including: Determine the directory tree whose size is to be counted according to the statistical task; Determine a first master node that has received the statistical task, and determine a second master node that is in an idle state corresponding to the file management service agent; The first master node divides the directory tree into a plurality of sub-directory trees according to the number of the second master nodes, so as to generate a plurality of statistical subtasks representing statistics corresponding to sizes of the sub-directory trees.

5. The directory statistics method according to claim 4, characterized in that: Processing the corresponding statistical subtask by each of the master nodes includes: Sending each of the statistical subtasks to the corresponding second master node through the first master node; The state of the corresponding file management service agent is changed from an idle state to a busy state through the second master node, and the size of the subdirectory tree in the corresponding statistical subtask is counted through the file management service agent to obtain the corresponding statistical subresult; Sending the corresponding statistical sub-result to the first master node through the second master node; The state of the corresponding file management service agent is changed from a busy state to an idle state through the second master node.

6. The directory statistics method according to claim 5, characterized in that: After sending the corresponding statistical sub-result to the first master node through the second master node, the method further includes: Determining, by the first master node, whether there is any unprocessed statistical subtask, and whether there is a second master node corresponding to the file management service agent in an idle state; If so, the unfinished statistical subtask is split again by the first master node to obtain a new statistical subtask; The new statistical subtask is sent to the second main node corresponding to the file management service agent in an idle state through the first main node, so as to process the new statistical subtask.

7. The directory statistics method according to claim 5, characterized in that: After sending each of the statistical subtasks to the corresponding second master node through the first master node, the method further includes: Monitoring the status of the file management service agent of each of the second master nodes through the first master node, and determining whether there is a file management service agent of the second master node that is continuously in a busy state; If so, determining the second master node corresponding to the file management service agent that is continuously in a busy state as a faulty master node; Acquire the statistical subtask corresponding to the faulty master node through the first master node, and reallocate the statistical subtask corresponding to the faulty master node to the remaining second master nodes that are in an idle state; The state of the file management service agent of the faulty master node is changed from a busy state to a faulty state.

8. The directory statistics method according to claim 7, characterized in that: After the state of the file management service agent of the faulty master node is changed from a busy state to a faulty state, the method further includes: Controlling the failed master node to restart, and determining whether the failed master node has resumed normal operation; If it is confirmed that the faulty master node has resumed normal operation, the state of the file management service agent of the faulty master node is modified from a faulty state to an idle state.

9. The directory statistics method according to claim 5, characterized in that: The first master node sends each of the statistical subtasks to the corresponding second master node, including: The first master node sends each of the statistical subtasks to the corresponding second master node respectively through a preset remote procedure call method; The data structure of the statistical subtask is composed of an identifier number, a file management service agent name, a node where the file management service agent is located, a file management service agent status, an allocated storage directory statistical path, an allocated statistical task node and a last modified time.

10. The directory statistics method according to claim 1, characterized in that: Also includes: When receiving a video transcoding task, dividing the video transcoding task into a plurality of video transcoding subtasks; The corresponding video transcoding subtask is processed by each of the master nodes.

11. The directory statistics method according to claim 10, characterized in that: The video transcoding task is divided into a plurality of video transcoding subtasks, including: Determine a third master node whose computing resource amount is greater than a threshold value among all the master nodes, and determine the number of the third master nodes; Dividing the video transcoding task into a plurality of the video transcoding subtasks according to the number of the third master nodes; wherein each of the video transcoding subtasks corresponds one-to-one to each of the third master nodes; Correspondingly, processing the corresponding video transcoding subtask by each of the master nodes includes: The corresponding video transcoding subtask is processed by each of the third master nodes.

12. The directory statistics method according to any one of claims 1 to 11, characterized in that: The statistical task is a task for representing statistics on the size of directories that have changed in each directory of the distributed cluster.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the directory statistics method described in any one of claims 1 to 12 are implemented.

14. A directory statistics device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the directory statistics method as described in any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the directory statistics method according to any one of claims 1 to 12 are implemented.

Images