Distributed storage node load prediction dynamic scheduling method and system

Abstract

The application discloses a distributed storage node load prediction dynamic scheduling method and system, and relates to the technical field of data processing; the method comprises the following steps: real-time statistics of disk data of each storage node is carried out, and a file attribute table is constructed; the file attribute table is split according to directory groups, a plurality of directory groups are obtained, and data subsets of the directory groups are formed; based on the data subsets of the directory groups and pre-labeled file sample data, a local anomaly detection model of each directory group is constructed; the local anomaly detection model is applied to file data of each directory group, anomaly determination is carried out, and a candidate false file list is obtained; the candidate false file list is calibrated, and a false occupied file list is obtained; the application improves the accuracy of false occupied file identification, ensures that cleaning tasks do not affect normal operation of business through dynamic scheduling, and improves storage resource utilization and system operation stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, more particularly, the present application relates to a distributed storage node load prediction dynamic scheduling method and system. BACKGROUND

[0002] In the actual deployment of large-scale distributed storage systems, many business scenarios commonly exist in the coexistence of extremely large directories and massive small files. In such scenarios, the file distribution of the storage nodes presents an uneven characteristic, with some directories collectively carrying a large number of associated or non-associated small files, and the attribute characteristic distribution of different types of files is different, forming a typical characteristic distribution unevenness and directory tilt phenomenon. In existing distributed storage systems, to solve the problem of resource waste caused by false disk occupied space, related technologies often use the isolated forest algorithm to realize the identification of false occupied space, and the core technical principle is as follows: by collecting the real-time total occupied amount of the storage node disk, the real-time effective data storage amount and the disk file attribute data, the difference between the total occupied amount and the effective data storage amount is calculated to preliminarily lock the potential false occupied space range, and the invalid file data of long-term non-access and non-associated business is filtered out, the above two types of data are input into the isolated forest algorithm model as features, and the algorithm is good at separating abnormal samples, and through the construction of multiple decision trees, the input samples are analyzed in isolation, and finally the identification result containing the false occupied file list and the false occupied space size is output, providing a basis for subsequent disk cleaning operation.

[0003] However, this technology has inherent algorithm sensitivity defects when dealing with characteristic distribution unevenness and directory tilt scenarios. The massive small files under the extremely large directory are highly homogenized in file attribute characteristics, and tend to form dense feature clusters, while the directory tilt will lead to the file characteristics of a few directories dominating the distribution of the entire sample set, making the feature distribution present a significant unevenness situation. The abnormal identification logic of the isolated forest algorithm depends on the dispersion of sample features, and when facing the above highly concentrated feature clusters, the algorithm has difficulty in effectively distinguishing between the effective files with concentrated distribution and the real abnormal false occupied files, and is prone to misjudging the feature homogenized effective files as abnormal samples; in addition, the unevenness of feature distribution caused by directory tilt will further amplify the identification deviation of the algorithm, leading to the concentrated occurrence of false positives of files in the same directory or the same type, and unable to accurately lock the real false occupied files.

[0004] The above problems can cause a series of influences, the cleaning hit rate of false occupied space recognition is greatly reduced, a large number of effective files are mistakenly included in the cleaning list, and the real false occupied files cannot be effectively recognized, so that the disk space release effect does not reach the expectation; in order to verify the attribute of the misjudged file and the false occupied attribute, the system needs to perform a scanning operation on a large amount of files, which produces a large amount of invalid disk IO and CPU resource consumption, causing waste of resources of the storage node and increasing disk wear; if the misjudged effective file involves core business data, the cleaning operation can directly affect the normal operation of the business, causing service interruption or data risk; in addition, the invalid scanning and miscleaning operation violates the core goal of reducing invalid disk operation and improving resource utilization in the technical design, which not only fails to solve the disk false occupied problem, but also increases the load pressure of the storage node, affecting the operation stability and service reliability of the distributed storage system.

[0005] In view of this, the present application provides a distributed storage node load prediction dynamic scheduling method and system to solve the above problems. SUMMARY

[0006] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned purposes, the present application provides the following technical scheme: a distributed storage node load prediction dynamic scheduling method and system.

[0007] The distributed storage node load prediction dynamic scheduling method comprises:

[0008] Real-time statistics of disk data of each storage node, constructing a file attribute table;

[0009] The file attribute table is split according to directory groups to obtain a plurality of directory groups, and data subsets of each directory group are formed;

[0010] Based on the data subsets of each directory group and the pre-labeled file sample data, a local anomaly detection model of each directory group is constructed;

[0011] The local anomaly detection model is applied to the file data of each directory group for anomaly determination to obtain a candidate false file list;

[0012] The candidate false file list is calibrated to obtain a false occupied file list;

[0013] The load conditions of each storage node are predicted to obtain node load prediction results of a plurality of prediction time slices in the future, and the false occupied file list is combined to dynamically schedule a cleaning task.

[0014] Further, the directory features of each directory in the file attribute table are counted, and the directories are clustered and hierarchically grouped based on the directory features to obtain a plurality of directory groups.

[0015] Furthermore, directory characteristics include the number of files in each directory, total size, average file size, percentage of unaccessed files, and percentage of unrelated files.

[0016] Furthermore, the directory is initially hierarchically divided based on the number of files and the total size, into a large file directory and other directories;

[0017] Construct a directory feature vector using directory features and then normalize it.

[0018] Based on the directory feature vectors, the remaining directories are clustered to obtain multiple directory groups divided according to directory features.

[0019] Furthermore, when analyzing the disk data of each storage node, the disk usage difference of each storage node is calculated by subtracting the real-time effective data storage from the total real-time disk usage of each storage node. This difference is used to characterize the size of the potential false occupied space on each storage node.

[0020] Furthermore, storage nodes whose disk usage difference as a percentage of the total real-time disk usage exceeds a preset first threshold are marked as false space usage risk nodes, and file attribute tables are constructed only for files on false space usage risk nodes.

[0021] Furthermore, the original anomaly measure of each file in each directory group is calculated using a local anomaly detection model and then normalized to obtain an anomaly score for each file.

[0022] When a file's anomaly score is greater than or equal to the anomaly threshold, the file is determined to be an abnormal file; when a file's anomaly score is less than the anomaly threshold, the file is determined to be a normal file.

[0023] The anomaly threshold is determined by the local anomaly detection model.

[0024] Further, the difference between the abnormal score and the abnormal threshold of the abnormal file is calculated and recorded as the abnormal difference value; abnormal files with an abnormal difference value greater than or equal to the preset high priority deviation threshold are marked as priority cleanup objects, and abnormal files with an abnormal difference value less than or equal to the preset edge deviation threshold are marked as secondary suspicious files; other abnormal files are not marked.

[0025] Furthermore, the upper limit of cleanable data in the predicted time slice is calculated, and it is determined whether the predicted time slice is an executable cleanup time slice;

[0026] Within any executable cleanup time slice on the same storage node, tasks in the priority cleanup object task queue are prioritized. If there is still a remaining amount of cleanupable data in the executable cleanup time slice and no priority cleanup object can be assigned, secondary suspicious files are allowed to enter the executable cleanup time slice.

[0027] The distributed storage node load prediction dynamic scheduling system comprises:

[0028] A data acquisition module is configured to statistically acquire disk data of each storage node in real time and construct a file attribute table;

[0029] A directory grouping module is configured to split the file attribute table according to directory groups to obtain a plurality of directory groups and form data subsets of the directory groups;

[0030] A model construction module is configured to construct a local anomaly detection model for each directory group based on the data subsets of the directory groups and pre-labeled file sample data;

[0031] An anomaly determination module is configured to apply the local anomaly detection model to file data of each directory group to determine anomalies and obtain a candidate false file list;

[0032] A file calibration module is configured to calibrate the candidate false file list to obtain a false occupied file list;

[0033] A dynamic scheduling module is configured to predict the load of each storage node to obtain node load prediction results of a plurality of prediction time slices, combine the false occupied file list, and dynamically schedule cleaning tasks.

[0034] Compared with the prior art, the distributed storage node load prediction dynamic scheduling method and system have the following technical effects and advantages:

[0035] The file attribute table is constructed by statistically acquiring disk data of each storage node in real time, the directories are hierarchically clustered according to directory characteristics such as the number of files and total size, a plurality of directory groups and corresponding data subsets are obtained, a local anomaly detection model is constructed based on the subsets and a small amount of labeled samples, a candidate false file list is obtained through anomaly determination, a false occupied file list is obtained through multi-dimensional calibration such as labeled sample verification and attribute review, the future load of each node is predicted, the priority cleaning objects and secondary suspicious files are distinguished based on the list, and the cleaning tasks are reasonably distributed within the executable cleaning time slice.

[0036] The present application effectively solves the problems of homogeneous effective file misjudgment, low cleaning hit rate, excessive invalid scanning, increased disk wear and tear and affected business operation caused by algorithm sensitivity in the uneven feature distribution and directory tilt scenarios of the prior art, improves the accuracy of false occupied file identification, reduces invalid disk operations, reduces the false positive rate and disk wear and tear, ensures that the cleaning task does not affect the normal operation of the business through dynamic scheduling, optimizes resource allocation, improves storage resource utilization and system operation stability. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1A schematic diagram of a distributed storage node load prediction dynamic scheduling system according to an embodiment of the present application;

[0038] Figure 2 A flowchart of a distributed storage node load prediction dynamic scheduling method according to an embodiment of the present application;

[0039] Figure 3 A flowchart of grouping directories according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described in detail, clearly and completely below with reference to the accompanying drawings. It should be particularly noted that the specific embodiments described below are only used to better illustrate and describe the technical solutions of the present application, and are intended to enable those skilled in the art to better understand and implement the present application, and should not be understood as limiting the scope of protection of the present application. Those skilled in the art can modify, adjust or equivalently replace the present application according to the content disclosed in the present application without departing from the spirit and essence of the present application, and these should be considered as falling within the scope of protection of the present application.

[0041] Embodiment 1:

[0042] Please refer to Figure 1 As shown in the figure, the embodiment discloses a distributed storage node load prediction dynamic scheduling system, which comprises a data acquisition module, a directory grouping module, a model construction module, an abnormality determination module, a file calibration module and a dynamic scheduling module. Each module is connected by wire and / or wireless connection to realize data transmission.

[0043] The data acquisition module is used to statistically record the disk data of each storage node in real time and to construct a file attribute table.

[0044] The original data is preprocessed. For each storage node, the real-time total disk occupancy of the storage node is subtracted from the real-time effective data storage of the storage node to calculate the disk occupancy difference of the storage node, which is used to represent the size of the potential false occupancy space on the storage node. All storage nodes are sorted in descending order of disk occupancy difference. The storage nodes whose disk occupancy difference accounts for more than a preset first threshold of the real-time total disk occupancy are marked as false occupancy space risk nodes. Only the files on the false occupancy space risk nodes are involved in subsequent directory hierarchical modeling and local anomaly detection. The storage nodes whose disk occupancy difference accounts for less than or equal to the preset first threshold are excluded from subsequent analysis, thereby reducing the overall computational overhead while ensuring the recognition effect. In the subsequent cleaning and scheduling stage, the false occupancy space risk nodes are assigned a higher cleaning priority and a larger cleaning space target according to the disk occupancy difference.

[0045] The disk file attribute data is cleaned and structured, and the specific method comprises: traversing each file record in the disk file attribute data, deleting abnormal records with empty file directory path, missing file size that cannot be obtained through underlying file system query, missing last access time that cannot be recovered from system log, retaining only one of repeated records with the same file directory path and file size, removing leading and trailing spaces and repeated path separators from the file directory path field, and uniformly adopting the agreed path separator format, converting the last access time field to a unified standard time format and a unified time zone representation, and setting the records with empty business association identifier field to unknown business identifier value.

[0046] After the above cleaning is completed, the file directory path of each legal disk file attribute data record is parsed, and each level directory name is extracted in sequence according to the path separator from left to right to generate the directory level information of the file. The files with the same bottommost directory path are classified into the same directory, and the storage node identifier, file directory path, directory level information, file size, file type, last access time, business association identifier of the file are written into the corresponding fields of the structured data table. Meanwhile, the file directory path and file name of each record are matched according to the long-term unvisited file list obtained by filtering according to the last access time. If they are consistent, the long-term unvisited file flag field of the record is set to yes, otherwise it is set to no. Then, the business association identifier of each record is matched according to the unassociated business file list obtained by filtering based on the business association identifier. If they are consistent, the unassociated business file flag field of the record is set to yes, otherwise it is set to no. Thus, the structured data table containing the complete attributes and derived flag fields of each file is obtained, and the disk occupancy difference of each storage node is reserved. Therefore, when the directory feature modeling and local anomaly detection are performed based on the structured data table, the directories and files on the false occupancy space risk nodes can be preferentially modeled and detected in combination with the disk occupancy difference, and the cleaning space target and cleaning order can be determined according to the disk occupancy difference of each storage node at the global level.

[0047] The directory grouping module is used for splitting the file attribute table according to directory groups to obtain multiple directory groups and form data subsets of the directory groups.

[0048] Please refer to Figure 3As shown, in order to solve the problems of uneven feature distribution and directory skew, when performing directory-level bucket modeling, the features explicitly indicate five pieces of data in the directory feature data, including the number of files, the total size, the average file size, the proportion of unvisited files, and the proportion of irrelevant files. In the specific implementation process, first, the disk file attribute data is aggregated and counted in units of directories, and a directory feature data record is generated for each directory. In the record, the number of files is the number of all files under the directory, the total size is the sum of the sizes of all files under the directory, the average file size is the total size divided by the number of files, the proportion of unvisited files is the ratio of the number of long-term unvisited files under the directory to the number of files, and the long-term unvisited files are files that have not been accessed for more than a set threshold period, for example, files that have not been accessed for more than six months or a year; the proportion of irrelevant files is the ratio of the number of irrelevant business files under the directory to the number of files, and the irrelevant business files are files that do not belong to any known business application according to the business association identifier in the disk file attribute data. After obtaining the directory feature data of all directories, the directories are preliminarily layered according to the number of files and the total size of the two pieces of record feature data, and directories with a number of files greater than a preset number threshold or a number of files in a certain percentage of the file number distribution of all directories and a total size greater than a preset size threshold are marked as large file directories, and each large file directory is separately divided into a directory group; the remaining directories that are not marked as large file directories are taken as a set of ordinary directories.

[0049] On the set of ordinary directories, the number of files, the total size, the average file size, the proportion of unvisited files, and the proportion of irrelevant files in the directory feature data are used to construct a directory feature vector, and each component is normalized. The specific way of normalization is as follows: for all directories in the set of ordinary directories, the overall minimum value and the overall maximum value of the five pieces of directory feature data, including the number of files, the total size, the average file size, the proportion of unvisited files, and the proportion of irrelevant files, are respectively calculated. The value of any directory on any piece of directory feature data is subtracted from the overall minimum value of the directory feature data, and then divided by the difference between the overall maximum value and the overall minimum value of the directory feature data, to obtain a normalized value between 0 and 1. The normalized number of files, the normalized total size, the normalized average file size, the normalized proportion of unvisited files, and the normalized proportion of irrelevant files are sequentially composed into the directory feature vector of the directory in a fixed order; after obtaining the directory feature vectors of all directories, a clustering algorithm based on the directory feature vectors is used for clustering. Each directory feature vector is input into the clustering algorithm, the distance between the directory feature vectors is calculated, and the clustering center is iteratively updated until the clustering result converges. Each directory feature vector is assigned a cluster number, and each cluster number corresponds to a directory group, so that the ordinary directories are divided into several directory groups.

[0050] In order to quantify the obvious difference in the directory feature data between different directory groups, for each directory group, the normalized average values of all directories in the directory group in the file quantity, total size, average file size, non-access file ratio and irrelevant file ratio are calculated to form the directory feature data average vector of the directory group. For any two directory groups, the difference in the normalized average values of the file quantity, total size, average file size, non-access file ratio and irrelevant file ratio is compared. When the absolute value of the difference between the normalized average values of the two directory groups in at least one directory feature data is greater than a preset difference threshold, it is determined that there is an obvious difference between the two directory groups in the directory feature data. The difference threshold can be set to 0.2. Since the directory feature data is normalized to a value range of 0 to 1, the threshold of 0.2 can ensure that the difference between different directory groups in the directory feature data reaches a certain range. Thus, through the above layering based on the file quantity and total size, and clustering grouping with all five data in the directory feature data as the feature vector, the directories in each directory group have similar directory feature data in terms of file quantity, total size, average file size, non-access file ratio and irrelevant file ratio. The difference between different directory groups reaches the obvious difference defined by the preset difference threshold in at least one directory feature data, thereby ensuring that the files in each group are relatively homogeneous in characteristics, and the difference between groups is obvious. This provides a stable and distinguishable directory feature data basis for subsequent training of local anomaly detection models in each directory group.

[0051] The file attribute table is split by directory groups to form grouped data subsets. Each data subset corresponds to a directory group, and each file data subset contains attribute records of all files in the corresponding group, which is used to retain all original attributes and flags. Disk file attributes include file directory path, file size, file type, last access time and business association identifier. The file directory path is the directory level information of the file, the file size is the number of bytes or blocks, the file type is the file extension or content type, the last access time is the timestamp of the last time the file is read or modified, and the business association identifier is used to mark whether the file belongs to a certain business application or user, and is empty if not. This embodiment effectively avoids the dominance of a few large directories on the overall feature distribution by introducing directory layering and bucketing, and reduces the interference of directory skew on subsequent models.

[0052] The model construction module constructs a local anomaly detection model for each directory group based on the data subset of each directory group and the pre-labeled file sample data.

[0053] For each data subset, a training set is first divided therefrom for the construction of the local anomaly detection model, and the remaining data is used as a validation set for subsequent evaluation and correction of the local anomaly detection model. In this way, the training of the local anomaly detection model is only based on the training set and does not directly use the real-time data to be detected in the validation set, thereby avoiding model bias caused by data mixing.

[0054] Subsequently, a small amount of labeled sample data is used to improve the accuracy of the local anomaly detection model. The small amount of labeled sample data is a small amount of file samples and their class labels marked by humans or history, including normal files and false occupation files. For example, a number of samples confirmed as false occupation files are selected according to past cleaning experience, and a number of normal business file samples are selected to guide model training. Specifically, a feature vector is extracted from the file attributes in the training set, which is composed of five features: file size, file type code, time since last access, whether it is a long-term unvisited file, and whether it is an unrelated business file. A feature vector set is generated for all files in the training set. On this basis, a complete training process is performed on the training set corresponding to each directory group. First, clustering analysis is performed on the feature vectors of the training set of the directory group. A predetermined number of clusters and a distance metric based on the Euclidean distance of the feature vectors are used to calculate the distance between each feature vector. Clustering is iteratively performed by updating the cluster center and reallocating the feature vectors until the change in the cluster center between adjacent iterations is less than a predetermined convergence threshold. Each file feature vector in the training set is assigned to a cluster.

[0055] After obtaining the cluster, an improved isolation forest model is constructed using the feature vectors of all files in the cluster as input samples. In constructing each isolation forest decision tree, only samples in the cluster are randomly selected as training samples, and in each node partition, the feature with the largest variance in the cluster is selected as the partition feature from the five features: file size, file type code, time since last access, whether it is a long-term unvisited file, and whether it is an unrelated business file. By preferring to use features with higher discrimination in the local feature space for partitioning, the improved isolation forest model can form more fine-grained distinctions between local dense and sparse regions in the cluster, thereby improving the ability to identify anomalies in dense feature clusters.

[0056] For the catalog group with large amount of training set data, the local density based anomaly detection model or the one-class support vector machine anomaly detection model can be additionally trained after the improved isolation forest model is trained, and the same feature vectors are used for training, and the anomaly scores output by the improved isolation forest model, the local density anomaly detection model and the one-class support vector machine anomaly detection model are weighted and fused according to the preset weight in the subsequent identification stage, so that the comprehensive anomaly score has higher identification accuracy under the premise of controllable calculation overhead.

[0057] When the local anomaly detection model is constrained by a small amount of labeled sample data, the labeled normal file samples are integrated into the training set, and the labeled normal file samples and the unlabeled normal samples in the training set are used as the basis for learning normal patterns when the local anomaly detection model is trained, wherein the labeled normal file samples and the unlabeled normal samples are ensured to participate in the node division and path generation of the isolation forest decision tree when the improved isolation forest model is constructed, so that the local anomaly detection model is dominated by normal patterns when dividing the feature space.

[0058] For the labeled false occupied file samples, after the preliminary training of the local anomaly detection model is completed, the labeled false occupied file samples are input into the local anomaly detection model to calculate the anomaly score, and the anomaly score of the labeled normal file samples is also calculated, and the anomaly threshold and internal parameters of the local anomaly detection model are adjusted by comparing the anomaly scores of the labeled false occupied file samples and the labeled normal file samples. The anomaly threshold and internal parameters are repeatedly adjusted so that the anomaly scores of all labeled false occupied file samples are higher than the anomaly threshold, and the anomaly scores of all labeled normal file samples are lower than the anomaly threshold, so as to complete the calibration of the local anomaly detection model. Finally, an anomaly detection model for each catalog group is trained, which is used for the catalog group. The anomaly detection model integrates the domain prior information of the catalog group and the supervision information provided by a small amount of labeled sample data, thereby improving the sensitivity of the local anomaly detection model to subtle anomalies in the centralized distribution data and reducing the misjudgment of homogenized normal files.

[0059] The anomaly determination module is configured to apply the local anomaly detection model to the file data of each catalog group to perform anomaly determination and obtain a candidate false file list.

[0060] The trained local anomaly detection model is applied to the grouped file data to calculate the anomaly score of each file. Specifically, for each directory group, each file feature vector in the directory group is input into the local anomaly detection model of the directory group, and the original anomaly metric is output by the local anomaly detection model. Then, the original anomaly metric is normalized according to the minimum value and the maximum value. The original anomaly metric is subtracted from the minimum value of all file original anomaly metrics and divided by the difference between the maximum value and the minimum value to obtain a value ranging from 0 to 1. The value is defined as the anomaly score of the file. The larger the anomaly score value, the more abnormal the file is. After obtaining the anomaly scores of all files in the directory group, the anomaly threshold value determined in the training and calibration stage is used to determine the anomaly of the files. When the anomaly score of a file is greater than or equal to the anomaly threshold value, the file is preliminarily determined to be an abnormal file. When the anomaly score of a file is less than the anomaly threshold value, the file is preliminarily determined to be a normal file and is not included in the candidate false file list.

[0061] To further distinguish the abnormality degree between different abnormal files, for the files preliminarily determined to be abnormal files, the difference between the anomaly score of the file and the anomaly threshold value is calculated, which is denoted as the anomaly difference value. The file with an anomaly difference value greater than or equal to a preset high-priority deviation threshold value is identified as a file with a particularly high abnormality degree. The high-priority deviation threshold value is 0.2. The file with an anomaly difference value less than or equal to a preset edge deviation threshold value is identified as an edge case close to the anomaly threshold value. The edge deviation threshold value is 0.05. Therefore, the abnormal file with an anomaly score greater than or equal to the anomaly threshold value plus the high-priority deviation threshold value is marked as a priority cleaning object. The abnormal file with an anomaly score between the anomaly threshold value minus the edge deviation threshold value and the anomaly threshold value plus the edge deviation threshold value is marked as a secondary suspicious file. Meanwhile, the anomaly score and the directory group identifier of each file preliminarily determined to be an abnormal file are recorded in the candidate false file list. The anomaly score is used as a quantitative basis for subsequent result calibration, priority sorting, and cleaning strategy formulation. Finally, all records of the files preliminarily determined to be abnormal files, together with the anomaly scores and the directory group identifiers, form the candidate false file list.

[0062] In the embodiment, since the local anomaly detection model only operates in the local range of each directory group, a large number of feature-similar files in the same directory group have been learned as normal patterns by the local anomaly detection model in the training stage. Therefore, in the application stage, only the files with a significantly higher anomaly score than the anomaly threshold value are marked as abnormal files, thereby effectively reducing the false positives of normal files caused by the tilt of directory feature data.

[0063] The file calibration module is configured to calibrate the candidate false file list to obtain a false occupancy file list.

[0064] The candidate false file list obtained in the previous step is subjected to calibration filtering to further reduce the false positive risk. The specific method includes:

[0065] First, a labeled sample verification operation is performed to check whether the candidate false file list contains previously labeled normal file samples. If such files exist, it is determined to be a false positive. For objects that are marked as abnormal but belong to normal files in the labeled data, the abnormal mark is immediately revoked, and the feature correlation reason for the false positive is analyzed in depth. The corresponding local anomaly detection model parameters or abnormal score threshold of the corresponding group are adjusted accordingly. If there are other files consistent with the feature dimension of the false positive file, the abnormal score of such files can be adjusted overall.

[0066] Similarly, if the labeled false occupation sample does not appear in the candidate false file list, it is determined that there is a false negative situation. At this time, the abnormal score threshold of the local anomaly detection model of the corresponding group needs to be appropriately reduced, or the missed false occupation sample is directly supplemented to the candidate false file list. Subsequently, attribute review is performed on each file in the candidate false file list based on the disk file attribute data, and the last access time and business association identifier of each candidate file are checked again. If it is found that the last access time of a candidate file is recent or has a clear business association identifier, it is determined to be a false positive and is removed from the candidate false file list.

[0067] The proportion of files marked as abnormal in each directory is counted. If the proportion of files marked as abnormal in a directory exceeds a preset proportion threshold, it is necessary to carefully judge whether the entire directory belongs to a situation where the feature distribution is special but actually carries valid business data. In such a situation, manual evaluation or temporary cleaning process can be performed on the candidate files in the directory.

[0068] Finally, commonality verification is performed on the candidate files across groups. If a set of candidate files with consistent feature dimensions appears in multiple different groups, the common features of such files are analyzed in depth to determine whether they belong to real invalid files. If it is confirmed that the batch false positives are caused by feature learning bias of the local anomaly detection model, the files are filtered out overall, or the feature weight of the corresponding local anomaly detection model is adjusted to reduce the recognition sensitivity of such features. After the above multi-dimensional calibration filtering operation, the obviously misjudged files in the candidate false file list are effectively removed, and the missed known false occupation samples are supplemented. Finally, a refined high-confidence false occupation file list is obtained. This process can effectively avoid the misjudgment of a large number of valid files into the cleaning range, significantly reduce the false positive rate, improve the accuracy and reliability of the false occupation file list, and provide a solid basis for subsequent cleaning operations.

[0069] In the process of summarizing the final confirmed false occupation file list to calculate the releaseable space capacity, first, each file record in the final confirmed false occupation file list is traversed, the file size corresponding to the file record is read, the file sizes of all file records are summed up, and the size of the recyclable false occupation space is obtained; after obtaining the size of the recyclable false occupation space, the real-time total disk occupation and the real-time effective data storage of the corresponding storage node are obtained, the difference between the real-time total disk occupation and the real-time effective data storage is calculated, and the recyclable false occupation space size is consistent with the difference. When the recyclable false occupation space size is less than or equal to the difference, it is determined that the final confirmed false occupation file list is consistent with the space occupation state of the storage node; when the recyclable false occupation space size is greater than the difference, it is determined that the final confirmed false occupation file list has repeated statistics or file size acquisition abnormality, triggering the deduplication processing of the final confirmed false occupation file list and recalculating the recyclable false occupation space size, until the recyclable false occupation space size is less than or equal to the difference. The real-time total disk occupation is the total amount of disk space currently used by each storage node. It can be obtained through the monitoring interface of the distributed storage system or the operating system command, for example, using the df command or the system API to periodically collect the used space values of each node. The real-time effective data storage is the total amount of space occupied by the actual effective data on each storage node. It can be calculated by summarizing the sizes of all valid files in the metadata of the distributed storage system, or obtained by using the effective data counter maintained by the system.

[0070] The recyclable false occupation space size is divided by the real-time total disk occupation to obtain the false occupation space proportion, and the false occupation space proportion is compared with the preset cleaning benefit threshold. When the false occupation space proportion is greater than or equal to the preset cleaning benefit threshold, it is determined that the cleaning of the corresponding storage node has a space benefit that meets the requirements; when the false occupation space proportion is less than the preset cleaning benefit threshold, it is determined that the space benefit of the cleaning of the corresponding storage node is insufficient, and the storage node is marked as a low-priority cleaning node.

[0071] Finally, the identification result is arranged and output as a data structure for cleaning scheduling, which at least includes the final confirmed false occupation file list, the recyclable false occupation space size, and the false occupation space proportion. The final confirmed false occupation file list includes the file directory path, the file size, and the storage node identification of each false occupation file, the recyclable false occupation space size is used to represent the releaseable space capacity of the storage node, and the false occupation space proportion is used to represent the benefit level of the cleaning operation of the storage node. Based on the data structure, a cleaning task is issued to the corresponding storage node during the business low peak period, so as to release the false occupation space and improve the storage resource utilization rate on the premise of ensuring the stability of the business load.

[0072] The embodiment successfully locates the invalid files to be cleaned up and estimates the releasable space capacity in the uneven feature distribution and catalog tilt scenario, thereby providing a reliable basis for subsequent disk cleaning.

[0073] The dynamic scheduling module is configured to predict the load conditions of each storage node, obtain node load prediction results of a plurality of prediction time slices in the future, and combine the false occupied file list to dynamically schedule the cleaning task.

[0074] The node running load data is subjected to time alignment processing, and the CPU utilization rate, disk I / O throughput, IOPS, network bandwidth utilization rate, request number, request queue length and other indexes of different sampling periods are unified to a unified time granularity. The obviously incorrect or missing data points are removed, and the short-time missing is interpolated and completed. For each time slice, a comprehensive load label of the time slice is generated for each storage node, for example, a weighted sum of the CPU utilization rate, disk I / O utilization rate and network bandwidth utilization rate, or a multi-dimensional load vector is defined. If necessary, the real-time total disk occupancy and real-time effective data storage are combined, and the remaining space ratio is taken as an auxiliary load index to be included in the label, which is used for subsequent dynamic scheduling to predict the comprehensive state of the load and capacity. The prediction features are constructed in a time sequence manner. For each node, at each time slice, the CPU utilization rate sequence, disk I / O throughput sequence, request number sequence and historical cleaning task execution amount of the node in the previous time slices are extracted to form a load feature vector of the node at the time slice.

[0075] The above feature vector and the load label of the corresponding time slice are combined to form a training sample. The future load can be defined as the average load or peak load of the node in the next time slices, for example, the prediction load of the next 15 minutes or the next 1 hour. Based on the node load prediction feature sample set, a suitable time series prediction model or regression model is selected for training, for example, a long short-term memory network model, a gradient boosting tree model or a random forest regression model. The training target is to learn the mapping relationship from the historical load features to the future load labels, so that the model can predict the load level of each node in the future multiple time slices. The mean square error, mean absolute error, P95 error and other prediction error indicators of different models are evaluated using the leave-out validation set or cross-validation method, and the model with better prediction performance and acceptable calculation overhead is selected as the node load prediction model.

[0076] During actual operation, the system receives the latest storage node load data in real time, constructs a load feature vector for the current moment, and inputs this feature vector into a pre-trained node load prediction model to obtain load prediction results for each storage node over multiple time slices, including predicted CPU utilization, disk I / O utilization, network utilization, or overall load index. Based on the final list of falsely occupied files, cleanable falsely occupied files are categorized according to their storage nodes. For each storage node, the total amount of freeable space and the number of files are calculated, forming a set of cleanup candidate tasks for that storage node. Each candidate task can be further refined into sub-tasks by directory or batch, facilitating granular control of scheduling.

[0077] Combining storage node load prediction results with storage node business importance and service level parameters, when calculating the available resource reserves for cleanup tasks for each storage node in future time slices, it is necessary to simultaneously consider the different processing strategies for abnormal files marked as priority cleanup targets and those marked as secondary suspicious files in the candidate fake file list. Node business importance and service level parameters describe the criticality of the services carried by different storage nodes, including the service type of the node, service level SLA targets, such as the maximum allowed average latency, maximum P99 latency, maximum error rate, and peak service periods. Node business importance and service level parameters can be provided by the operation and maintenance management system or the service configuration center and synchronized periodically. First, a business load security threshold is set for each storage node. This threshold is determined by the storage node's business importance and service level parameters, and the upper limits for each resource dimension are given, including the upper limit for CPU utilization, disk I / O utilization, and network bandwidth utilization.

[0078] Then, for each predicted time slice, the storage node load prediction result for that time slice is read, and the available resource margin is calculated in three dimensions: CPU utilization, disk I / O utilization, and network bandwidth utilization. Specifically, the predicted load value of each resource dimension is subtracted from the service load safety threshold for that resource dimension to obtain the available resource margin for that resource dimension in the predicted time slice. When the available resource margin in any resource dimension is less than or equal to zero, the predicted time slice is determined to lack the resource conditions for executing the cleanup task and is marked as an unexecutable cleanup time slice; when the available resource margin in all three resource dimensions is greater than zero, the predicted time slice is determined to have the resource conditions for executing the cleanup task and is marked as an executable cleanup time slice.

[0079] On this basis, in order to reflect the differential processing of the priority cleaning object and the secondary suspicious file, the executable cleaning time slice is further calculated the upper limit of the cleanable data volume of the prediction time slice. Specifically, based on the cleaning and data migration task execution log data, the average consumption coefficient of the central processing unit utilization rate, the disk input / output utilization rate and the network bandwidth utilization rate is extracted, and the minimum value of the cleanable data volume of the prediction time slice in the three resource dimensions is taken as the upper limit of the cleanable data volume of the prediction time slice, so as to ensure that the cleaning task allocated to the prediction time slice will not exceed the business load safety threshold in the execution process. The cleaning and data migration task execution log data is the starting time, the ending time, the involved file quantity, the released space size, the resource consumption during task execution and the like of the disk cleaning task and the data migration task actually executed by each node in each time period. The task ID can be generated by the cleaning scheduling module when the task is issued, and the key indicators are continuously recorded during the task execution process, and the complete log is formed after the task is completed.

[0080] When constructing the cleaning candidate task set of the storage node, the abnormal files marked as priority cleaning objects in the candidate false file list and the abnormal files marked as secondary suspicious files are respectively classified into different task queues, wherein the priority cleaning object task queue only contains the abnormal files corresponding to the priority cleaning objects and is sorted in descending order of the releasable space, and the secondary suspicious file task queue only contains the abnormal files corresponding to the secondary suspicious files and is sorted in descending order of the abnormal score. The scheduling allocation rule is set to preferentially allocate the tasks in the priority cleaning object task queue in any executable cleaning time slice of the same storage node, and only when there is still a remaining upper limit of the cleanable data volume in the executable cleaning time slice and there is no task to be allocated in the priority cleaning object task queue, the tasks in the secondary suspicious file task queue are allowed to be allocated into the executable cleaning time slice, so as to realize the strategy of preferentially cleaning the abnormal files marked as priority cleaning objects and cautiously cleaning the abnormal files marked as secondary suspicious files.

[0081] After completing the resource residual amount calculation and the task queue construction, the total amount of the globally releasable false occupied space, the future load situation of each storage node and the business importance of the storage node are comprehensively considered, the scheduling target and the constraint condition are set to maximize the total amount of the false occupied space release in the preset time window without breaking through the business load safety threshold of each prediction time slice of each storage node, and the storage node with high business importance and service level parameter is set to have a peak period limit, so that the prediction time slice corresponding to the business peak period is not marked as an executable cleaning time slice and no cleaning task is allocated.

[0082] When constructing the scheduling algorithm based on the scheduling target and the constraint condition, first, the upper limit of the cleanable data volume of all executable cleaning time slices within the preset time window of each storage node is summed to obtain the cleanable capacity of the storage node, and then the cleaning demand volume of the storage node is generated in combination with the false occupied space size of the storage node, and the storage nodes are sorted according to the matching degree of the cleanable capacity and the cleaning demand volume, so that the storage nodes with more sufficient cleanable capacity and higher cleaning demand volume are preferentially entered into the scheduling and distribution process.

[0083] For each storage node entering the scheduling and distribution process, the executable cleaning time slices thereof are sequentially traversed, the tasks in the priority cleaning object task queue are sequentially distributed to the current executable cleaning time slice in the sorting order, and the cleaning data volume already distributed in the time slice is accumulated, until the cleaning data volume already distributed in the time slice reaches the upper limit of the cleanable data volume or the priority cleaning object task queue is empty; under the condition that the cleaning data volume already distributed in the time slice does not exceed the upper limit of the cleanable data volume and the priority cleaning object task queue is empty, the tasks in the secondary suspicious file task queue are sequentially distributed to the current executable cleaning time slice in the sorting order until the cleaning data volume already distributed in the time slice reaches the upper limit of the cleanable data volume or the secondary suspicious file task queue is empty.

[0084] When all the executable cleaning time slices of the storage node within the preset time window have reached the upper limit of the cleanable data volume and there are still undistributed tasks in the priority cleaning object task queue, the undistributed tasks are marked as tasks needing cross-storage node processing and are entered into a cross-storage node migration candidate set, a target storage node with lower predicted load and more sufficient disk space is selected from the cross-storage node migration candidate set based on the storage node load prediction result, and the data corresponding to the undistributed tasks is migrated to the target storage node for cleaning, thereby forming a storage node load prediction dynamic scheduling plan covering all the storage nodes, the storage node load prediction dynamic scheduling plan including a cleaning task set corresponding to each time slice in the future for each storage node and execution order information of the priority cleaning object task and the secondary suspicious file task in the cleaning task set, and ensuring that an explicit correspondence between the storage node, the time slice and the cleaning task is output for the scheduling and execution module to issue and execute.

[0085] According to the cleaning task dynamic scheduling plan, the cleaning tasks in the corresponding time slice are issued to the execution agent of each storage node, and the running load data of the storage node is continuously monitored in the actual execution process, if it is found that the real-time load of a certain storage node exceeds the preset safety threshold, the current or subsequent cleaning task of the storage node can be paused, or the cleaning task volume of the storage node can be dynamically reduced.

[0086] During the execution of the cleaning task, the start time, end time, actual released space size, CPU and disk I / O occupancy during the execution, and the like of each task are recorded in real time, and the information is appended to the cleaning and data migration task execution log data, and the storage node running load data is updated.

[0087] Periodically or after a certain amount of data is reached, the storage node running load data and the cleaning and data migration task execution log data are used to reconstruct the storage node load prediction feature sample set and the disk false occupancy identification related data set, and the load prediction model and the anomaly detection model are incrementally trained or periodically retrained; at the same time, the parameters in the scheduling algorithm are adjusted according to the actual scheduling execution effect, for example, the deviation of the released space from the prediction, whether the load control meets the standard, to adjust the safety threshold, the storage node weight, and the task priority rule.

[0088] Embodiment 2:

[0089] Referring to Figure 2 The embodiment provides a distributed storage node load prediction dynamic scheduling method, which comprises the following steps:

[0090] Real-time statistics of disk data of each storage node are performed, and a file attribute table is constructed;

[0091] The file attribute table is split according to directory groups to obtain a plurality of directory groups, and data subsets of the directory groups are formed;

[0092] Based on the data subsets of the directory groups and pre-labeled file sample data, a local anomaly detection model of each directory group is constructed;

[0093] The local anomaly detection model is applied to file data of each directory group to perform anomaly determination, and a candidate false file list is obtained;

[0094] The candidate false file list is calibrated to obtain a false occupancy file list;

[0095] The load conditions of each storage node are predicted to obtain node load prediction results of a plurality of prediction time slices in the future, and the cleaning task is dynamically scheduled in combination with the false occupancy file list.

[0096] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0097] Finally: the above only for the preferred embodiments of the present application, and not for limiting the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application, should be included in the scope of protection of the present application.

Claims

1. A distributed storage node load prediction and dynamic scheduling method, characterized in that, include: Real-time statistics of disk data on each storage node are collected to construct a file attribute table; The file attribute table is split according to directory groups to obtain multiple directory groups, and data subsets are formed for each directory group; Based on the data subsets of each directory group and the pre-labeled file sample data, a local anomaly detection model for each directory group is constructed. The local anomaly detection model is applied to the file data of each directory group to determine anomalies and obtain a list of candidate fake files. The candidate list of fake documents is calibrated to obtain a list of fake occupied documents; The load of each storage node is predicted to obtain the node load prediction results for multiple future time slices. Combined with the list of false occupied files, the cleanup tasks are dynamically scheduled.

2. The distributed storage node load prediction and dynamic scheduling method according to claim 1, characterized in that, The directory characteristics of each directory in the statistical file attribute table are analyzed, and the directories are clustered hierarchically based on the directory characteristics to obtain multiple directory groups.

3. The distributed storage node load prediction and dynamic scheduling method according to claim 2, characterized in that, Directory characteristics include the number of files in each directory, total size, average file size, percentage of unaccessed files, and percentage of unrelated files.

4. The distributed storage node load prediction and dynamic scheduling method according to claim 3, characterized in that, Based on the number of files and the total size, the directory is initially divided into a large file directory and other directories. Construct a directory feature vector using directory features and then normalize it. Based on the directory feature vectors, the remaining directories are clustered to obtain multiple directory groups divided according to directory features.

5. The distributed storage node load prediction and dynamic scheduling method according to claim 1, characterized in that, When analyzing the disk data of each storage node, the disk usage difference of each storage node is calculated by subtracting the real-time effective data storage from the total real-time disk usage of each storage node. This difference is used to characterize the size of the potential false occupied space on each storage node.

6. The distributed storage node load prediction and dynamic scheduling method according to claim 5, characterized in that, Storage nodes whose disk usage difference as a percentage of the total real-time disk usage exceeds a preset first threshold are marked as false space usage risk nodes, and file attribute tables are only constructed for files on false space usage risk nodes.

7. The distributed storage node load prediction and dynamic scheduling method according to claim 1, characterized in that, The original anomaly measure of each file in each directory group is calculated by using a local anomaly detection model and then normalized to obtain an anomaly score for each file. When a file's anomaly score is greater than or equal to the anomaly threshold, the file is determined to be an anomaly file. When the abnormal score of a file is less than the abnormal threshold, the file is determined to be a normal file; The anomaly threshold is determined by the local anomaly detection model.

8. The distributed storage node load prediction and dynamic scheduling method according to claim 7, characterized in that, Calculate the difference between the abnormal score and the abnormal threshold of the abnormal file, and record it as the abnormal difference value; mark the abnormal files with an abnormal difference value greater than or equal to the preset high priority deviation threshold as priority cleanup objects, and mark the abnormal files with an abnormal difference value less than or equal to the preset edge deviation threshold as secondary suspicious files; Other abnormal files are not marked.

9. The distributed storage node load prediction and dynamic scheduling method according to claim 8, characterized in that, Calculate the upper limit of cleanable data in the predicted time slice and determine whether the predicted time slice is an executable cleanup time slice; Within any executable cleanup time slice on the same storage node, tasks in the priority cleanup object task queue are prioritized. If there is still a remaining amount of cleanupable data in the executable cleanup time slice and no priority cleanup object can be assigned, secondary suspicious files are allowed to enter the executable cleanup time slice.

10. A distributed storage node load prediction dynamic scheduling system, used to implement the distributed storage node load prediction dynamic scheduling method according to any one of claims 1-9, characterized in that, include: The data acquisition module is used to collect disk data from each storage node in real time and build a file attribute table. The directory grouping module is used to split the file attribute table according to directory groups, resulting in multiple directory groups and forming a data subset for each directory group; The model building module constructs local anomaly detection models for each directory group based on data subsets of each directory group and pre-labeled file sample data. The anomaly detection module is used to apply the local anomaly detection model to the file data of each directory group to perform anomaly detection and obtain a list of candidate fake files. The file calibration module is used to calibrate the candidate fake file list to obtain a fake occupied file list; The dynamic scheduling module is used to predict the load of each storage node, obtain the node load prediction results for multiple future time slices, and dynamically schedule the cleanup tasks in conjunction with the list of false occupied files.

Images