Data access prediction scheduling method and system based on dynamic threshold

Abstract

The present application relates to a dynamic threshold-based data access prediction scheduling method and system, belonging to the technical field of data processing. The abnormal detection threshold in the access behavior is dynamically adjusted by using reinforcement learning and error feedback mechanism, combined with abnormal point classification processing and access trend prediction, to realize the pre-identification of potential high-frequency access data, and generate a scheduling request to drive the cache preheating or interface priority scheduling of data. The present application combines access behavior modeling, abnormal identification, trend prediction and strategy feedback mechanism, and faces the interface access optimization demand in the mass data storage system, to solve the key problems such as unpredictable access behavior and lagging scheduling response.

Description

Technical Field

[0001] This invention relates to a data access prediction and scheduling method and system based on dynamic thresholds, belonging to the field of data processing technology. Background Technology

[0002] With the ever-increasing demand for long-term storage of large-scale data from national supercomputing centers and scientific research data platforms, long-term data storage systems face numerous challenges, including massive data volumes, extremely long storage periods, significant bursts and uncertainties in access behavior, and multi-layered heterogeneous storage media architectures. Data may experience sudden, discontinuous, high-density access requests. These access behaviors often lack a fixed periodicity and are difficult to identify and handle using static strategies. Traditional anomaly detection methods generally rely on static thresholds (such as setting a fixed upper limit on access frequency), lacking awareness of time variations, user behavior patterns, and system states, easily leading to missed or false positives, thus affecting system preprocessing efficiency, scheduling strategy execution, and interface service capabilities.

[0003] Recent studies have explored prediction-driven data preheating scheduling, which involves predicting access hotspots and preloading target data onto high-performance media to improve response speed. However, most of these studies rely on static scoring rules or model outputs for scheduling decisions, lacking a feedback mechanism that links with system performance (such as cache hit rate and interface latency), making it difficult to adapt to dynamically changing access behaviors and diverse scheduling objectives.

[0004] Therefore, an intelligent mechanism is needed that integrates data anomaly identification, access trend prediction, and scheduling signal generation to identify potential high-frequency access data in advance and achieve intelligent preheating; at the same time, the strategy parameters should be optimized by combining feedback indicators during system operation, thereby improving the scheduling response efficiency of the data storage system in the event of sudden access. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a data access prediction and scheduling method and system based on dynamic thresholds. It combines access behavior modeling, anomaly identification, trend prediction, and policy feedback mechanisms to meet the interface access optimization needs of massive data storage systems and solve key problems such as unpredictable access behavior and delayed scheduling response.

[0006] The core of this invention lies in: using reinforcement learning and error feedback mechanisms to dynamically adjust the anomaly detection threshold in access behavior, combining anomaly point classification processing and access trend prediction to achieve pre-identification of potentially high-frequency access data, and generating scheduling requests to drive data cache preheating or interface priority scheduling.

[0007] Terminology Explanation:

[0008] Anomalies are records in the data access behavior logs where the access patterns (such as frequency, time distribution, interface response characteristics, etc.) deviate significantly from the system's historical behavior model. Such deviations may be caused by abnormal task scheduling, log collection failures, unexpected user operations, or sudden bursts of real load. Anomalies may be invalid noise or carry critical information, and have an important impact on system performance modeling and scheduling strategies. They need to be identified and classified.

[0009] Anomaly candidate samples: These refer to the preliminary anomaly identification of data samples based on the Z-score method during the data preprocessing stage. The identification results are not used as the final judgment criteria. They are mainly used to eliminate extreme noise, assist in building the training set of the One-Class SVM model, and are used for subsequent fine-grained classification of anomalies.

[0010] One-Class SVM: An unsupervised anomaly detection method based on support vector machines. It primarily learns the boundary distribution of normal samples in the training set to construct a decision region containing most normal samples, thereby identifying anomalous data deviating from this region. Radial basis function kernels are often used to enhance the model's nonlinear fitting capability.

[0011] Reinforcement learning is a machine learning method that optimizes anomaly detection threshold adjustment strategies through continuous interaction between a policy decision-making module and the environment. Its core mechanism includes: the policy decision-making module selects actions based on the current state, the environment provides feedback reward signals based on the execution results, and the policy is iteratively updated through a value function, ultimately achieving autonomous optimization of the objective.

[0012] Rules: In this invention, rules refer to predefined policy conditions based on business logic or statistical constraints, used to impose soft / hard constraints on the reinforcement learning action space.

[0013] Scheduling request: refers to the migration trigger command generated by the system after the scheduling score exceeds the threshold. It contains metadata such as data identifier, target cache path, and scheduling window time, and serves as the input for the preheating scheduling process.

[0014] The technical solution of the present invention is as follows:

[0015] The data access prediction and scheduling method based on dynamic thresholds comprises the following steps:

[0016] Step 1: Data Collection: Collect behavioral data periodically or in real time from access logs, API call records, and file metadata information of the data storage system through interface protocols such as RESTful API, SFTP, and Kafka.

[0017] Step 2: Data Preprocessing: The collected raw access behavior data is cleaned and standardized, including deduplication, missing field imputation, and initial anomaly screening. The Z-score method is used to identify high-deviation access behaviors such as abnormal frequency and sudden request bursts, and these are marked as anomaly candidate samples for retention in subsequent normal sample screening and anomaly classification. Simultaneously, the feature data is enhanced by combining access interface status, access source type, and other access behavior context information to support subsequent anomaly detection.

[0018] Step 3: Data Anomaly Detection: A support vector machine model with one-class SVM unsupervised learning is used to model the access behavior and build an anomaly detection model. The boundary is constructed by learning the normal access pattern and potential abnormal behavior that deviates from the boundary is identified. The initial anomaly threshold is set based on the "mean ± k × standard deviation" of the anomaly score. In order to improve the stability of the model, only normal samples selected in the preprocessing stage are used for training and suspicious access points are removed.

[0019] Step 4: Dynamic adjustment of outlier identification threshold: An adaptive strategy optimization mechanism based on reinforcement learning is used to dynamically adjust the outlier identification threshold according to model feedback. This strategy combines the prediction error trend, feedback reward value and rules (such as peak period constraints and access periodicity) to jointly determine the direction and magnitude of threshold adjustment.

[0020] Step 5: Anomaly Classification and Handling: Input the anomalies detected in Step 3 into the anomaly classification model. Combine the original access features, context information and anomaly scores to further refine the classification into noise anomalies and real anomalies. Noise anomalies are deleted or repaired by interpolation, while real anomalies are retained and explicitly labeled to improve the predictive model's adaptability in sudden scenarios.

[0021] Step 6: Access Trend Prediction and Feedback: After anomaly handling, machine learning or deep learning models are used to model and predict access behavior trends, outputting access frequency / probability prediction results. The system monitors the error between the prediction results and actual behavior, and determines whether feedback correction is needed based on a time + event dual-trigger mechanism. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete closed-loop optimization.

[0022] Step 7: Scheduling Request Generation: Based on the output access probability or frequency results, determine whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to trigger the data object to be preloaded to high-performance media, thereby improving the response speed of potential access requests and the system's preprocessing capabilities. The scheduling request content includes information such as the target data ID, scheduling trigger time, and cache loading path.

[0023] Preferably, in step 1, the behavioral data includes, but is not limited to: access frequency, access time, user ID, interface response latency, and data location.

[0024] Preferably, in step 2, the steps for identifying and marking outlier data are as follows:

[0025] Step 2.1: Z-score anomaly score calculation. For each behavioral data point, the standard deviation of the numerical field is calculated using the Z-score method. This measures the degree of deviation of the behavioral data from the normal behavioral distribution. The Z-score calculation formula is:

[0026]

[0027] Where, x i This represents the feature value of the field in the current behavioral data, μ represents the mean of the field, and σ represents the standard deviation;

[0028] Step 2.2: Anomaly Candidate Sample Identification. When the absolute value of the Z-score of one or more fields in a behavioral data point exceeds a set threshold, i.e. If the value is >3, it is preliminarily determined that the behavioral data is abnormal and it is marked as an abnormal candidate sample. This marking is not directly used for subsequent One-Class SVM model training, but is used to exclude obvious abnormal points, assist in building a high-quality training set, and participate in the initial abnormal boundary and threshold setting of the subsequent model.

[0029] Step 2.3: Contextual information enhancement. Further introduce access behavior contextual information, including interface type, access source, user identity tags, etc., and fuse it with the original behavior data through field concatenation or feature fusion to generate composite features with semantic enhancement, which are used to improve the accuracy of subsequent fine-grained classification and identification of anomalies.

[0030] In step 3, the construction and execution steps of the anomaly detection model are as follows:

[0031] Step 3.1: Construct training data. Use the unlabeled behavioral data after preprocessing in Step 2 as the input for model training. These data are considered as samples representing normal access patterns. The behavioral data includes fields such as access frequency, interface response latency, and user access role. The behavioral data marked as abnormal candidate samples in Step 2 are not directly input into the model. Instead, they are used as auxiliary information to participate in the selection of training samples and the initial threshold estimation to exclude extreme behavioral records that may seriously deviate from the access pattern, thereby improving the model's detection performance and stability.

[0032] Step 3.2: Construct an anomaly detection model, using a support vector machine model based on unsupervised learning (One-Class SVM) as the anomaly detector to learn the boundary distribution of typical access behaviors in the data system;

[0033] Step 3.3: Setting the initial anomaly threshold. In the initial stage, statistical methods are used to analyze the anomaly scores and set the initial anomaly threshold. This initial threshold is not only used to perform preliminary anomaly identification on the data access behavior collected in the early stage, but also to provide a basic reference for the subsequent dynamic threshold adjustment mechanism.

[0034] Step 3.4: Anomaly detection is performed. The trained anomaly detection model is used to score the new input behavioral data for anomalies. The anomaly scores are compared with the initial anomaly threshold, and a label is output to indicate whether each piece of behavioral data is abnormal. This label serves as the basis for deciding whether to proceed to the next prediction step.

[0035] Preferably, the specific implementation process of step 3.2 is as follows:

[0036] Step 3.21: Selecting the kernel function: The Radial Basis Function (RBF) is selected. This kernel function has good nonlinear mapping ability and can effectively construct decision boundaries under conditions of complex data distribution and ambiguous boundaries. Its form is as follows:

[0037]

[0038] Where: x i and x j represents two distinct sample vectors in the input sample space; γ represents the kernel function width parameter, controlling the scale of the nonlinear mapping; ||x|| i -x j || 2 Represents vector x i With x j The Euclidean square distance; exp() represents the natural exponential function;

[0039] Step 3.22, Parameter Settings:

[0040] Set the hyperparameter nu to represent the maximum allowable anomaly ratio. The preferred range is 0.01-0.10, and the default value is 0.05. This is used to control the model's tolerance for outliers. The smaller the value, the stricter the judgment criteria.

[0041] Set gamma as the width parameter of the radial basis kernel function, preferably set to 1 / n_features, where n_features represents the number of feature dimensions, to ensure that the model achieves a balance between sensitivity and stability. To improve model performance, the gamma parameter is optimized through grid search, with the search space being logarithmically uniformly distributed (e.g., [0.001, 10]). Cross-validation is used, with the validation set F1-score as the optimization objective.

[0042] Step 3.23, Training Process: The anomaly detection model is trained using sample data from normal access patterns. The optimal hyperplane boundary is learned through the following objective function to encompass most normal access behavior records:

[0043]

[0044] constraint:

[0045]

[0046] Where: w is the hyperplane normal vector, controlling the direction and position of the hyperplane; ξ i ρ is a slack variable, representing the error tolerance of sample i; ρ is the offset of the decision boundary. This represents the kernel mapping function.

[0047] Preferably, in step 3.3, the initial threshold setting steps are as follows:

[0048] Step 3.31, Anomaly Score Acquisition: Using the trained anomaly detection model, calculate the corresponding anomaly score for all samples in the validation set. 'decision_function' represents the signed distance from the sample to the decision boundary, and these scores are treated as a set of one-dimensional numerical data, denoted as:

[0049]

[0050] in, This represents the anomaly score of the nth data sample;

[0051] Step 3.32, Statistical Analysis:

[0052] Calculate the mean μ and standard deviation σ of the outlier scores;

[0053] The initial outlier threshold θ is defined based on the strategy of mean ± k × standard deviation, as shown in the following formula:

[0054]

[0055] Among them, parameter k is a hyperparameter, initially set to 3, which controls the strictness of anomaly detection;

[0056] Step 3.33, Judgment Strategy: For any test sample x, if its abnormal score... < If the value is 0, it is considered an anomaly; otherwise, it is considered a normal point.

[0057] Preferably, in step 4, the specific implementation steps are as follows:

[0058] Step 4.1: State space construction, defining the state vector S of the current environment. t The state vector, used as input to the reinforcement learning module, includes, but is not limited to, the following four dimensions: the current outlier identification threshold θ. t The prediction error metrics MAPE and MSE within the sliding window will be described in step 6 later; the currently detected FP t and FN t ; Cache hit rate for the current cycle , = Number of hits / Total number of accesses; where the number of hits represents the number of times a user's access request hits the cache within the current statistical period. That is, if the access request for the data object is successfully responded to directly from the cache without needing to be read from the underlying cold storage medium, it is counted as one hit.

[0059] FP t This represents the proportion of normal samples that are identified as abnormal at the current anomaly detection threshold; FN t The specific formula for calculating the proportion of abnormal samples that are missed as normal samples under the current anomaly detection threshold is as follows:

[0060]

[0061]

[0062] Wherein, TP (True Positive Instance): represents a sample that is actually abnormal and is correctly identified as abnormal by the model; TN (True Negative Instance): represents a sample that is actually normal and is correctly identified as normal by the model; FP (False Positive Instance): represents a sample that is actually normal but is incorrectly identified as abnormal by the model; FN (False Negative Instance): represents a sample that is actually abnormal but is incorrectly identified as normal by the model.

[0063] The state vector above can reflect the system's current detection performance, prediction results, and interface operation status, serving as the perceptual basis for adjusting reinforcement learning strategies.

[0064] Step 4.2: Action Space Setting. This step sets a set of executable threshold adjustment actions to adjust the current anomaly detection threshold. It includes the following basic operations:

[0065] Basic action type: Increase threshold, Reduce the threshold. Maintain the current threshold, 0;

[0066] in, The step size is a configurable parameter, initially set to a fixed value proportional to the range of the threshold standard deviation. , where σ t Let be the threshold standard deviation at any time t;

[0067] To improve the system's adaptability to data fluctuations, a dynamic step size adjustment mechanism based on prediction error fluctuations is introduced to achieve sensitivity control of action adjustments. The adjustment formula is as follows:

[0068]

[0069] in: : Variance of prediction error within the current sliding window; : Historical error variance mean; α: Step size adjustment factor, preferably 0.5; K: Error amplification threshold factor, preferably 2.0;

[0070] To prevent system oscillations and frequent adjustments, the step size variation range is set to... This ensures that the threshold change is both sensitive and not overly aggressive.

[0071] Step 4.3: Reward Function Design. To guide the reinforcement learning strategy in dynamically adjusting the outlier threshold while balancing prediction accuracy, detection accuracy, and system resource utilization, the following composite reward function is designed:

[0072]

[0073] Where: R t MAPE is the total reward value for the current round t. t FP is the average prediction error for the current window. t 、FN t These represent the false positive rate and false negative rate of the current anomaly detection results, respectively; CacheHit t This represents the cache hit rate for the current period; α, β, and γ are adjustable weight parameters.

[0074] Step 4.4: Policy Learning and Update. The Q-learning algorithm is used to achieve adaptive learning and dynamic updating of the outlier detection threshold adjustment strategy. During the policy learning process, the state vector S is used as the basis for... t The state-action mapping function Q(S) is constructed from reward feedback and a set of optional actions. t A t), and optimize the strategy through iterative updates;

[0075] The Q-learning update rules are as follows:

[0076]

[0077] Among them: A t Perform a threshold adjustment action (such as increasing, decreasing, or maintaining) for the current execution. The maximum Q value for all possible actions in the next state; To control the speed of integration between the old and new policies, the learning rate is preferably set to 0.1-0.5. As a discount factor, weighing current returns against future returns, an optimal value of 0.8-0.95 is preferred;

[0078] To adapt to the need for modeling more complex access behaviors in the future, this invention extends the policy learning module to a DQN form based on deep neural networks, in order to improve the generalization ability and learning efficiency under large states.

[0079] Step 4.5: Rule System Integration Mechanism. To improve adaptability and interpretability of execution strategies across different business scenarios, a configurable rule system module is introduced to perform soft constraints or legality verification on threshold adjustment actions. Rule content includes, but is not limited to:

[0080] Threshold boundary definition rules: to prevent threshold adjustments from exceeding a reasonable statistical range;

[0081] Adjust frequency limit rules: Limit the number of times the strategy action is executed per unit of time;

[0082] Scenario adaptation rules: such as restricting excessive adjustment behavior during peak access periods or important business cycles.

[0083] Preferably, in step 5, specifically: anomaly classification and processing involves fine-grained classification of the suspected abnormal behavioral data identified in step 3.4, dividing them into noise anomalies and real anomalies, and implementing differentiated processing strategies based on different types. Specifically:

[0084] An anomaly classification model is constructed, and binary classification models such as LightGBM, XGBoost or Support Vector Machine (SVM) are used to identify anomalies. The training data includes labeled anomaly samples, and the input features of the classifier include anomaly scores, access behavior context information, and current cache hit status.

[0085] Based on the classification results, the following processing strategy will be implemented:

[0086] For access behaviors identified as noise anomalies, if the log collection format is incorrect or the field is missing: discard it directly; if the behavior is an isolated deviation: perform data smoothing or sliding window statistical replacement.

[0087] For access behaviors determined to be genuinely abnormal, the log and its characteristics are retained and labeled as genuine abnormal. These are then used as key feature samples in subsequent access trend prediction models to improve the model's ability to identify and predict extreme behaviors such as sudden access events and abnormal loads.

[0088] Preferably, in step 6, specifically: model prediction and feedback, after completing the classification and processing of outliers, model training and trend prediction operations are performed on the cleaned and repaired access behavior data. In view of the characteristics of low frequency, weak periodicity but sudden fluctuations in access behavior in the data system, a deep learning model with strong time series modeling ability and long-term dependency capture ability is used for modeling.

[0089] During model training, the input sequence is constructed using historical access logs. The target variables include key indicators such as data access frequency and access density within the future time window. The training adopts a sliding time window strategy and a multi-step prediction mechanism. After training, the latest data of the current period is input into the model for prediction to obtain the access behavior trend of the future time period. In order to build a closed-loop optimization mechanism for the system, a feedback mechanism combining time-driven and event-driven approaches is introduced to continuously optimize the anomaly detection threshold and prediction strategy.

[0090] Time-driven feedback: Set a periodic evaluation window that adapts to the data scenario, preferably once a day or once a week, to avoid excessively frequent feedback due to the low frequency and sparse nature of the access behavior itself. After each period, automatically summarize the prediction error in the current window and conduct a comprehensive evaluation by combining the anomaly detection accuracy index and system operation index. If the error index is detected to continuously exceed the set threshold for multiple consecutive periods, the anomaly detection threshold strategy optimization process will be automatically triggered.

[0091] Event-driven feedback: Set up targeted anomaly monitoring mechanisms to capture drastic fluctuations in access behavior or system performance anomalies in a short period of time;

[0092] To prevent frequent policy changes or system oscillations caused by short-term error fluctuations, a cooling control mechanism is designed: after each round of threshold adjustment, a freeze window period (preferably set to 30 minutes to 2 hours) is entered, during which new feedback signals are suspended and only observation indicators continue to be accumulated. After the freeze window period ends, the need for policy adjustment is reassessed to achieve a balance between feedback response and policy stability.

[0093] Preferably, in step 7, specifically: scheduling request generation: after completing the access behavior trend prediction, a scheduling scoring function is constructed based on the prediction results and the multi-dimensional state of the current access object. This function is used to determine whether the target data has the necessity to be prioritized and loaded into the high-performance medium, so as to achieve a warm-up response to sudden access and improve interface performance. The scheduling scoring function is constructed as follows:

[0094]

[0095] Where: α, β, γ, δ, and θ are weighting parameters, adjusted according to the actual scenario; F pred This represents the probability value of the target data output by the access prediction module being accessed in the future. It can be normalized by models such as Transformer to output the predicted access frequency, and then mapped to a probability value between 0 and 1 using a Sigmoid function; D idle This indicates the storage time since the last access to the current data, calculated by the system access log statistics module; L cur This indicates the class code of the current storage medium of the data object, such as SSD=1, HDD=2, tape=3; the higher the value, the lower the performance of the current storage medium, and the greater the necessity of pre-scheduling to high-speed media.

[0096] C hit This represents the hit rate of the data in the cache system during the current period, ranging from [0, 1]. To avoid repeatedly scheduling data already in the high-speed medium, the score uses 1 - C. hit Expressing the "degree of miss," the lower the hit rate, the higher the necessity for scheduling; the formula for calculating the hit rate is: (If T) i >0), otherwise 0, where T i H represents the total number of times a data object is accessed within a statistical period. i This indicates the number of times a data object is cached; when a data object is not accessed within this period (i.e., T), the cache hit count is recorded. i When C = 0, the hit rate is C. hit Defined as 0, it indicates "missed" to avoid division by zero errors and to reflect the current state of data not being actively accessed;

[0097] Sl oad This represents the current load metric, which can comprehensively evaluate dimensions such as current I / O utilization, cache utilization, and scheduling queue length. It is normalized to a floating-point number in the range [0,1], reflecting whether the scheduling timing is suitable for preloading operations.

[0098] The higher the Score value, the higher the urgency and scheduling priority of the current data object being accessed. This value is compared with the set scheduling threshold T_sched: if Score > T_sched, a scheduling request is generated; or the data with the top-K Score values ​​among all candidate objects is selected as the scheduling target. The scheduling request includes metadata such as data identifier, target medium, and scheduling window time. The scheduling task is added to the asynchronous task queue, and the scheduling execution engine will execute the migration at an opportune time within the load-allowed window to avoid resource contention and interface jitter during peak periods.

[0099] A data access prediction and scheduling system based on dynamic thresholds includes:

[0100] Data acquisition module: used to collect behavioral data periodically or in real time from access logs, interface call records, and file metadata information of the data storage system;

[0101] Data preprocessing module: used to clean and standardize the collected raw access behavior data, including deduplication, missing field imputation, and anomaly screening. It uses the Z-score method to mark extreme access behaviors as anomaly candidate samples, and enhances the feature data by combining access behavior context information.

[0102] Data anomaly detection module: Used to model access behavior using an unsupervised learning support vector machine model, build an anomaly detection model, construct boundaries by learning normal access patterns, and identify potential abnormal behaviors that deviate from the boundaries. The initial anomaly threshold is set based on the anomaly score. To improve model stability, only normal samples selected in the preprocessing stage are used for training, and suspicious access points are removed.

[0103] Anomaly detection threshold dynamic adjustment module: Used to dynamically adjust the anomaly detection threshold based on model feedback using an adaptive policy optimization mechanism based on reinforcement learning;

[0104] Anomaly Classification and Processing Module: The detected anomalies are input into the anomaly classification model. Combining the original access features, context information, and anomaly scores, the model is further refined into noise anomalies and real anomalies. Noise anomalies are removed or repaired by interpolation, while real anomalies are retained and explicitly labeled to improve the predictive model's adaptability in sudden scenarios.

[0105] Access Trend Prediction and Feedback Module: After anomaly handling, machine learning or deep learning models are used to model and predict access behavior trends, outputting access frequency / probability prediction results. The system monitors the error between the prediction results and actual behavior, and determines whether feedback correction is needed based on a time + event dual-trigger mechanism. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete closed-loop optimization.

[0106] Scheduling request generation module: Based on the output access probability or frequency results, it determines whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, it generates a scheduling request to instruct the data object to be preloaded into the high-performance medium to improve the response speed of potential access requests and the system's preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time, and cache loading path.

[0107] The beneficial effects of this invention are as follows:

[0108] This invention introduces a dynamic threshold optimization mechanism based on reinforcement learning, which combines prediction error feedback and system operating status to achieve adaptive adjustment of the data access anomaly detection threshold. This effectively reduces the risk of false positives and false negatives, and improves detection accuracy and system robustness.

[0109] Meanwhile, this invention constructs a closed-loop process of prediction-feedback-optimization, combining access trend prediction and scheduling request generation mechanisms to support pre-loading of high-frequency data and accelerated interface operations, thereby improving the system's responsiveness and resource utilization efficiency in scenarios with sudden access surges. The overall solution possesses good scalability and cross-scenario adaptability, making it suitable for scenarios such as long-term data storage in data storage systems. Attached Figure Description

[0110] Figure 1 This is a schematic diagram of the entire data processing flow of the present invention;

[0111] Figure 2 This is a schematic diagram of the anomaly detection and classification processing logic architecture of the present invention;

[0112] Figure 3 This is a schematic diagram of the prediction error-driven dynamic threshold adjustment closed-loop mechanism of the present invention. Detailed Implementation

[0113] The present invention will be further described below with reference to the embodiments and accompanying drawings, but is not limited thereto.

[0114] Example 1:

[0115] like Figure 1 As shown, this embodiment provides a data access prediction and scheduling method based on dynamic thresholds, with the following steps:

[0116] Step 1: Data Collection: Collect behavioral data periodically or in real time from access logs, API call records, and file metadata information of the data storage system through interface protocols such as RESTful API, SFTP, and Kafka.

[0117] Behavioral data includes, but is not limited to: access frequency, access time, user ID, interface response latency, and data location.

[0118] Step 2: Data preprocessing: The collected raw access behavior data is cleaned and standardized, including deduplication, missing field filling, and anomaly screening. The Z-score method is used to mark extreme access behaviors as anomaly candidate samples. At the same time, the feature data is enhanced by combining access behavior context information (such as interface status and access source type) to support subsequent anomaly detection.

[0119] The steps for identifying outlier data and marking them as candidate samples are as follows:

[0120] Step 2.1: Z-score anomaly score calculation. For each behavioral data point, the standard deviation of the numerical field is calculated using the Z-score method. This measures the degree of deviation of the behavioral data from the normal behavioral distribution. The Z-score calculation formula is:

[0121]

[0122] Where, x i This represents the feature value of the field in the current behavioral data, μ represents the mean of the field, and σ represents the standard deviation;

[0123] Step 2.2: Anomaly Candidate Sample Identification: When the absolute value of the Z-score of one or more fields in a behavioral data point exceeds a set threshold, i.e. If the value is >3, it is preliminarily determined that the behavioral data is abnormal and it is marked as an abnormal candidate sample. This marking is not directly used for subsequent One-Class SVM model training, but is used to exclude obvious abnormal points, assist in building a high-quality training set, and participate in the initial abnormal boundary and threshold setting of the subsequent model.

[0124] Step 2.3: Contextual information enhancement. Further introduce access behavior contextual information, including interface type, access source, user identity tags, etc., and fuse it with the original behavior data through field concatenation or feature fusion to generate composite features with semantic enhancement, which are used to improve the accuracy of subsequent fine-grained classification and identification of anomalies.

[0125] Step 3: Data Anomaly Detection: An unsupervised learning support vector machine model (such as One-Class SVM) is used to model the access behavior and build an anomaly detection model. The boundary is constructed by learning the normal access pattern and potential abnormal behavior that deviates from the boundary is identified. The initial anomaly threshold is set based on the "mean ± k × standard deviation" of the anomaly score. In order to improve the stability of the model, only normal samples selected in the preprocessing stage are used for training and suspicious access points are removed.

[0126] The steps for constructing and executing the anomaly detection model are as follows:

[0127] Step 3.1: Construct training data. Use the unlabeled behavioral data after preprocessing in Step 2 as the input for model training. These data are considered as samples representing normal access patterns. The behavioral data includes fields such as access frequency, interface response latency, and user access role. The behavioral data marked as abnormal candidate samples in Step 2 are not directly input into the model. Instead, they are used as auxiliary information to participate in the selection of training samples and the initial threshold estimation to exclude extreme behavioral records that may seriously deviate from the access pattern, thereby improving the model's detection performance and stability.

[0128] Step 3.2: Construct an anomaly detection model, using a support vector machine model based on unsupervised learning (One-Class SVM) as the anomaly detector to learn the boundary distribution of typical access behaviors in the data system;

[0129] The specific implementation process is as follows:

[0130] Step 3.21: Selecting the kernel function: The Radial Basis Function (RBF) is selected. This kernel function has good nonlinear mapping ability and can effectively construct decision boundaries under conditions of complex data distribution and ambiguous boundaries. Its form is as follows:

[0131]

[0132] Where: x i and x j This represents two distinct sample vectors in the input sample space. They come from the same dataset but represent different instances of access behavior. In the radial basis function kernel, the goal is to measure the similarity between any two samples; therefore, it requires calculating the distance between pairs of input samples. Specifically: This represents the feature vector of the first sample; The smaller the Euclidean distance between the two samples, the more similar they are, and the closer the kernel function's output value is to 1. A kernel function output value closer to 1 indicates greater similarity between the two samples in the feature space, suggesting that the current sample is more likely to fall within the "normal access behavior" distribution range of the training set. Conversely, a kernel function output value closer to 0 indicates greater difference between the samples, potentially representing abnormal access behavior deviating from the normal pattern.

[0133] γ represents the kernel function width parameter, which controls the scale of the nonlinear mapping; ||x i -x j || 2 Represents vector x i With x jThe Euclidean square distance; exp() represents the natural exponential function;

[0134] Step 3.22, Parameter Settings:

[0135] Set the hyperparameter nu to represent the maximum allowable anomaly ratio. The preferred range is 0.01-0.10, and the default value is 0.05. This is used to control the model's tolerance for outliers. The smaller the value, the stricter the judgment criteria.

[0136] Set gamma as the width parameter of the radial basis kernel function, preferably set to 1 / n_features, where n_features represents the number of feature dimensions, to ensure that the model achieves a balance between sensitivity and stability. To improve model performance, the gamma parameter is optimized through grid search, with the search space being logarithmically uniformly distributed (e.g., [0.001, 10]). Cross-validation is used, with the validation set F1-score as the optimization objective.

[0137] Step 3.23, Training Process: The anomaly detection model is trained using sample data from normal access patterns. The optimal hyperplane boundary is learned through the following objective function to encompass most normal access behavior records:

[0138]

[0139] constraint:

[0140]

[0141] Where: w is the hyperplane normal vector, controlling the direction and position of the hyperplane; ξ i ρ is a slack variable, representing the error tolerance of sample i; ρ is the offset of the decision boundary. This represents the kernel mapping function.

[0142] Step 3.3: Setting the initial anomaly threshold. In the initial stage, statistical methods are used to analyze the anomaly scores and set the initial anomaly threshold. This initial threshold is not only used to perform preliminary anomaly identification on the data access behavior collected in the early stage, but also to provide a basic reference for the subsequent dynamic threshold adjustment mechanism.

[0143] The initial threshold setting steps are as follows:

[0144] 3.31 Anomaly Score Acquisition: Using the trained anomaly detection model, calculate the corresponding anomaly score for all samples in the validation set. 'decision_function' represents the signed distance from the sample to the decision boundary, and these scores are treated as a set of one-dimensional numerical data, denoted as:

[0145]

[0146] in, This represents the anomaly score of the nth data sample;

[0147] The validation set is obtained by partitioning data samples that were never roughly marked as anomalous by the Z-score (i.e., the portion considered normal). By default, it is partitioned chronologically, using 20% ​​of the most recent period's data as the validation set. This dataset is not used for model training; it is only used for statistical setting of the anomaly threshold and performance evaluation.

[0148] 3.32. Statistical Analysis:

[0149] Calculate the mean μ and standard deviation σ of the outlier scores;

[0150] The initial outlier threshold θ is defined based on the strategy of mean ± k × standard deviation, as shown in the following formula:

[0151]

[0152] Among them, parameter k is a hyperparameter, initially set to 3, which controls the strictness of anomaly detection;

[0153] 3.33 Judgment Strategy: For any test sample x, if its score is abnormal... < If the value is 0, it is considered an anomaly; otherwise, it is considered a normal point.

[0154] Step 3.4: Anomaly detection is performed. The trained anomaly detection model is used to score the new input behavioral data for anomalies. The anomaly scores are compared with the initial anomaly threshold, and a label is output to indicate whether each piece of behavioral data is abnormal. This label serves as the basis for deciding whether to proceed to the next prediction step.

[0155] Step 4: Dynamically adjust the outlier identification threshold, such as... Figure 3 As shown: An adaptive policy optimization mechanism based on reinforcement learning is used to dynamically adjust the outlier identification threshold according to the prediction model error rate and system feedback performance. This policy combines the prediction error trend, feedback reward value and rules to jointly determine the direction and magnitude of the threshold adjustment.

[0156] To enhance the adaptability and robustness of the anomaly detection module in various data scenarios, a dynamic anomaly threshold adjustment mechanism based on reinforcement learning, prediction accuracy, and system operation feedback is introduced, building upon the initial anomaly threshold setting. This mechanism dynamically adjusts the anomaly judgment criteria in real time by linking prediction accuracy and system operation feedback, thereby strengthening the model's adaptability to data fluctuations and changes in access behavior. The specific implementation steps of this dynamic adjustment strategy are as follows:

[0157] Step 4.1: State space construction. Every hour, construct the state vector S of the current environment. t The state vector, used as input to the reinforcement learning module, includes, but is not limited to, the following four dimensions: the current outlier identification threshold θ. t The prediction error metrics MAPE and MSE within the sliding window will be described in step 6 later; the currently detected FP t and FN t ; Cache hit rate for the current cycle , = Number of hits / Total number of accesses; where the number of hits represents the number of times a user's access request hits the cache within the current statistical period. That is, if the access request for the data object is successfully responded to directly from the cache without needing to be read from the underlying cold storage medium, it is counted as one hit.

[0158] FP t This represents the proportion of normal samples that are identified as abnormal at the current anomaly detection threshold; FN t The specific formula for calculating the proportion of abnormal samples that are missed as normal samples under the current anomaly detection threshold is as follows:

[0159]

[0160]

[0161] Wherein, TP (True Positive Instance): represents a sample that is actually abnormal and is correctly identified as abnormal by the model; TN (True Negative Instance): represents a sample that is actually normal and is correctly identified as normal by the model; FP (False Positive Instance): represents a sample that is actually normal but is incorrectly identified as abnormal by the model; FN (False Negative Instance): represents a sample that is actually abnormal but is incorrectly identified as normal by the model.

[0162] The state vector above can reflect the system's current detection performance, prediction results, and interface operation status, serving as the perceptual basis for adjusting reinforcement learning strategies.

[0163] Step 4.2: Action Space Setting. This step sets a set of executable threshold adjustment actions to adjust the current anomaly detection threshold. It includes the following basic operations:

[0164] Basic action type: Increase threshold, Reduce the threshold. Maintain the current threshold, 0;

[0165] in, The step size is a configurable parameter, initially set to a fixed value proportional to the range of the threshold standard deviation. , where σt Let be the threshold standard deviation at any time t;

[0166] Threshold standard deviation refers to the degree of fluctuation in the anomaly detection threshold recorded over multiple recent periods, reflecting the stability of the system's judgment over a past period. Specifically, in each judgment period, the threshold after the current execution is... Stored as historical data, with a sliding window length of w, the threshold standard deviation σ at any time t... t The calculation formula is as follows:

[0167]

[0168] in, The average threshold within the sliding window is represented by the following formula:

[0169]

[0170] To improve the system's adaptability to data fluctuations, a dynamic step size adjustment mechanism based on prediction error fluctuations is introduced to achieve sensitivity control of action adjustments. The adjustment formula is as follows:

[0171]

[0172] in: : Variance of prediction error within the current sliding window; : Historical error variance mean; α: Step size adjustment factor, preferably 0.5; K: Error amplification threshold factor, preferably 2.0;

[0173] To prevent system oscillations and frequent adjustments, the step size variation range is set to... This ensures that the threshold change is both sensitive and not overly aggressive.

[0174] Step 4.3: Reward Function Design. To guide the reinforcement learning strategy in dynamically adjusting the outlier threshold while balancing prediction accuracy, detection accuracy, and system resource utilization, the following composite reward function is designed:

[0175]

[0176] Where: R t MAPE is the total reward value for the current round t. t FP is the average prediction error for the current window. t 、FN t These represent the false positive rate and false negative rate of the current anomaly detection results, respectively; CacheHit t This represents the cache hit rate for the current period; α, β, and γ are adjustable weight parameters.

[0177] Step 4.4: Policy learning and update. The Q-learning algorithm is used to implement the adaptive learning and dynamic update of the outlier recognition threshold adjustment policy. During the policy learning process, based on the state vector S t , a state-action mapping function Q(S t , A t ) is constructed through reward feedback and the set of optional actions, and the policy is optimized through iterative updates;

[0178] The Q-learning update rule is as follows:

[0179]

[0180] Where: A t is the current action for threshold adjustment (such as increasing, decreasing, or maintaining); is the maximum Q value of all possible actions in the next state; is the learning rate, which controls the speed of integrating the old and new policies, and is preferably set to 0.1 - 0.5; is the discount factor, which weighs the current reward and future rewards, and is preferably set to 0.8 - 0.95;

[0181] To meet the requirements of modeling more complex future access behaviors, the present invention extends the policy learning module to the DQN form based on a deep neural network to improve the generalization ability and learning efficiency in large states.

[0182] Step 4.5: Rule system fusion mechanism. To improve the adaptability in different business scenarios and the interpretability of the execution policy, a configurable rule system module is introduced to perform soft constraints or legality verification on the threshold adjustment actions. The rule content includes but is not limited to:

[0183] Threshold boundary definition rule: Prevent the threshold adjustment from exceeding a reasonable statistical range; Define the allowable outlier determination threshold range as θ min ≤θ t ≤θ max , set θ min =0.2, θ max =0.8, which can be configured according to different business scenarios. If the action output by the current Q-learning model causes the adjusted threshold to exceed this boundary, the current action is rejected and the policy is rolled back to the previous valid threshold θ t-1 .

[0184] Adjustment frequency limit rule: Limit the number of policy action executions within a unit time; Set the time window T_window. Within each time window, only a maximum of one threshold adjustment operation is allowed. If the time interval between the application time of a certain action adjustment and the time of the last successful adjustment < T_window, the action is rejected, the original policy remains unchanged, and it is recorded as "suppressed during the cooling period";

[0185] Scenario adaptation rules: such as restricting excessive adjustment behavior during peak access periods or important business cycles.

[0186] Step 5: Anomaly Classification and Handling: such as Figure 2 As shown in the figure, the anomalies detected in step 3 are input into the anomaly classification model. Combining the original access features, context information, and anomaly scores, the classification is further refined into noise anomalies and real anomalies. Noise anomalies are removed or repaired by interpolation, while real anomalies are retained and explicitly labeled to improve the predictive model's adaptability in sudden scenarios.

[0187] Specifically: Anomaly classification and processing involves fine-grained classification of the suspected abnormal behavioral data identified in step 3.4, dividing them into noise anomalies and genuine anomalies, and implementing differentiated processing strategies based on the different types.

[0188] A training dataset is constructed using both manual and rule-based automatic annotation to train a binary classification model that supports fine-grained anomaly classification, such as LightGBM or SVM. Access behaviors with high field missing rates (e.g., exceeding 30%) or malformed formats are labeled as "noisy anomalies," while access behaviors with anomaly scores consistently above the detection threshold and a significant decrease in cache hit rate over multiple consecutive periods are labeled as "true anomalies."

[0189] After the model training is completed, the abnormal candidate samples in step 3 are used as input, and the classification model is used to further determine their specific types, so as to achieve fine-grained division of the abnormal points between "noise abnormalities" and "real abnormalities".

[0190] An anomaly classification model is constructed, and binary classification models such as LightGBM, XGBoost or Support Vector Machine (SVM) are used to identify anomalies. The training data includes labeled anomaly samples, and the input features of the classifier include anomaly scores, access behavior context information, and current cache hit status.

[0191] Based on the classification results, the following processing strategy will be implemented:

[0192] For access behaviors identified as noise anomalies, if the log collection format is incorrect or the field is missing: discard it directly; if the behavior is an isolated deviation: perform data smoothing or sliding window statistical replacement.

[0193] For access behaviors determined to be genuinely abnormal, the log and its characteristics are retained and labeled as genuine abnormal. These are then used as key feature samples in subsequent access trend prediction models to improve the model's ability to identify and predict extreme behaviors such as sudden access events and abnormal loads.

[0194] Step 6: Access Trend Prediction and Feedback: After anomaly handling, machine learning or deep learning models are used to model and predict access behavior trends, outputting access frequency / probability prediction results. The system monitors the error (such as MAPE, RMSE) between the prediction results and actual behavior, and determines whether feedback correction is needed based on a time + event dual-trigger mechanism. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete closed-loop optimization.

[0195] Specifically, in model prediction and feedback, after anomaly classification and processing, model training and trend prediction are performed on the cleaned and repaired access behavior data. Given the low-frequency, weakly periodic but suddenly fluctuating access behavior characteristics of the data system, deep learning models with strong temporal modeling capabilities and long-term dependency capture abilities are employed, particularly Transformer-based structures such as Informer and Autoformer, which utilize self-attention mechanisms. These models, by introducing sparse self-attention mechanisms, multi-scale temporal feature encoding, and trend separation mechanisms, can accurately model future access trends over long historical sequences, making them suitable for directory-level or task-level data access behavior prediction tasks in data storage systems.

[0196] During model training, the input sequence is constructed using historical access logs. The target variables include key indicators such as data access frequency and access density within the future time window. The training adopts a sliding time window strategy and a multi-step prediction mechanism. After training, the latest data of the current period is input into the model for prediction to obtain the access behavior trend of the future time period. In order to build a closed-loop optimization mechanism for the system, a feedback mechanism combining time-driven and event-driven approaches is introduced to continuously optimize the anomaly detection threshold and prediction strategy.

[0197] Time-driven feedback: Set a periodic evaluation window that adapts to the data scenario, preferably once a day or once a week, to avoid excessively frequent feedback due to the low frequency and sparse nature of the access behavior itself. After each period, automatically summarize the prediction error in the current window and conduct a comprehensive evaluation by combining the anomaly detection accuracy index and system operation index. If the error index is detected to continuously exceed the set threshold for multiple consecutive periods, the anomaly detection threshold strategy optimization process will be automatically triggered.

[0198] Event-driven feedback: Set up a targeted anomaly monitoring mechanism to capture drastic fluctuations in access behavior or system performance anomalies in the short term. For example, if the prediction error rises sharply, the access behavior deviates from the model fitting trend, or the cache hit rate drops sharply within a short window, the system will immediately trigger a rapid feedback response, call the reinforcement learning module to dynamically optimize the current anomaly detection threshold, and realize adaptive strategy updates to improve the system's responsiveness under sudden load scenarios.

[0199] To prevent frequent policy changes or system oscillations caused by short-term error fluctuations, a cooling control mechanism is designed: after each round of threshold adjustment, a freeze window period (preferably set to 30 minutes to 2 hours) is entered, during which new feedback signals are suspended and only observation indicators continue to be accumulated. After the freeze window period ends, the need for policy adjustment is reassessed to achieve a balance between feedback response and policy stability.

[0200] Step 7: Scheduling Request Generation: Based on the output access probability or frequency results, determine whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to instruct the data object to be preloaded to high-performance media in order to improve the response speed of potential access requests and the system's preprocessing capabilities. The scheduling request content includes information such as the target data ID, scheduling trigger time, and cache loading path.

[0201] Specifically: Scheduling request generation: After predicting access behavior trends, a scheduling scoring function is constructed based on the prediction results and the multi-dimensional state of the currently accessed object. This function is used to determine whether the target data is necessary to be loaded into high-performance media (such as SSDs or caches) first, in order to achieve a warm-up response to sudden access and improve interface performance. The scheduling scoring function is constructed as follows:

[0202]

[0203] Where: α, β, γ, δ, and θ are weighting parameters, adjusted according to the actual scenario; F pred This represents the probability value of the target data output by the access prediction module being accessed in the future. It can be normalized by models such as Transformer to output the predicted access frequency, and then mapped to a probability value between 0 and 1 using a Sigmoid function; D idle This indicates the storage time since the last access to the current data, calculated by the system access log statistics module; L cur This indicates the class code of the current storage medium of the data object, such as SSD=1, HDD=2, tape=3; the higher the value, the lower the performance of the current storage medium, and the greater the necessity of pre-scheduling to high-speed media.

[0204] Lcur represents the storage medium class code of the current data object, used to measure the read / write performance and access latency of the storage medium it resides in. The specific definition is as follows:

[0205] SSD (Solid State Drive): Lcur = 1, indicating the highest performance hot data storage medium;

[0206] HDD (Hard Disk Drive): Lcur = 2, indicating a medium-performance warm data storage medium.

[0207] Magnetic tape: Lcur = 3, indicating the lowest performance data medium;

[0208] The higher the value of Lcur, the lower the access performance of the current storage medium and the higher the data response latency. Therefore, the higher the priority that the system determines is "needs to be pre-scheduled to high-performance storage media".

[0209] C hit This represents the hit rate of the data in the cache system during the current period, ranging from [0, 1]. To avoid repeatedly scheduling data already in the high-speed medium, the score uses 1 - C. hit Expressing the "degree of miss," the lower the hit rate, the higher the necessity for scheduling; the formula for calculating the hit rate is: (If T) i >0), otherwise 0, where T i H represents the total number of times a data object is accessed within a statistical period. i This indicates the number of times a data object is cached; when a data object is not accessed within this period (i.e., T), the cache hit count is recorded. i When C = 0, the hit rate is C. hit Defined as 0, it indicates "missed" to avoid division by zero errors and to reflect the current state of data not being actively accessed;

[0210] Sl oad This represents the current load metric, which can comprehensively evaluate dimensions such as current I / O utilization, cache utilization, and scheduling queue length. It is normalized to a floating-point number in the range [0,1], reflecting whether the scheduling timing is suitable for preloading operations.

[0211] A higher Score value indicates a higher urgency and scheduling priority for accessing the current data object. This value is compared with a set scheduling threshold T_sched: if Score > T_sched (default is 0.8), a scheduling request is generated; or the top-K (default K=10) Score values ​​from all candidate objects are selected as the scheduling target. The scheduling request includes metadata such as data identifier, target medium, and scheduling window time. The scheduling request is written to the asynchronous task queue. The scheduling engine periodically scans the task queue and, based on the current running status, such as I / O load, CPU utilization, and cache free capacity, determines whether execution conditions are met. If current system resources allow, the scheduling engine triggers a data migration operation, preloading data from cold storage to high-performance media to ensure critical data is hot-backed before future access, thus optimizing access performance in advance.

[0212] Example 2:

[0213] This embodiment provides a data access prediction and scheduling system based on dynamic thresholds, including:

[0214] Data acquisition module: used to collect behavioral data periodically or in real time from access logs, interface call records, and file metadata information of the data storage system;

[0215] Data preprocessing module: used to clean and standardize the collected raw access behavior data, including deduplication, missing field imputation, and anomaly screening. It uses the Z-score method to mark extreme access behaviors as anomaly candidate samples, and enhances the feature data by combining access behavior context information.

[0216] Data anomaly detection module: Used to model access behavior using an unsupervised learning support vector machine model, build an anomaly detection model, construct boundaries by learning normal access patterns, and identify potential abnormal behaviors that deviate from the boundaries. The initial anomaly threshold is set based on the anomaly score. To improve model stability, only normal samples selected in the preprocessing stage are used for training, and suspicious access points are removed.

[0217] Anomaly detection threshold dynamic adjustment module: Used to dynamically adjust the anomaly detection threshold based on model feedback using an adaptive policy optimization mechanism based on reinforcement learning;

[0218] Anomaly Classification and Processing Module: The detected anomalies are input into the anomaly classification model. Combining the original access features, context information, and anomaly scores, the model is further refined into noise anomalies and real anomalies. Noise anomalies are removed or repaired by interpolation, while real anomalies are retained and explicitly labeled to improve the predictive model's adaptability in sudden scenarios.

[0219] Access Trend Prediction and Feedback Module: After anomaly handling, machine learning or deep learning models are used to model and predict access behavior trends, outputting access frequency / probability prediction results. The system monitors the error between the prediction results and actual behavior, and determines whether feedback correction is needed based on a time + event dual-trigger mechanism. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete closed-loop optimization.

[0220] Scheduling request generation module: Based on the output access probability or frequency results, it determines whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, it generates a scheduling request to instruct the data object to be preloaded into the high-performance medium to improve the response speed of potential access requests and the system's preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time, and cache loading path.

Claims

1. A data access prediction and scheduling method based on dynamic thresholds, characterized in that, The steps are as follows: Step 1: Data Acquisition: Collect behavioral data periodically or in real-time from the access logs, API call records, and file metadata information of the data storage system; Step 2: Data preprocessing: The collected raw access behavior data is cleaned and standardized, including deduplication, missing field imputation, and anomaly screening. The Z-score method is used to mark extreme access behaviors as anomaly candidate samples, and the feature data is enhanced by combining access behavior context information. Step 3: Data Anomaly Detection: An unsupervised learning support vector machine model is used to model the access behavior and build an anomaly detection model. Boundaries are constructed by learning normal access patterns, and potential abnormal behaviors that deviate from the boundaries are identified. The initial anomaly threshold is set based on the anomaly score. To improve the stability of the model, only normal samples selected in the preprocessing stage are used for training, and suspicious access points are removed. Step 4: Dynamic Adjustment of Anomaly Identification Threshold: An adaptive policy optimization mechanism based on reinforcement learning is used to dynamically adjust the anomaly identification threshold according to model feedback. The specific implementation steps are as follows: Step 4.1: State space construction, defining the state vector S of the current environment. t The state vector, used as input to the reinforcement learning module, includes the following four dimensions: the current outlier identification threshold θ. t Prediction error index within the sliding window; currently detected FP t and FN t ; Cache hit rate for the current period (CacheHit) t CacheHit t = Number of hits / Total number of accesses; where the number of hits represents the number of times a user's access request hits the cache within the current statistical period; FP t This represents the proportion of normal samples that are identified as abnormal at the current anomaly detection threshold; FN t The specific formula for calculating the proportion of abnormal samples that are missed as normal samples under the current anomaly detection threshold is as follows: Wherein, TP: represents a sample that is actually abnormal and is correctly identified as abnormal by the model; TN: represents a sample that is actually normal and is correctly identified as normal by the model; FP: represents a sample that is actually normal but is incorrectly identified as abnormal by the model; FN: represents a sample that is actually abnormal but is incorrectly identified as normal by the model. Step 4.2: Action Space Setting. This step sets a set of executable threshold adjustment actions to adjust the current anomaly detection threshold. It includes the following basic operations: Basic action types: Increase threshold, +Δ; Decrease threshold, -Δ; Keep current threshold, 0; Where Δ is the configurable step size, a dynamically configurable parameter, whose initial value is set to a fixed value proportional to the range of the threshold standard deviation. init =0.1*σ t , where σ t Let be the threshold standard deviation at any time t; To improve the system's adaptability to data fluctuations, a dynamic step size adjustment mechanism based on prediction error fluctuations is introduced to achieve sensitivity control of action adjustments. The adjustment formula is as follows: Where: σ e 2 σ represents the variance of the prediction error within the current sliding window. e_hist 2 : Historical error variance mean; α: Step size adjustment factor; K: Error amplification threshold factor; To prevent system oscillations and frequent adjustments, the step size variation range is set to Δ∈[0.5Δ]. init ,2Δ init ]; Step 4.3: Design the reward function, and set the following composite reward function: R t =-α·MAPE t -b·(FP t +FN t )+γ·CacheHit t Where: R t MAPE is the total reward value for the current round t. t The average prediction error for the current window; CacheHit t This represents the cache hit rate for the current period; α, β, and γ are adjustable weight parameters. Step 4.4: Policy Learning and Update. The Q-learning algorithm is used to achieve adaptive learning and dynamic updating of the outlier detection threshold adjustment strategy. During the policy learning process, the state vector S is used as the basis for... t The state-action mapping function Q(S) is constructed from reward feedback and a set of optional actions. t A t And optimize the strategy through iterative updates; The Q-learning update rules are as follows: Q(S t ,A t )←Q(S t ,A t )+α·[R t +γ·maxQ(S t+1 ,a)-Q(S t ,A t )] Among them: A t To perform the current threshold adjustment action; maxQ(S) t+1 a) represents the maximum Q value of all possible actions in the next state; α is the learning rate, which controls the speed of integration between the old and new policies; γ is the discount factor, which weighs current and future benefits. Step 4.5: Rule System Integration Mechanism. To enhance adaptability and interpretability of execution strategies across different business scenarios, a rule system module is introduced to perform soft constraints or legality verification on threshold adjustment actions. The rule content includes: Threshold boundary definition rules: to prevent threshold adjustments from exceeding the statistical range; Adjust frequency limit rules: Limit the number of times the strategy action is executed per unit of time; Scenario adaptation rules: Restrict adjustment behavior during peak access periods or important business cycles; Step 5: Anomaly Classification and Handling: Input the detected anomalies into the anomaly classification model. Combine the original access features, context information and anomaly scores to further refine the classification into noise anomalies and real anomalies. Noise anomalies are deleted or repaired by interpolation, while real anomalies are retained and explicitly labeled to improve the predictive model's adaptability in sudden scenarios. Step 6: Access Trend Prediction and Feedback: After anomaly handling, use machine learning or deep learning models to model and predict access behavior, output access frequency / probability prediction results, monitor the error between the prediction results and actual behavior, and determine whether feedback correction is needed based on the time + event dual trigger mechanism. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete the closed-loop optimization. Step 7: Scheduling Request Generation: Based on the output access probability or frequency results, determine whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to instruct the data object to be preloaded into the high-performance medium to improve the response speed of potential access requests and the system's preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time, and cache loading path.

2. The data access prediction and scheduling method based on dynamic thresholds as described in claim 1, characterized in that, In step 1, the behavioral data includes: access frequency, access time, user ID, interface response latency, and data location; In step 2, the steps for identifying and marking outlier data are as follows: Step 2.1: Z-score anomaly score calculation. For each behavioral data point, the standard deviation of the numerical field is calculated using the Z-score method. This measures the degree of deviation of the behavioral data from the normal behavioral distribution. The Z-score calculation formula is: Where, x i σ represents the feature value of the field in the current behavioral data, μ represents the mean of the field, and σ represents the standard deviation. Step 2.2: Anomaly Candidate Sample Identification: When the absolute value of the Z-score of one or more fields in a behavioral data point exceeds a set threshold, i.e., |Z i If |>3, then it is preliminarily determined that the behavioral data is abnormal, and the behavior is marked as an abnormal candidate sample; Step 2.3: Contextual information enhancement. Further introduce access behavior contextual information, and fuse it with the original behavior data through field concatenation or feature fusion to generate composite features with semantic enhancement.

3. The data access prediction and scheduling method based on dynamic thresholds as described in claim 2, characterized in that, In step 3, the construction and execution steps of the anomaly detection model are as follows: Step 3.1: Construct training data. Use the unlabeled behavioral data after preprocessing in Step 2 as the input for model training, and regard it as a sample representing the normal access mode. The abnormal candidate samples labeled in Step 2 are used as auxiliary information to participate in the selection of training samples and the initial threshold estimation. Step 3.2: Construct an anomaly detection model, using a support vector machine model based on unsupervised learning as an anomaly detector to learn the boundary distribution of typical access behaviors in the data system; Step 3.3: Setting the initial anomaly threshold. In the initial stage, statistical methods are used to analyze the anomaly scores and set the initial anomaly threshold. Step 3.4: Anomaly detection is performed. The trained anomaly detection model is used to score the new input behavioral data for anomalies. The anomaly scores are compared with the initial anomaly threshold, and a label is output to indicate whether each piece of behavioral data is abnormal. This label serves as the basis for deciding whether to proceed to the next prediction step.

4. The data access prediction and scheduling method based on dynamic thresholds as described in claim 3, characterized in that, The specific implementation process of step 3.2 is as follows: Step 3.21, Select kernel function: Choose the radial basis kernel function, which has the following form: K(x i ,x j )=exp(-γ||x i -x j || 2 ) Where: x i and x j represents two distinct sample vectors in the input sample space; γ represents the kernel function width parameter, controlling the scale of the nonlinear mapping; ||x|| i -x j || 2 Represents vector x i With x j The Euclidean square distance; exp() represents the natural exponential function; Step 3.22, Parameter Settings: Set the hyperparameter nu to represent the maximum allowable proportion of outliers. This controls the model's tolerance for outliers. The smaller the value, the stricter the judgment criteria. Set gamma as the width parameter of the radial basis kernel function; Step 3.23, Training Process: The anomaly detection model is trained using sample data from normal access patterns. The optimal hyperplane boundary is learned through the following objective function to enclose the normal access behavior records: constraint: Where: ω is the hyperplane normal vector, controlling the direction and position of the hyperplane; ξ i ρ is a slack variable, representing the error tolerance of sample i; ρ is the offset of the decision boundary. This represents the kernel mapping function.

5. The data access prediction and scheduling method based on dynamic thresholds as described in claim 4, characterized in that, In step 3.3, the initial threshold setting steps are as follows: Step 3.31, Anomaly Score Acquisition: Using the trained anomaly detection model, calculate the corresponding anomaly score for all samples in the validation set, f(x) = decision_function(x), where decision_function is the signed distance from the sample to the decision boundary. These scores are then treated as a set of one-dimensional numerical data, denoted as: S={f(x1),f(x2),…f(x n )} Where, f(x) n ) represents the anomaly score of the nth data sample; Step 3.32, Statistical Analysis: Calculate the mean μ and standard deviation σ of the outlier scores; The initial outlier threshold θ is defined based on the strategy of mean ± k × standard deviation, as shown in the following formula: θ=μ-k*σ Wherein, parameter k is a hyperparameter; Step 3.33, Judgment Strategy: For any test sample x, if its abnormal score f(x) < θ, it is judged as an abnormal point; otherwise, it is regarded as a normal point.

6. The data access prediction and scheduling method based on dynamic thresholds as described in claim 5, characterized in that, In step 5, specifically: anomaly classification and processing involves fine-grained classification of the identified suspected abnormal behavioral data, dividing it into noise anomalies and genuine anomalies, and implementing differentiated processing strategies based on the different types. Specifically: An anomaly classification model is constructed, and a binary classification model is used to identify anomalies. The training data includes labeled anomaly samples, and the input features of the classifier include anomaly scores, access behavior context information, and current cache hit status. Based on the classification results, the following processing strategy will be implemented: For access behaviors identified as noise anomalies, if the log collection format is incorrect or the field is missing: discard it directly; if the behavior is an isolated deviation: perform data smoothing or sliding window statistical replacement. For access behaviors determined to be genuine anomalies, retain the log and its characteristics, and label them as genuine anomalies.

7. The data access prediction and scheduling method based on dynamic thresholds as described in claim 6, characterized in that, In step 6, specifically: model prediction and feedback, after completing the classification and processing of outliers, model training and trend prediction operations are performed on the cleaned and repaired access behavior data. In view of the characteristics of low frequency, weak periodicity but sudden fluctuations in access behavior in the data system, a deep learning model with strong time series modeling ability and long-term dependency capture ability is used for modeling. During model training, the input sequence is constructed using historical access logs. The target variables include the data access frequency and access density within the future time window. The training adopts a sliding time window strategy and a multi-step prediction mechanism. After training, the latest data of the current period is input into the model for prediction to obtain the access behavior trend of the future time period. In order to build a closed-loop optimization mechanism for the system, a feedback mechanism combining time-driven and event-driven approaches is introduced to continuously optimize the anomaly detection threshold and prediction strategy. Time-driven feedback: Set a periodic evaluation window for the data scenario. After each period, automatically summarize the prediction error in the current window and conduct a comprehensive evaluation by combining the anomaly detection accuracy index and system operation index. If the error index is detected to continuously exceed the set threshold for multiple consecutive periods, the anomaly detection threshold strategy optimization process will be automatically triggered. Event-driven feedback: Set up targeted anomaly monitoring mechanisms to capture drastic fluctuations in access behavior or system performance anomalies in a short period of time; To prevent frequent policy changes or system oscillations caused by short-term error fluctuations, a cooling control mechanism is designed: after each round of threshold adjustment, a freeze window period is entered, during which new feedback signals are suspended and only observation indicators continue to accumulate. After the freeze window period ends, the need for policy adjustment is reassessed to achieve a balance between feedback response and policy stability.

8. The data access prediction and scheduling method based on dynamic thresholds as described in claim 7, characterized in that, In step 7, specifically: Scheduling request generation: After completing the access behavior trend prediction, a scheduling scoring function is constructed based on the prediction results and the multi-dimensional state of the current accessed object. This function is used to determine whether the target data is necessary to be prioritized and loaded into the high-performance medium, so as to achieve a warm-up response to sudden access and improve interface performance. The scheduling scoring function is constructed as follows: Score=α·F pred +β·D idle +γ·L cur +δ·(1-C hit )+θ·S load Where: α, β, γ, δ, θ are weighting parameters; F pred This represents the probability value of the target data output by the access prediction module being accessed in the future; D idle Indicates the storage time since the last access to the current data; L cur This indicates the class code of the current storage medium for the data object. The higher the value, the lower the performance of the current storage medium, and the greater the necessity for pre-scheduling to a high-speed medium. C hit This indicates the hit rate of the data in the cache system during the current period. To avoid repeatedly scheduling data already in high-speed media, a 1-C score is used. hit Expressing the "degree of miss," the lower the hit rate, the higher the scheduling necessity; the formula for calculating the hit rate is: C hit =H i / T i If T i >0, otherwise 0, where T i H represents the total number of times a data object is accessed within a statistical period. i This indicates the number of times the data object has been hit by the cache. Sl oad This represents the current load metric. A higher Score indicates a higher urgency and scheduling priority for accessing the current data object. This value is compared with a set scheduling threshold T_sched: if Score > T_sched, a scheduling request is generated; or, the top-K data with the highest Score ranking among all candidate objects are selected as the scheduling target. The scheduling request includes the data identifier, target medium, and scheduling window time. The scheduling task is added to the asynchronous task queue, and the scheduling execution engine performs the migration within the load-allowed window.

9. A data access prediction and scheduling system based on dynamic thresholds, applied to the data access prediction and scheduling method based on dynamic thresholds as described in claim 1, characterized in that, include: Data acquisition module: used to collect behavioral data periodically or in real time from access logs, interface call records, and file metadata information of the data storage system; Data preprocessing module: used to clean and standardize the collected raw access behavior data, including deduplication, missing field imputation, and anomaly screening. It uses the Z-score method to mark extreme access behaviors as anomaly candidate samples, and enhances the feature data by combining access behavior context information. Data anomaly detection module: Used to model access behavior using an unsupervised learning support vector machine model, build an anomaly detection model, construct boundaries by learning normal access patterns, and identify potential abnormal behaviors that deviate from the boundaries. The initial anomaly threshold is set based on the anomaly score. To improve model stability, only normal samples selected in the preprocessing stage are used for training, and suspicious access points are removed. Anomaly detection threshold dynamic adjustment module: Used to dynamically adjust the anomaly detection threshold based on model feedback using an adaptive policy optimization mechanism based on reinforcement learning; Anomaly Classification and Processing Module: The detected anomalies are input into the anomaly classification model. Combining the original access features, context information and anomaly scores, the model is further refined into noise anomalies and real anomalies. Noise anomalies are deleted or repaired by interpolation, while real anomalies are retained and explicitly labeled to improve the predictive model’s adaptability in sudden scenarios. Access trend prediction and feedback module: After anomaly handling, machine learning or deep learning models are used to model and predict access behavior, output access frequency / probability prediction results, the system monitors the error between the prediction results and the actual behavior, and judges whether feedback correction is needed based on the time + event dual trigger mechanism. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete the closed-loop optimization. Scheduling request generation module: Based on the output access probability or frequency results, it determines whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, it generates a scheduling request to instruct the data object to be preloaded into the high-performance medium to improve the response speed of potential access requests and the system's preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time, and cache loading path.

Images