Data access prediction scheduling method and system based on dynamic threshold

Abstract

The invention relates to a data access prediction scheduling method and system based on a dynamic threshold value, and belongs to the technical field of data processing. The method comprises the following steps of: dynamically adjusting an abnormal detection threshold in an access behavior by utilizing reinforcement learning and an error feedback mechanism, realizing preposition identification of potential high-frequency access data by combining abnormal point classification processing and access trend prediction, and generating a scheduling request to drive cache preheating or interface priority scheduling of the data. According to the method, access behavior modeling, anomaly recognition, trend prediction and strategy feedback mechanisms are combined, interface access optimization requirements in a mass data storage system are met, and the key problems that access behaviors are unpredictable, scheduling response lags and the like are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to a data access prediction scheduling method and system based on dynamic threshold, belonging to the technical field of data processing. BACKGROUND

[0002] With the growing demand for long-term storage of large-scale data in national supercomputing centers and scientific data platforms, data long-term storage systems face many challenges, including large data volume, long storage period, burstiness and obvious uncertainty of access behavior, and multi-layer heterogeneous architecture of storage media. Data may experience bursty, non-continuous, and high-density access requests. These access behaviors often do not have fixed periodicity and are not easily identified and processed by static strategies. Traditional anomaly detection methods generally rely on static thresholds (such as setting a fixed upper limit on access frequency), lack of perception of time variation, user behavior patterns, and system state, and are prone to false negatives and false positives, which can affect system preprocessing efficiency, scheduling strategy execution, and interface service capability.

[0003] In recent years, some research has driven data preheating scheduling by predicting access hotspots and preloading target data to high-performance media to improve response speed. However, most of them rely on static scoring rules or model output results for scheduling decisions, lack feedback mechanisms linked to system running results (such as cache hit rate and interface latency), and are difficult to adapt to dynamically changing access behaviors and diverse scheduling goals.

[0004] Therefore, there is a need for an intelligent mechanism that integrates data anomaly identification, access trend prediction, and scheduling signal generation to identify potential high-frequency access data in advance and implement intelligent preheating. At the same time, by combining feedback indicators in system operation to optimize strategy parameters, the scheduling response efficiency of data storage systems in bursty access scenarios can be improved. SUMMARY

[0005] To address the shortcomings of existing technologies, the present application provides a data access prediction scheduling method and system based on dynamic threshold, which combines access behavior modeling, anomaly identification, trend prediction, and strategy feedback mechanisms to address key issues such as unpredictable access behavior and scheduling response lag in interface access optimization requirements for massive data storage systems.

[0006] The core of the present application is to dynamically adjust the anomaly detection threshold in access behavior using reinforcement learning and error feedback mechanisms, combine anomaly point classification processing and access trend prediction, and implement pre-identification of potential high-frequency access data, and generate scheduling requests to drive cache preheating or interface priority scheduling of data.

[0007] Terminology explanation: Anomaly point: refers to the record in the data access behavior log, the access mode (such as frequency, time distribution, interface response characteristics, etc.) of which deviates significantly from the system historical behavior model, such deviation may be caused by abnormal task scheduling, log collection failure, unexpected user operation or real load burst, the anomaly point may be invalid noise, or may carry key information, has an important influence on system performance modeling and scheduling strategy, and needs to be identified and classified.

[0008] Anomaly candidate sample: refers to the preliminary anomaly identification of data samples based on the Z-score method in the data preprocessing stage, the identification result is not used as the final judgment standard, mainly used for excluding extreme noise, assisting in building the training set of One-Class SVM model, and used for subsequent fine-grained classification of anomaly points.

[0009] One-Class SVM: an unsupervised anomaly detection method based on support vector machine, which mainly learns the boundary distribution of normal samples in the training set, constructs a decision region containing most of the normal samples, and identifies the abnormal data deviating from the region. Radial basis kernel function is often used to enhance the nonlinear fitting ability of the model.

[0010] Reinforcement learning: a machine learning method for optimizing anomaly detection threshold adjustment strategy through continuous interaction between policy decision module and environment, the core mechanism of which includes: the policy decision module selects actions based on the current state, the environment feeds back reward signals based on the execution results, and the policy is updated through value function iteration, and finally realizes the autonomous optimization of the goal.

[0011] Rule: in the present application, it refers to a policy condition defined in advance based on business logic or statistical constraints, which is used to soft / hardly constrain the action space of reinforcement learning.

[0012] Scheduling request: refers to the migration trigger command generated by the system after the scheduling score exceeds the threshold, including data identification, target cache path, scheduling window time and other meta information, as the input of the preheating scheduling process.

[0013] The technical scheme of the present application is as follows: The data access prediction scheduling method based on dynamic threshold is as follows: Step 1: data collection: through RESTful API, SFTP, Kafka and other interface protocols, the behavior data is collected from the access log, interface call record and file metadata information of the data storage system in time or real time; Step 2: Data preprocessing: Clean and standardize the collected raw access behavior data, including deduplication, missing field filling, and initial anomaly screening. Use the Z-score method to identify highly deviant access behaviors such as frequency anomalies and request bursts, and mark them as abnormal candidate samples for retention for subsequent normal sample screening and anomaly point classification. At the same time, combine access behavior context information such as access interface status and access source type to enhance feature data to support subsequent anomaly detection.

[0014] Step 3: Data anomaly detection: We use a one-class support vector machine (SVM) unsupervised learning model to model access behavior and construct an anomaly detection model. This model builds boundaries by learning normal access patterns and identifies potential anomalies that deviate from these boundaries. The initial anomaly threshold is set based on the mean ± k × standard deviation of the anomaly score. To improve model stability, we only use normal samples screened in the preprocessing phase for training, eliminating suspicious access points. Step 4: Dynamically adjust the outlier identification threshold: Use an adaptive strategy optimization mechanism based on reinforcement learning to dynamically adjust the outlier identification threshold based on model feedback. This strategy combines the prediction error trend, feedback reward value, and rules (such as peak period constraints and access periodicity) to determine the direction and magnitude of the threshold adjustment. Step 5: Outlier classification and processing: The outliers detected in step 3 are input into the outlier classification model. Combined with the original access features, contextual information, and anomaly scores, they are further classified into noise anomalies and true anomalies. Noise anomalies are deleted or repaired by interpolation, while true anomalies are retained and explicitly labeled to improve the adaptability of the prediction model in emergency scenarios.

[0015] Step 6: Access Trend Prediction and Feedback: After exception handling, machine learning or deep learning models are used to model access behavior and predict trends, outputting access frequency / probability predictions. The system monitors the error between the prediction and actual behavior and, based on a dual-trigger mechanism of time and events, determines whether feedback correction is needed. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module, completing the closed-loop optimization.

[0016] Step 7: Scheduling request generation: Based on the output access probability or frequency results, determine whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to trigger the preloading of the data object to high-performance media to improve the response speed of potential access requests and the system preprocessing capabilities. The scheduling request content includes information such as the target data ID, scheduling trigger time, and cache loading path.

[0017] Preferably, in step 1, the behavior data includes but is not limited to: access frequency, access time, user ID, interface response delay and data location, etc.

[0018] Preferably, in step 2, the steps of identifying outlier data and marking them are as follows: Step 2.1: Z-score anomaly score calculation. For each piece of behavioral data, the standard deviation of the numeric field is calculated based on the Z-score method to measure the degree of deviation of the behavioral data from the normal behavior distribution. The Z-score calculation formula is:

[0019] Among them, x i Represents the characteristic value of the field in the current behavior data, μ represents the mean of the field, and σ represents the standard deviation; Step 2.2: Identification of abnormal candidate samples. When the absolute value of the Z-score value of one or more fields in a certain behavioral data exceeds the set threshold, that is, If the value is >3, the behavior data is preliminarily judged to be abnormal and marked as an abnormal candidate sample. This mark is not directly used for subsequent One-Class SVM model training, but is used to exclude obvious abnormal points, assist in building a high-quality training set, and participate in the initial abnormal boundary and threshold setting of the subsequent model; Step 2.3: Contextual information enhancement: Further introduce access behavior context information, including interface type, access source, user identity label, etc., and fuse it with the original behavior data through field splicing or feature fusion to generate composite features with semantic enhancement, which is used to improve the subsequent fine-grained classification and recognition accuracy of anomaly points.

[0020] In step 3, the construction and execution steps of the anomaly detection model are as follows: Step 3.1: Construct training data. Use the unlabeled behavioral data preprocessed in Step 2 as model training input, treating it as samples representing normal access patterns. The behavioral data includes fields such as access frequency, interface response latency, and user access roles. The behavioral data marked as abnormal candidate samples in Step 2 is not directly input into the model. Instead, it is used as auxiliary information for training sample screening and initial threshold estimation to eliminate extreme behavior records that may seriously deviate from access patterns, thereby improving model detection performance and stability. Step 3.2: Build an anomaly detection model. Use the unsupervised support vector machine model (One-Class SVM) as an anomaly detector to learn the boundary distribution of typical access behaviors in the data system. Step 3.3: Initial anomaly threshold setting. In the initial stage, statistical methods are used to analyze the anomaly scores and set the initial anomaly threshold. This initial threshold is not only used to identify anomalies in the data access behavior collected in the early stage, but also provides a basic reference for the subsequent dynamic threshold adjustment mechanism. Step 3.4: Anomaly detection is performed. The trained anomaly detection model is used to score the newly input behavior data for anomalies. The anomaly score is compared with the initial anomaly threshold, and a label is output to indicate whether each behavior data is abnormal or not, which serves as the basis for entering the next prediction process.

[0021] Preferably, the specific implementation process of step 3.2 is as follows: Step 3.21. Select kernel function: Use the radial basis function (RBF). This kernel function has good nonlinear mapping capabilities and can effectively construct decision boundaries under conditions of complex data distribution and fuzzy boundaries. Its form is:

[0022] Where: x i and x j Represents two different sample vectors in the input sample space; γ represents the kernel function width parameter, which controls the scale of the nonlinear mapping; ‖x i -x j ‖ 2 Represents vector x i with x j The Euclidean square distance; exp() represents the natural exponential function; Step 3.22, parameter setting: Set the hyperparameter nu, which represents the maximum allowed anomaly ratio. The preferred range is 0.01-0.10, and the default setting is 0.05. It is used to control the model's tolerance for anomalies. The smaller the value, the stricter the judgment standard. Set gamma to the width parameter of the radial basis kernel function, preferably set to 1 / n_features, where n_features represents the number of feature dimensions, to ensure a balance between model sensitivity and stability. To improve model performance, the gamma parameter is optimized through grid search over a log-uniform search space (e.g., [0.001, 10]). Cross-validation is used, with the F1-score on the validation set as the optimization target. Step 3.23, Training Process: The anomaly detection model is trained using sample data of normal access patterns. The optimal hyperplane boundary is learned through the following objective function to encompass most normal access behavior records:

[0023] Constraints:

[0024] where: w is the hyperplane normal vector, controlling the orientation and position of the hyperplane; ξ i is the slack variable, representing the error tolerance of sample i; ρ is the offset of the decision boundary; represents the kernel mapping function.

[0025] Preferably, in step 3.3, the initial threshold setting step is as follows: Step 3.31, Abnormal Score Acquisition: Using the trained anomaly detection model, calculate the corresponding abnormal score of all samples in the validation set, , decision_function is the signed distance of the sample to the decision boundary, and these scores are regarded as a set of one-dimensional numerical data, denoted as:

[0026] where, represents the abnormal score of the nth data sample; Step 3.32, Statistical Analysis: Calculate the mean μ and standard deviation σ of the abnormal score; Define the initial abnormal threshold θ based on the mean ± k × standard deviation strategy, the formula is as follows:

[0027] where, parameter k is a hyperparameter, initially set to 3, controlling the strictness of abnormal judgment; Step 3.33, Decision Strategy: For any test sample x, if its abnormal score , it is determined as an abnormal point; otherwise, it is regarded as a normal point.

[0028] Preferably, in step 4, the specific implementation steps are as follows: Step 4.1: State Space Construction, define the state vector S t of the current environment, which is used as the input of the reinforcement learning module. The state vector includes but is not limited to the following 4 dimensions: the current abnormal point recognition threshold θ t ; the prediction error indicators MAPE and MSE within the sliding window, where the error indicators will be described in subsequent step 6; the current detected FP t and FN t ; the cache hit rate of the current period , ​= Number of hits / Total number of accesses; where the number of hits indicates the number of times a user access request hits the cache during the current statistical period. That is, a hit is counted when an access request for a data object is successfully responded to directly from the cache without reading from the underlying cold storage medium. FP t Indicates the proportion of normal samples judged as abnormal under the current anomaly detection threshold; FN t The ratio of abnormal samples that are missed as normal under the current anomaly detection threshold is calculated as follows:

[0029]

[0030] Among them, TP (True Positive): represents samples that are actually abnormal and are correctly identified as abnormal by the model; TN (True Negative): represents samples that are actually normal and are correctly identified as normal by the model; FP (False Positive): represents samples that are actually normal but are mistakenly identified as abnormal by the model; FN (Failed Negative): represents samples that are actually abnormal but are mistakenly identified as normal by the model.

[0031] The above state vector can reflect the system's current detection performance, prediction effect, and interface operation status, serving as the perception basis for reinforcement learning strategy adjustment.

[0032] Step 4.2: Action space setting: Set a set of executable threshold adjustment actions to adjust the current anomaly determination threshold, including the following basic operations: Basic action type: increase threshold, ; Decrease the threshold, ; Maintain the current threshold, 0; in, The step size is a dynamically configurable parameter whose initial value is set to a fixed value proportional to the threshold standard deviation range. , where σ t is the standard deviation of the threshold at any time t; In order to improve the system's adaptability to data fluctuations, a dynamic step adjustment mechanism based on prediction error fluctuations is introduced to achieve sensitivity control of action adjustment. The adjustment formula is as follows:

[0033] in: : The prediction error variance in the current sliding window; : historical error variance mean; α: step size adjustment factor, preferably 0.5; K: error amplification threshold factor, preferably set to 2.0; To prevent system oscillation and frequent adjustments, the step size range is set to , ensure that the threshold change is both sensitive and not overly aggressive.

[0034] Step 4.3: Reward function design, to guide the reinforcement learning strategy in the process of dynamically adjusting the anomaly point threshold, considering the prediction accuracy, detection accuracy and system resource utilization, set the following compound reward function:

[0035] Where: R t is the total reward value of the current round t; MAPE t is the average prediction error of the current window; FP t , FN t are the false positive rate and false negative rate of the current anomaly detection result respectively; CacheHit t is the current period cache hit rate; α, β, γ are adjustable weight parameters; Step 4.4: Strategy learning and updating, using Q-learning algorithm to realize adaptive learning and dynamic updating of anomaly point identification threshold adjustment strategy, in the strategy learning process, based on state vector S t , reward feedback and optional action set to build state-action mapping function Q(S t ,A t ), and realize policy optimization through iterative updating; The Q-learning update rule is as follows:

[0036] Where: A t is the current threshold adjustment action (such as increasing, decreasing or keeping); Q is the maximum Q value of all possible actions in the next state; is the learning rate, which controls the fusion speed of new and old policies, and is preferably set to 0.1-0.5; is the discount factor, which weighs current income and future income, and is preferably set to 0.8-0.95; To adapt to the modeling needs of more complex access behaviors in the future, the strategy learning module is extended to the DQN form based on deep neural network to improve the generalization ability and learning efficiency in large states.

[0037] Step 4.5: Rule system fusion mechanism, to improve the adaptability and explainability of the execution strategy in different business scenarios, introduce a configurable rule system module to soft constraint or legality verification of threshold adjustment action, the rule content includes but not limited to: Threshold boundary definition rule: to prevent threshold adjustment from exceeding the reasonable statistical range; Adjust frequency limit rules: limit the number of times a policy action is executed per unit time; Scenario adaptation rules: For example, restricting excessive adjustments during peak access periods or important business cycles.

[0038] Preferably, in step 5, specifically: outlier classification and processing, the suspected abnormal behavior data identified in step 3.4 is finely classified into noise anomalies and real anomalies, and differentiated processing strategies are implemented according to different types, specifically: Build an outlier classification model and use a binary classification model such as LightGBM, XGBoost, or Support Vector Machine (SVM) to identify outliers. The training data includes labeled outlier samples, and the classifier input features include anomaly scores, access behavior context information, and current cache hit status. Based on the classification results, the following processing strategies are implemented: For access behaviors that are determined to be noise anomalies, if the log collection format is incorrect or fields are missing, they are discarded directly. If they are isolated deviations, data smoothing or sliding window statistical replacement is performed. For access behaviors that are determined to be truly abnormal, the log and its features are retained and marked with true abnormality labels. They are used as key feature samples in subsequent access trend prediction models to improve the model's ability to identify and predict extreme behaviors such as access bursts and load anomalies.

[0039] Preferably, in step 6, specifically: model prediction and feedback, after completing the classification and processing of abnormal points, perform model training and trend prediction operations on the cleaned and repaired access behavior data. Targeting the low frequency, weak periodicity, but sudden fluctuation characteristics of access behavior in the data system, a deep learning model with strong time series modeling capabilities and long-term dependency capture capabilities is used for modeling; During model training, historical access logs are used to construct input sequences. Target variables include key indicators such as data access frequency and access density within a future time window. Training utilizes a sliding time window strategy and a multi-step prediction mechanism. After training is complete, the latest data from the current cycle is input into the model for prediction, yielding access behavior trends for future time periods. To build a closed-loop optimization mechanism for the system, a feedback mechanism combining time-driven and event-driven approaches is introduced to continuously optimize anomaly detection thresholds and prediction strategies. Time-driven feedback: Set a periodic evaluation window that adapts to the data scenario, preferably daily or weekly, to avoid excessive feedback due to the low-frequency and sparse nature of access behavior. After each cycle, the prediction error within the current window is automatically summarized and comprehensively evaluated based on the anomaly detection accuracy index and system operation indicators. If the error index is detected to exceed the set threshold for multiple consecutive cycles, the anomaly detection threshold strategy optimization process is automatically triggered. Event-driven feedback: Set up a targeted anomaly monitoring mechanism to capture sharp fluctuations in access behavior or system performance anomalies in the short term; To prevent frequent strategy changes or system oscillations caused by short-term fluctuations in errors, a cooling control mechanism is designed: after each round of threshold adjustment is completed, a freezing window period (preferably set to 30 minutes to 2 hours) is entered, during which time the acceptance of new feedback signals is suspended and only the accumulation of observation indicators continues. After the freezing window period ends, the need for strategy adjustment is reassessed to achieve a balance between feedback response and strategy stability.

[0040] Preferably, in step 7, specifically: Scheduling request generation: After completing the access behavior trend prediction, a scheduling scoring function is constructed based on the prediction results and the multi-dimensional state of the current access object to determine whether the target data has the necessity of being loaded into a high-performance medium in priority, so as to achieve a warm-up response to sudden access and improve interface performance. The scheduling scoring function is constructed as follows:

[0041] Among them: α, β, γ, δ, θ are weight parameters, which are adjusted according to the actual scenario; F pred It represents the probability value of the target data output by the access prediction module being accessed in the future. It can output the normalized predicted access frequency through models such as Transformer, and then map it to a probability value between 0 and 1 through Sigmoid. idle Indicates the storage time of the current data from the last access to the current time, which is calculated by the system access log statistics module; L cur Indicates the level code of the current storage medium of the data object, for example, SSD=1, HDD=2, tape=3. The higher the value, the lower the performance of the current storage medium, and the more necessary it is to pre-schedule to high-speed media. C hit Indicates the hit rate of the data in the cache system in the current cycle, ranging from [0, 1]. To avoid repeated scheduling of data already in high-speed media, 1 - C is used in the score. hit Expressing the "miss rate", the lower the hit rate, the higher the necessity of scheduling; the calculation formula of the hit rate is: (If T i >0), otherwise 0, where T i Indicates the total number of times the data object is accessed during the statistical period, H i Indicates the number of times a data object is cached; when a data object is not accessed during this cycle (i.e., T i =0), the hit rate C hit Defined as 0, indicating "not hit" to avoid division by zero errors and reflect the state that the data is not currently being actively accessed; Sload Indicates the current load index, which can comprehensively evaluate the current I / O utilization, cache usage, scheduling queue length and other dimensions, and is normalized to a floating point number in the range of [0,1] to reflect whether the scheduling timing is suitable for preloading operations. The higher the Score value, the higher the urgency and scheduling priority of the current data object being accessed. This value is compared with the set scheduling threshold T_sched. If Score>T_sched, a scheduling request is generated; or the data with the top-K score values ​​among all candidate objects are selected as the scheduling target. The scheduling request contains metadata such as data identification, target medium, and scheduling window time. The scheduling task is added to the asynchronous task queue, and the scheduling execution engine selects an opportunity to perform migration within the window allowed by the load to avoid resource competition and interface jitter during peak periods.

[0042] The data access prediction and scheduling system based on dynamic thresholds includes: Data collection module: used to collect behavioral data from the access logs, interface call records, and file metadata of the data storage system in a scheduled or real-time manner; Data preprocessing module: This module cleans and standardizes the collected raw access behavior data, including deduplication, missing field filling, and initial anomaly screening. It uses the Z-score method to mark extreme access behaviors as abnormal candidate samples, and enhances feature data by combining access behavior context information. Data anomaly detection module: This module uses an unsupervised support vector machine model to model access behavior and build an anomaly detection model. It establishes boundaries by learning normal access patterns and identifies potential abnormal behaviors that deviate from these boundaries. The initial anomaly threshold is set based on the anomaly score. To improve model stability, only normal samples screened in the preprocessing phase are used in training, eliminating suspicious access points. Anomaly identification threshold dynamic adjustment module: used to use the adaptive strategy optimization mechanism based on reinforcement learning to dynamically adjust the anomaly identification threshold according to model feedback; Anomaly classification and processing module: Detected anomalies are input into the anomaly classification model. Combining the original access features, contextual information, and anomaly scores, the model further refines the classification into noise anomalies and true anomalies. Noise anomalies are deleted or repaired through interpolation, while true anomalies are retained and explicitly labeled to improve the adaptability of the prediction model in emergency scenarios.

[0043] Access Trend Prediction and Feedback Module: After exception handling, this module uses machine learning or deep learning models to model access behavior and predict trends, outputting access frequency / probability predictions. The system monitors the error between the prediction and actual behavior and, based on a dual-trigger mechanism of time and events, determines whether feedback correction is needed. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module, completing a closed-loop optimization.

[0044] Scheduling request generation module: Based on the output access probability or frequency results, it determines whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to instruct the data object to be preloaded into high-performance media to improve the response speed of potential access requests and the system preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time and cache loading path.

[0045] The beneficial effects of the present invention are: The present invention introduces a dynamic threshold optimization mechanism based on reinforcement learning, combines prediction error feedback with system operation status, and realizes adaptive adjustment of data access anomaly detection threshold, effectively reducing the risk of misjudgment and missed judgment, and improving detection accuracy and system robustness.

[0046] Furthermore, this invention establishes a closed-loop prediction-feedback-optimization process, combining access trend prediction with a scheduling request generation mechanism. This supports pre-loading of high-frequency data and interface acceleration, improving the system's responsiveness and resource utilization efficiency in bursty access scenarios. The overall solution offers excellent scalability and cross-scenario adaptability, making it suitable for scenarios such as long-term data preservation in data storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of the entire data processing process of the present invention; Figure 2 Schematic diagram of the outlier detection and classification processing logic architecture of the present invention; Figure 3 Schematic diagram of the closed-loop mechanism of dynamic threshold adjustment driven by prediction error of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to embodiments and accompanying drawings, but is not limited thereto.

[0049] Example 1: like Figure 1 As shown, this embodiment provides a data access prediction and scheduling method based on dynamic thresholds, and the steps are as follows: Step 1: Data collection: Collect behavior data from access logs, interface call records, and file metadata information of data storage systems in a timely or real-time manner through interface protocols such as RESTful API, SFTP, Kafka, etc. Behavior data includes but is not limited to access frequency, access time, user ID, interface response delay, and data location, etc.

[0050] Step 2: Data preprocessing: Clean and standardize the collected raw access behavior data, including deduplication, missing field filling, and initial anomaly screening, etc. Use Z-score method to mark extreme access behavior as abnormal candidate samples, and combine access behavior context information (such as interface state, access source type) to enhance feature data, providing support for subsequent anomaly detection. The steps to identify abnormal point data and mark as abnormal candidate samples are as follows: Step 2.1: Z-score anomaly score calculation, for each behavior data, calculate the standard deviation deviation of numerical fields based on Z-score method, which is used to measure the deviation of the behavior data from the normal behavior distribution. The Z-score calculation formula is:

[0051] Where x i represents the feature value of the field in the current behavior data, μ represents the mean of the field, and σ represents the standard deviation. Step 2.2: Abnormal candidate sample identification: When the absolute value of the Z-score value of one or more fields in a behavior data exceeds the set threshold, i.e. >3, it is preliminarily determined that the behavior data is abnormal, and the behavior data is marked as an abnormal candidate sample. This marking is not directly used for subsequent One-Class SVM model training, but is used to exclude obvious outliers, assist in building high-quality training sets, and participate in the initial anomaly boundary and threshold setting of the subsequent model. Step 2.3: Context information enhancement, further introduce access behavior context information, including interface type, access source, user identity label, etc., through field splicing or feature fusion to fuse with the original behavior data, generate composite features containing semantic enhancement, to improve the accuracy of subsequent abnormal point fine-grained classification and identification.

[0052] Step 3: Data anomaly detection: Use an unsupervised learning support vector machine model (such as One-Class SVM) to model access behavior, build an anomaly detection model, and identify potential abnormal behavior by learning normal access patterns and building boundaries. The initial anomaly threshold is set based on the "mean ± k × standard deviation" of the anomaly score. To improve model stability, only normal samples selected in the preprocessing stage are used for training, and suspicious access points are excluded; The construction and execution steps of the anomaly detection model are as follows: Step 3.1: Build training data, use the unmarked behavior data after preprocessing in step 2 as model training input, and consider it as a sample representing normal access patterns. The behavior data includes access frequency, interface response time, user access role, etc. The behavior data marked as abnormal candidate samples in step 2 is not directly input into the model, but is used as auxiliary information to exclude extreme behavior records that may deviate significantly from the access pattern, thereby improving model detection performance and stability. Step 3.2: Build an anomaly detection model, use a support vector machine model based on unsupervised learning (One-Class SVM) as an anomaly detector to learn the boundary distribution of typical access behavior in the data system. The specific implementation process is as follows: Step 3.21, select kernel function: select radial basis kernel function (Radial Basis Function, RBF), which has good non-linear mapping ability and can effectively construct decision boundaries under complex data distribution and fuzzy boundaries. Its form is:

[0053] Where: x i and x j represent two different sample vectors in the input sample space, which come from the same data set but represent different access behavior instances. In the radial basis kernel function, the goal of the kernel function is to measure the similarity between any two samples, so it needs to calculate the distance between the two samples. Specifically: represents the feature vector of the first sample; represents the feature vector of the second sample, and the smaller the Euclidean distance, the more similar the two samples are, and the closer the output value of the kernel function to 1. The closer the output value of the kernel function to 1, the more similar the two samples in the feature space, indicating that the current sample is more likely to belong to the distribution range of "normal access behavior" in the training set; otherwise, the closer the output of the kernel function to 0, the greater the difference between the samples, which may be an abnormal access behavior deviating from the normal pattern;

[0054] γ represents the kernel function width parameter, which controls the scale of the nonlinear mapping; ‖x i -x j ‖ 2 Represents vector x i with x j The Euclidean square distance; exp() represents the natural exponential function; Step 3.22, parameter setting: Set the hyperparameter nu, which represents the maximum allowed anomaly ratio. The preferred range is 0.01-0.10, and the default setting is 0.05. It is used to control the model's tolerance for anomalies. The smaller the value, the stricter the judgment standard. Set gamma to the width parameter of the radial basis kernel function, preferably set to 1 / n_features, where n_features represents the number of feature dimensions, to ensure a balance between model sensitivity and stability. To improve model performance, the gamma parameter is optimized through grid search over a log-uniform search space (e.g., [0.001, 10]). Cross-validation is used, with the F1-score on the validation set as the optimization target. Step 3.23, Training Process: The anomaly detection model is trained using sample data of normal access patterns. The optimal hyperplane boundary is learned through the following objective function to encompass most normal access behavior records:

[0055] constraint:

[0056] Where: w is the normal vector of the hyperplane, which controls the direction and position of the hyperplane; ξ i is a slack variable, which represents the error tolerance of sample i; ρ is the offset of the decision boundary; represents the kernel mapping function.

[0057] Step 3.3: Initial anomaly threshold setting. In the initial stage, statistical methods are used to analyze the anomaly scores and set the initial anomaly threshold. This initial threshold is not only used to identify anomalies in the data access behavior collected in the early stage, but also provides a basic reference for the subsequent dynamic threshold adjustment mechanism. The initial threshold setting steps are as follows: 3.31. Obtaining anomaly scores: Using the trained anomaly detection model, calculate the corresponding anomaly scores for all samples in the validation set. , decision_function is the signed distance from the sample to the decision boundary, and these scores are regarded as a set of one-dimensional numerical data sets, recorded as:

[0058] wherein, represents the anomaly score of the nth data sample; The verification set is obtained by dividing from the data samples that are not roughly marked as abnormal by the Z-score (i.e., part of the normal samples), and by default, the division is in chronological order, using 20% of the samples in the latest period as the verification set. This data set does not participate in model training, and is only used for statistical setting of the anomaly threshold and performance evaluation.

[0059] 3.32, statistical analysis: Calculate the mean μ and standard deviation σ of the anomaly score; Define the initial anomaly threshold θ based on the mean ± k × standard deviation strategy, as follows:

[0060] wherein, the parameter k is a hyperparameter, initially set to 3, controlling the strictness of anomaly determination; 3.33, determination strategy: for any test sample x, if its anomaly score , it is determined as an abnormal point; otherwise, it is considered as a normal point.

[0061] Step 3.4: anomaly detection execution, using the trained anomaly detection model to perform anomaly scoring on the newly input behavior data, and comparing the anomaly score with the initial anomaly threshold, outputting the label of whether each behavior data is abnormal, as the basis for whether to enter the next prediction process.

[0062] Step 4: abnormal point identification threshold dynamic adjustment, as shown in Figure 3 : using a reinforcement learning-based adaptive strategy optimization mechanism, dynamically adjusting the abnormal point identification threshold according to the prediction model error rate and system feedback performance. This strategy combines prediction error trends, feedback reward values, and rules to determine the threshold adjustment direction and amplitude; To improve the adaptive ability and robustness of the anomaly detection module in the data scenario, on the basis of the initial anomaly threshold setting, a dynamic anomaly threshold adjustment mechanism based on reinforcement learning + prediction accuracy and system running feedback mechanism is further introduced. Through the linkage perception of prediction accuracy and system running feedback, real-time dynamic adjustment of the abnormal point determination standard is realized, enhancing the model's adaptability to data fluctuations and changes in access behavior. The specific implementation steps of this dynamic adjustment strategy are:

[0063] Step 4.1: state space construction, every 1 hour, construct the state vector S t of the current environment for input to the reinforcement learning module. The state vector includes but is not limited to the following 4 dimensions: the current abnormal point identification threshold θ t ​; The prediction error indicators MAPE and MSE in the sliding window, the error indicators here will be described in the subsequent step 6; the currently detected FP t and FN t ; Cache hit rate of the current cycle , = Number of hits / Total number of accesses; where the number of hits indicates the number of times a user access request hits the cache during the current statistical period. That is, a hit is counted when an access request for a data object is successfully responded to directly from the cache without reading from the underlying cold storage medium. FP t Indicates the proportion of normal samples judged as abnormal under the current anomaly detection threshold; FN t The ratio of abnormal samples that are missed as normal under the current anomaly detection threshold is calculated as follows:

[0064]

[0065] Among them, TP (True Positive): represents samples that are actually abnormal and are correctly identified as abnormal by the model; TN (True Negative): represents samples that are actually normal and are correctly identified as normal by the model; FP (False Positive): represents samples that are actually normal but are mistakenly identified as abnormal by the model; FN (Failed Negative): represents samples that are actually abnormal but are mistakenly identified as normal by the model.

[0066] The above state vector can reflect the system's current detection performance, prediction effect, and interface operation status, serving as the perception basis for reinforcement learning strategy adjustment.

[0067] Step 4.2: Action space setting: Set a set of executable threshold adjustment actions to adjust the current anomaly determination threshold, including the following basic operations: Basic action type: increase threshold, ; Decrease the threshold, ; Maintain the current threshold, 0; in, The step size is a dynamically configurable parameter whose initial value is set to a fixed value proportional to the threshold standard deviation range. , where σ t is the standard deviation of the threshold at any time t; The threshold standard deviation refers to the fluctuation degree of the abnormal judgment threshold changes recorded in multiple recent cycles, which reflects the judgment stability of the system in the past period of time. Specifically, in each judgment cycle, the threshold after the current execution is Stored as historical records, let the sliding window length be w, then the threshold standard deviation σ at any time t t The calculation is shown as follows:

[0068] in, Represents the average threshold within the sliding window, and the calculation formula is:

[0069] In order to improve the system's adaptability to data fluctuations, a dynamic step adjustment mechanism based on prediction error fluctuations is introduced to achieve sensitivity control of action adjustment. The adjustment formula is as follows:

[0070] in: : The prediction error variance in the current sliding window; : historical error variance mean; α: step size adjustment factor, preferably 0.5; K: error amplification threshold factor, preferably set to 2.0; To prevent system oscillation and frequent adjustments, the step size range is set to , ensuring that the threshold change is both sensitive and not overly radical.

[0071] Step 4.3: Reward function design. To guide the reinforcement learning strategy in dynamically adjusting the outlier threshold while taking into account prediction accuracy, detection accuracy, and system resource utilization, the following composite reward function is set:

[0072] Where: R t is the total reward value of the current round t; MAPE t is the average prediction error of the current window; FP t 、FN t are the false positive rate and missed positive rate of the current anomaly detection result; CacheHit t is the cache hit rate of the current cycle; α, β, and γ are adjustable weight parameters; Step 4.4: Strategy learning and updating, using Q-learning algorithm to achieve adaptive learning and dynamic updating of the outlier recognition threshold adjustment strategy. In the strategy learning process, based on the state vector S t , reward feedback and optional action set to construct the state-action mapping function Q (S t ,A t ), and achieve strategy optimization through iterative updates; The Q-learning update rule is as follows:

[0073] Among them: A t Execute the current threshold adjustment action (such as increase, decrease or maintain); is the maximum Q value of all possible actions in the next state; The learning rate controls the fusion speed of the old and new policies, and is preferably set to 0.1-0.5; is the discount factor, which weighs current benefits against future benefits and is preferably set between 0.8 and 0.95; To meet the needs of more complex access behavior modeling in the future, the present invention expands the strategy learning module into a DQN form based on a deep neural network to improve the generalization ability and learning efficiency in large states.

[0074] Step 4.5: To improve the adaptability and explainability of execution strategies in different business scenarios, a rule system fusion mechanism is introduced. A configurable rule system module is used to perform soft constraints or legitimacy verification on threshold adjustment actions. The rules include but are not limited to: Threshold boundary definition rules: prevent threshold adjustment from exceeding the reasonable statistical range; define the allowable abnormality judgment threshold range as θ min ≤θ t ≤θ max , set to θ min =0.2,θ max =0.8, which can be configured according to different business scenarios. If the current Q-learning model output action causes the adjusted threshold to exceed this boundary, the action will be rejected and fall back to the last valid threshold θ t-1 .

[0075] Adjustment frequency limit rule: limit the number of times the policy action is executed per unit time; set the time window T_window, and only allow a maximum of one threshold adjustment operation in each time window. If the time interval between the application time of an action adjustment and the last successful adjustment is <T_window,则该动作被拒绝,维持原策略不变,并记录为“冷却期内抑制”; Scenario adaptation rules: For example, restricting excessive adjustments during peak access periods or important business cycles.

[0076] Step 5: Classification and processing of outliers: Figure 2 The outliers detected in step 3 are input into the outlier classification model. Combining the original access features, contextual information, and anomaly scores, the model further refines the classification into noise anomalies and true anomalies. Noise anomalies are deleted or repaired through interpolation, while true anomalies are retained and explicitly labeled to improve the predictive model's adaptability in emergency scenarios.

[0077] Specifically: Classify and process abnormal points. Perform fine-grained classification on the suspected abnormal behavior data identified in step 3.4, dividing them into noise anomalies and real anomalies, and implement differentiated processing strategies based on different types. Specifically: A training dataset is constructed through manual and rule-based automatic annotation, and a binary classification model, such as LightGBM or SVM, is trained to support detailed anomaly classification. Access behaviors with high field missing rates (e.g., exceeding 30%) or malformed formats are labeled "noise anomalies." Access behaviors with anomaly scores consistently exceeding the detection threshold for multiple consecutive cycles and a significant decrease in cache hit rates are labeled "true anomalies."

[0078] After the model training is completed, the abnormal candidate samples in step 3 are used as input, and the classification model is used to further determine their specific types, so as to achieve fine-grained division of abnormal points between "noise anomalies" and "real anomalies".

[0079] Build an outlier classification model and use a binary classification model such as LightGBM, XGBoost, or Support Vector Machine (SVM) to identify outliers. The training data includes labeled outlier samples, and the classifier input features include anomaly scores, access behavior context information, and current cache hit status. Based on the classification results, the following processing strategies are implemented: For access behaviors that are determined to be noise anomalies, if the log collection format is incorrect or fields are missing, they are discarded directly. If they are isolated deviations, data smoothing or sliding window statistical replacement is performed. For access behaviors that are determined to be truly abnormal, the log and its features are retained and marked with true abnormality labels. They are used as key feature samples in subsequent access trend prediction models to improve the model's ability to identify and predict extreme behaviors such as access bursts and load anomalies.

[0080] Step 6: Access Trend Prediction and Feedback: After exception handling, machine learning or deep learning models are used to model access behavior and predict trends, outputting access frequency / probability predictions. The system monitors the error between the prediction and actual behavior (such as MAPE and RMSE) and, based on a dual-trigger mechanism of time and events, determines whether feedback correction is needed. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module, completing the closed-loop optimization.

[0081] Specifically: model prediction and feedback, after completing the classification and processing of abnormal points, the model training and trend prediction operation is performed on the cleaned and repaired access behavior data. For the characteristics of low frequency, periodicity and weak but sudden fluctuation of access behavior in the data system, a deep learning model with strong time series modeling ability and long-term dependence capture ability is used for modeling, especially the Transformer series structure based on self-attention mechanism, such as Informer, Autoformer, etc. Through the introduction of sparse self-attention mechanism, multi-scale time feature encoding and trend separation mechanism, this kind of model can accurately model the future access trend in a long historical sequence, and is suitable for directory-level or task-level data access behavior prediction tasks in data storage systems;

[0082] During model training, input sequences are constructed from historical access logs, and target variables include data access frequency, access density and other key indicators in the future time window. The training adopts a sliding time window strategy and a multi-step prediction mechanism. After training is completed, the latest data in the current period is input into the model for prediction to obtain the access behavior trend in the future time period. To build a closed-loop optimization mechanism for the system, a feedback mechanism combining time-driven and event-driven is introduced to continuously optimize the abnormal detection threshold and prediction strategy. Time-driven feedback: Set a periodic evaluation window that adapts to the data scenario, preferably once a day or once a week, to avoid excessive frequent feedback due to the low-frequency and sparse characteristics of access behavior. At the end of each period, the prediction errors in the current window are automatically summarized and combined with the abnormal detection accuracy index and system running index for comprehensive evaluation. If the error index continuously exceeds the set threshold for multiple consecutive periods, the strategy optimization process of the abnormal detection threshold is automatically triggered. Event-driven feedback: Set a targeted abnormal monitoring mechanism to capture short-term access behavior fluctuations or system performance abnormalities, such as sudden increase in prediction error, deviation of access behavior from model fitting trend, or sudden drop in cache hit rate in a short time window. The system will immediately trigger a rapid feedback response, calling the reinforcement learning module to dynamically optimize the current abnormal detection threshold, achieving adaptive strategy update to improve the system's response ability in sudden load scenarios. To prevent frequent strategy changes or system shocks caused by short-term error fluctuations, a cooling control mechanism is designed: after each threshold adjustment, enter the freezing window period (preferably set to 30 minutes to 2 hours), during which new feedback signals are temporarily suspended and only observation indicators are continued to be accumulated. After the freezing window period ends, re-evaluate whether to perform strategy adjustment to balance feedback response and strategy stability.

[0083] Step 7: Scheduling request generation: Based on the output access probability or frequency results, determine whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to instruct the data object to be preloaded to high-performance media to improve the response speed of potential access requests and the system preprocessing capabilities. The scheduling request content includes information such as the target data ID, scheduling trigger time, and cache loading path.

[0084] Specifically: Scheduling request generation: After completing the access behavior trend prediction, a scheduling scoring function is constructed based on the prediction results and the multi-dimensional status of the current access object. This function is used to determine whether the target data needs to be loaded into high-performance media (such as SSDs and caches) first, in order to achieve a warm-up response to sudden accesses and improve interface performance. The scheduling scoring function is constructed as follows:

[0085] Among them: α, β, γ, δ, θ are weight parameters, which are adjusted according to the actual scenario; F pred It represents the probability value of the target data output by the access prediction module being accessed in the future. It can output the normalized predicted access frequency through models such as Transformer, and then map it to a probability value between 0 and 1 through Sigmoid. idle Indicates the storage time of the current data from the last access to the current time, which is calculated by the system access log statistics module; L cur Indicates the level code of the current storage medium of the data object, for example, SSD=1, HDD=2, tape=3. The higher the value, the lower the performance of the current storage medium, and the more necessary it is to pre-schedule to high-speed media. Lcur represents the storage medium level code of the current data object, which is used to measure the read and write performance and access latency of the storage medium. The specific definition is as follows: SSD (Solid State Drive): Lcur = 1, indicating the highest performance thermal data storage medium; HDD (mechanical hard disk): Lcur = 2, indicating medium-performance warm data storage media Tape: Lcur = 3, indicating the lowest performance data medium; The larger the Lcur value, the lower the current storage medium access performance and the higher the data response latency. Therefore, the higher the priority of the system for "needing to be pre-scheduled to high-performance storage media".

[0086] C hit Indicates the hit rate of the data in the cache system in the current cycle, ranging from [0, 1]. To avoid repeated scheduling of data already in high-speed media, 1 - C is used in the score. hitExpressing the "miss rate", the lower the hit rate, the higher the necessity of scheduling; the calculation formula of the hit rate is: (If T i >0), otherwise 0, where T i Indicates the total number of times the data object is accessed during the statistical period, H i Indicates the number of times a data object is cached; when a data object is not accessed during this cycle (i.e., T i =0), the hit rate C hit Defined as 0, indicating "not hit" to avoid division by zero errors and reflect the state that the data is not currently being actively accessed; Sl oad Indicates the current load index, which can comprehensively evaluate the current I / O utilization, cache usage, scheduling queue length and other dimensions, and is normalized to a floating point number in the range of [0,1] to reflect whether the scheduling timing is suitable for preloading operations. A higher Score value indicates a higher urgency and scheduling priority for the current data object. This value is then compared with the set scheduling threshold, T_sched. If Score > T_sched (default is 0.8), a scheduling request is generated. Alternatively, the top-K (default K = 10) data with the highest Score values ​​among all candidate objects are selected as scheduling targets. The scheduling request includes metadata such as the data identifier, target medium, and scheduling window time. The scheduling request is written to the asynchronous task queue. The scheduling engine periodically scans the task queue and determines whether execution conditions are met based on current operating status, such as I / O load, CPU utilization, and cache free capacity. If current system resources permit, the scheduling engine triggers a data migration operation, preloading data from cold storage media to high-performance media. This ensures that critical data is hot-backed up before future access, optimizing access performance.

[0087] Example 2: This embodiment provides a data access prediction and scheduling system based on dynamic thresholds, including: Data collection module: used to collect behavioral data from the access logs, interface call records, and file metadata of the data storage system in a scheduled or real-time manner; Data preprocessing module: This module cleans and standardizes the collected raw access behavior data, including deduplication, missing field filling, and initial anomaly screening. It uses the Z-score method to mark extreme access behaviors as abnormal candidate samples, and enhances feature data by combining access behavior context information. Data anomaly detection module: This module uses an unsupervised support vector machine model to model access behavior and build an anomaly detection model. It establishes boundaries by learning normal access patterns and identifies potential abnormal behaviors that deviate from these boundaries. The initial anomaly threshold is set based on the anomaly score. To improve model stability, only normal samples screened in the preprocessing phase are used in training, eliminating suspicious access points. Anomaly identification threshold dynamic adjustment module: used to use the adaptive strategy optimization mechanism based on reinforcement learning to dynamically adjust the anomaly identification threshold according to model feedback; Anomaly classification and processing module: Detected anomalies are input into the anomaly classification model. Combining the original access features, contextual information, and anomaly scores, the model further refines the classification into noise anomalies and true anomalies. Noise anomalies are deleted or repaired through interpolation, while true anomalies are retained and explicitly labeled to improve the adaptability of the prediction model in emergency scenarios.

[0088] Access Trend Prediction and Feedback Module: After exception handling, this module uses machine learning or deep learning models to model access behavior and predict trends, outputting access frequency / probability predictions. The system monitors the error between the prediction and actual behavior and, based on a dual-trigger mechanism of time and events, determines whether feedback correction is needed. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module, completing a closed-loop optimization.

[0089] Scheduling request generation module: Based on the output access probability or frequency results, it determines whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to instruct the data object to be preloaded into high-performance media to improve the response speed of potential access requests and the system preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time and cache loading path.

Claims

1. A data access prediction and scheduling method based on dynamic thresholds, characterized in that: Here are the steps: Step 1: Data collection: collect behavioral data periodically or in real time from the data storage system's access logs, interface call records, and file metadata information; Step 2: Data preprocessing: Clean and standardize the collected raw access behavior data, including deduplication, missing field filling, and initial anomaly screening. Use the Z-score method to mark extreme access behaviors as abnormal candidate samples. At the same time, enhance the feature data by combining access behavior context information. Step 3: Data anomaly detection: An unsupervised support vector machine model is used to model access behavior and build an anomaly detection model. This model establishes boundaries by learning normal access patterns and identifies potential anomalies that deviate from these boundaries. The initial anomaly threshold is set based on the anomaly score. To improve model stability, only normal samples screened in the preprocessing phase are used in training, eliminating suspicious access points. Step 4: Dynamically adjust the outlier identification threshold: Use an adaptive strategy optimization mechanism based on reinforcement learning to dynamically adjust the outlier identification threshold based on model feedback; Step 5: Outlier Classification and Processing: Detected outliers are input into the outlier classification model. Combining the original access features, contextual information, and anomaly scores, the model further refines the classification into noise anomalies and true anomalies. Noise anomalies are deleted or repaired through interpolation, while true anomalies are retained and explicitly labeled to improve the predictive model's adaptability to sudden emergencies. Step 6: Access Trend Prediction and Feedback: After exception handling, machine learning or deep learning models are used to model access behavior and predict trends. The system outputs access frequency / probability prediction results. The system monitors the error between the prediction results and actual behavior and determines whether feedback correction is needed based on a dual-trigger mechanism of time and events. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete the closed-loop optimization. Step 7: Scheduling request generation: Based on the output access probability or frequency results, determine whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to instruct the data object to be preloaded to high-performance media to improve the response speed of potential access requests and the system preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time and cache loading path.

2. The data access prediction and scheduling method based on dynamic threshold according to claim 1, characterized in that: In step 1, the behavioral data includes but is not limited to: access frequency, access time, user ID, interface response delay, and data location; In step 2, the steps to identify outlier data and mark them are as follows: Step 2.1: Z-score anomaly score calculation. For each piece of behavioral data, the standard deviation of the numeric field is calculated based on the Z-score method to measure the degree of deviation of the behavioral data from the normal behavior distribution. The Z-score calculation formula is: Among them, x i Represents the characteristic value of the field in the current behavior data, μ represents the mean of the field, and σ represents the standard deviation; Step 2.2: Identification of abnormal candidate samples: When the absolute value of the Z-score value of one or more fields in a certain behavioral data exceeds the set threshold, that is, >3, the behavior data is preliminarily judged to be abnormal and the behavior is marked as an abnormal candidate sample; Step 2.3: Contextual information enhancement: further introduce access behavior context information and fuse it with the original behavior data through field splicing or feature fusion to generate composite features with semantic enhancement.

3. The data access prediction and scheduling method based on dynamic threshold according to claim 2, characterized in that: In step 3, the construction and execution steps of the anomaly detection model are as follows: Step 3.1: Construct training data. Use the unlabeled behavioral data preprocessed in Step 2 as model training input, treating it as samples representing normal access patterns. The abnormal candidate samples labeled in Step 2 serve as auxiliary information for training sample screening and initial threshold estimation. Step 3.2: Build an anomaly detection model. Use the unsupervised learning-based support vector machine model as an anomaly detector to learn the boundary distribution of typical access behaviors in the data system. Step 3.3: Initial anomaly threshold setting. In the initial stage, statistical methods are used to analyze the anomaly scores and set the initial anomaly threshold. Step 3.4: Anomaly detection is performed. The trained anomaly detection model is used to score the newly input behavior data for anomalies. The anomaly score is compared with the initial anomaly threshold, and a label is output to indicate whether each behavior data is abnormal or not, which serves as the basis for entering the next prediction process.

4. The data access prediction and scheduling method based on dynamic threshold according to claim 3, characterized in that: The specific implementation process of step 3.2 is as follows: Step 3.

21. Select kernel function: Use radial basis kernel function, which is in the form of: Where: x i and x j Represents two different sample vectors in the input sample space; γ represents the kernel function width parameter, which controls the scale of the nonlinear mapping; ‖x i -x j ‖ 2 Represents vector x i with x j The Euclidean square distance; exp() represents the natural exponential function; Step 3.22, parameter setting: Set the hyperparameter nu, which represents the maximum allowed anomaly ratio, to control the model's tolerance for anomalies. The smaller the value, the stricter the judgment criteria. Set gamma as the width parameter of the radial basis kernel function; Step 3.23, Training Process: The anomaly detection model is trained using sample data of normal access patterns. The optimal hyperplane boundary is learned through the following objective function to enclose the normal access behavior records: constraint: Where: w is the normal vector of the hyperplane, which controls the direction and position of the hyperplane; ξ i is a slack variable, which represents the error tolerance of sample i; ρ is the offset of the decision boundary; represents the kernel mapping function.

5. The data access prediction and scheduling method based on dynamic threshold according to claim 4, characterized in that: In step 3.3, the initial threshold setting steps are as follows: Step 3.31, obtain anomaly scores: use the trained anomaly detection model to calculate the corresponding anomaly scores for all samples in the validation set. , decision_function is the signed distance from the sample to the decision boundary, and these scores are regarded as a set of one-dimensional numerical data sets, recorded as: in, represents the abnormality score of the nth data sample; Step 3.32, statistical analysis: Calculate the mean μ and standard deviation σ of the anomaly score; The initial anomaly threshold θ is defined based on the mean ± k × standard deviation strategy, as follows: Among them, the parameter k is a hyperparameter; Step 3.33, judgment strategy: For any test sample x, if its anomaly score < , it is considered as an abnormal point; otherwise it is considered as a normal point.

6. The data access prediction and scheduling method based on dynamic threshold according to claim 5, characterized in that: In step 4, the specific implementation steps are: Step 4.1: State space construction, define the state vector S of the current environment t , used as the input of the reinforcement learning module, the state vector includes but is not limited to the following four dimensions: the current anomaly recognition threshold θ t ; Prediction error indicator within the sliding window; Currently detected FP t and FN t ; Cache hit rate of the current cycle , = Number of hits / total number of visits; where the number of hits indicates the number of times user access requests hit the cache during the current statistical period; FP t Indicates the proportion of normal samples judged as abnormal under the current anomaly detection threshold; FN t The ratio of abnormal samples that are missed as normal under the current anomaly detection threshold is calculated as follows: TP: represents a sample that is actually abnormal and is correctly identified as abnormal by the model; TN: represents a sample that is actually normal and is correctly identified as normal by the model; FP: represents a sample that is actually normal but is incorrectly identified as abnormal by the model; FN: represents a sample that is actually abnormal but is incorrectly identified as normal by the model. Step 4.2: Action space setting: Set a set of executable threshold adjustment actions to adjust the current anomaly determination threshold, including the following basic operations: Basic action type: increase threshold, ; Decrease the threshold, ; Maintain the current threshold, 0; in, The step size is a dynamically configurable parameter whose initial value is set to a fixed value proportional to the threshold standard deviation range. , where σ t for The standard deviation of the threshold at any time t; In order to improve the system's adaptability to data fluctuations, a dynamic step adjustment mechanism based on prediction error fluctuations is introduced to achieve sensitivity control of action adjustment. The adjustment formula is as follows: in: : The prediction error variance in the current sliding window; : historical error variance mean; α: step size adjustment factor; K: error amplification threshold factor; To prevent system oscillation and frequent adjustments, the step size range is set to ; Step 4.3: Reward function design, set the following composite reward function: Where: R t is the total reward value of the current round t; MAPE t is the average prediction error of the current window; CacheHit t is the cache hit rate of the current cycle; α, β, and γ are adjustable weight parameters; Step 4.4: Strategy learning and updating, using Q-learning algorithm to achieve adaptive learning and dynamic updating of the outlier recognition threshold adjustment strategy. In the strategy learning process, based on the state vector S t , reward feedback and optional action set to construct the state-action mapping function Q (S t ,A t ), and achieve strategy optimization through iterative updates; The Q-learning update rule is as follows: Among them: A t Execute threshold adjustment action for the current execution; is the maximum Q value of all possible actions in the next state; is the learning rate, which controls the speed of integration of new and old policies; is the discount factor, weighing current benefits against future benefits; Step 4.5: To improve the adaptability and explainability of execution strategies in different business scenarios, a rule system module is introduced to implement soft constraints or legitimacy verification on threshold adjustment actions. The rules include but are not limited to: Threshold boundary definition rules: prevent threshold adjustments from exceeding the statistical range; Adjust frequency limit rules: limit the number of times a policy action is executed per unit time; Scenario adaptation rules: Limit adjustment behavior during peak access periods or important business cycles.

7. The data access prediction and scheduling method based on dynamic threshold according to claim 6, characterized in that: In step 5, specifically: outlier classification and processing, the identified suspected abnormal behavior data is fine-grainedly classified into noise anomalies and true anomalies, and differentiated processing strategies are implemented according to different types. Specifically: Build an outlier classification model and use a binary classification model to identify outliers. The training data includes labeled outlier samples, and the classifier input features include anomaly scores, access behavior context information, and current cache hit status. Based on the classification results, the following processing strategies are implemented: For access behaviors that are determined to be noise anomalies, if the log collection format is incorrect or fields are missing, they are discarded directly. If they are isolated deviations, data smoothing or sliding window statistical replacement is performed. For access behaviors that are determined to be true anomalies, the log and its features are retained and marked with true anomaly labels.

8. The data access prediction and scheduling method based on dynamic threshold according to claim 7, characterized in that: In step 6, specifically: model prediction and feedback, after completing the classification and processing of outliers, model training and trend prediction operations are performed on the cleaned and repaired access behavior data. Given the low frequency and weak periodicity of access behavior in the data system, but with sudden fluctuations, a deep learning model with strong time series modeling capabilities and the ability to capture long-term dependencies is used for modeling. During model training, historical access logs are used to construct input sequences. Target variables include data access frequency and density within a future time window. Training utilizes a sliding time window strategy and a multi-step prediction mechanism. After training is complete, the latest data from the current cycle is fed into the model for prediction, yielding access behavior trends for future time periods. To build a closed-loop optimization mechanism for the system, a feedback mechanism combining time-driven and event-driven approaches is introduced to continuously optimize anomaly detection thresholds and prediction strategies. Time-driven feedback: Set a periodic evaluation window for the data scenario. After each cycle, automatically summarize the prediction errors within the current window and conduct a comprehensive evaluation based on anomaly detection accuracy indicators and system operation indicators. If the error indicators are detected to exceed the set threshold for multiple consecutive cycles, the anomaly detection threshold strategy optimization process is automatically triggered. Event-driven feedback: Set up a targeted anomaly monitoring mechanism to capture sharp fluctuations in access behavior or system performance anomalies in the short term; To prevent frequent strategy changes or system oscillations caused by short-term fluctuations in errors, a cooling control mechanism is designed: after each round of threshold adjustment, a freezing window period begins, during which new feedback signals are not accepted and only observation indicators are accumulated. After the freezing window period ends, the need for strategy adjustment is reassessed to achieve a balance between feedback response and strategy stability.

9. The data access prediction and scheduling method based on dynamic threshold according to claim 8, characterized in that: In step 7, specifically: Scheduling request generation: After completing the access behavior trend prediction, a scheduling scoring function is constructed based on the prediction results and the multi-dimensional status of the current access object to determine whether the target data needs to be loaded into high-performance media first, so as to achieve preheating response to sudden access and improve interface performance. The scheduling scoring function is constructed as follows: Among them: α, β, γ, δ, θ are weight parameters; F pred represents the probability value of the target data output by the access prediction module being accessed in the future; D idle Indicates the storage time of the current data from the last access to the current time; L cur Indicates the level code of the current storage medium of the data object. The higher the value, the lower the performance of the current storage medium and the greater the need for pre-scheduling to high-speed media. C hit Indicates the hit rate of the data in the cache system in the current cycle. To avoid repeated scheduling of data already in high-speed media, 1 - C is used in the score. hit Expressing the "miss rate", the lower the hit rate, the higher the necessity of scheduling; the calculation formula of the hit rate is: (If T i >0), otherwise 0, where T i Indicates the total number of times the data object is accessed during the statistical period, H i Indicates the number of times the data object is hit by the cache; Sl oad Represents the current load indicator. The higher the Score value, the higher the urgency of accessing the current data object and the higher the scheduling priority. This value is compared with the set scheduling threshold T_sched. If Score>T_sched, a scheduling request is generated; or the data with the top-K score values ​​among all candidate objects are selected as the scheduling target. The scheduling request contains the data identifier, target medium, and scheduling window time. The scheduling task is added to the asynchronous task queue, and the scheduling execution engine performs the migration within the window allowed by the load.

10. A data access prediction and scheduling system based on dynamic thresholds, characterized in that: include: Data collection module: used to collect behavioral data from the access logs, interface call records, and file metadata of the data storage system in a scheduled or real-time manner; Data preprocessing module: This module cleans and standardizes the collected raw access behavior data, including deduplication, missing field filling, and initial anomaly screening. It uses the Z-score method to mark extreme access behaviors as abnormal candidate samples, and enhances feature data by combining access behavior context information. Data anomaly detection module: This module uses an unsupervised support vector machine model to model access behavior and build an anomaly detection model. It establishes boundaries by learning normal access patterns and identifies potential abnormal behaviors that deviate from these boundaries. The initial anomaly threshold is set based on the anomaly score. To improve model stability, only normal samples screened in the preprocessing phase are used in training, eliminating suspicious access points. Anomaly identification threshold dynamic adjustment module: used to use the adaptive strategy optimization mechanism based on reinforcement learning to dynamically adjust the anomaly identification threshold according to model feedback; Anomaly Classification and Processing Module: Detected anomalies are input into the anomaly classification model. Combining the original access features, contextual information, and anomaly scores, the model further refines the classification into noise anomalies and true anomalies. Noise anomalies are deleted or repaired through interpolation, while true anomalies are retained and explicitly labeled to improve the predictive model's adaptability to sudden emergencies. Access trend prediction and feedback module: After exception handling, it uses machine learning or deep learning models to model access behavior and predict trends, outputting access frequency / probability prediction results. The system monitors the error between the prediction results and actual behavior and determines whether feedback correction is needed based on a time-and-event dual-trigger mechanism. If the error exceeds the set threshold tolerance range, the error signal is fed back to the threshold optimization module to complete closed-loop optimization. Scheduling request generation module: Based on the output access probability or frequency results, it determines whether the target data object meets the scheduling conditions. If the predicted value exceeds the set scheduling threshold, a scheduling request is generated to instruct the data object to be preloaded into high-performance media to improve the response speed of potential access requests and the system preprocessing capabilities. The scheduling request content includes the target data ID, scheduling trigger time and cache loading path.

Images