A data center storage space dynamic monitoring and early warning method

Abstract

The application relates to the technical field of data center storage management, and discloses a storage space dynamic monitoring and early warning method for a data center, which comprises the following steps: S1, collecting and monitoring the running state data of the storage equipment of the data center in real time; S2, predicting future storage demand through a deep learning model based on the collected historical storage data; S3, dynamically scheduling the storage resources by using a reinforcement learning algorithm based on the storage demand prediction result, and automatically adjusting the storage equipment configuration to optimize the use of the storage resources; and S4, when potential failure occurs in the storage equipment, early warning is carried out by using a failure prediction model. The application predicts storage demand through deep learning, optimizes resource scheduling through reinforcement learning, and intelligently predicts failure, so that the configuration efficiency of the storage resources of the data center, the accuracy of the failure early warning, the storage cost and the performance are improved, and the system is ensured to be efficiently and stably operated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data center storage management, in particular to a data center storage space dynamic monitoring and early warning method. BACKGROUND

[0002] With the advent of the information age, data centers have become an important part of modern enterprise core IT architecture. In order to ensure that the data center can operate efficiently and stably, dynamic monitoring and early warning of storage space become crucial. The monitoring of storage space not only involves the usage status and performance evaluation of storage devices, but also needs to predict future demand for storage resources, so as to adjust the configuration in time and avoid system performance degradation or resource waste due to insufficient or excessive configuration of storage resources. Therefore, the data center storage space dynamic monitoring and early warning method plays a key role in ensuring optimal configuration of storage resources, improving system reliability and reducing costs.

[0003] The existing data center storage space monitoring and early warning method generally collects basic operating state data of storage devices (such as temperature, storage capacity, I / O load, etc.) and combines historical data for predictive analysis. These methods generally rely on rule engines or simple statistical analysis, which can monitor and warn about device abnormalities, such as monitoring the temperature, response time, etc. of storage devices by setting thresholds, and triggering an alarm once the predetermined range is exceeded. These methods have the advantage of being intuitive and can monitor and warn the system simply, and respond to abnormal conditions of storage devices in real time through rule engines.

[0004] However, the existing technology has some obvious shortcomings, especially in dealing with rapid changes in data center storage demand and optimizing the configuration of different storage device performance. The existing monitoring and early warning system relies on static threshold settings and cannot accurately predict the dynamic changes in storage demand in real time. It also fails to effectively optimize cross-layer storage and automatically schedule resources. In addition, due to the lack of intelligent data analysis and prediction capabilities, the existing system has a lag in warning about storage failures, making it difficult to intervene effectively before the storage device fails. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application provides a data center storage space dynamic monitoring and early warning method, which solves the problems of existing data center storage space monitoring and early warning methods in dealing with dynamic changes in storage demand and device failure prediction.

[0006] To achieve the above purpose, the present application is implemented by the following technical scheme: a data center storage space dynamic monitoring and early warning method, the method comprising the following steps:

[0007] S1, collect and monitor the running state data of the storage devices in the data center in real time;

[0008] S2, based on the collected historical storage data, predict future storage demand through a deep learning model;

[0009] S3, based on the storage demand prediction results, use a reinforcement learning algorithm to dynamically schedule storage resources, automatically adjust storage device configurations to optimize the use of storage resources;

[0010] S4, when potential failures occur in the storage devices, use a fault prediction model to provide early warning and perform automatic data migration or redundant backup through a self-healing mechanism;

[0011] S5, based on data access frequency and importance, implement cross-tier storage optimization to automatically migrate data to different storage tiers, optimizing storage cost and performance.

[0012] Preferably, the S1 includes the following steps:

[0013] Real-time collection of data on temperature, I / O load, response time and storage capacity utilization of the storage devices;

[0014] The above monitoring data is input into a pre-set data storage system for subsequent analysis and prediction.

[0015] Preferably, the S2 includes the following steps:

[0016] Use a long short-term memory network (LSTM) model to train historical storage demand data to predict data storage demand in the future;

[0017] Evaluate the trained LSTM model, select the optimal prediction model for deployment, and use mean squared error (MSE) as the loss function to optimize the model.

[0018] Preferably, the use of mean squared error (MSE) as the loss function to optimize the model includes the following steps:

[0019] Calculate the error between the actual storage demand and the model prediction demand, and optimize the weights of the LSTM network through a backpropagation algorithm;

[0020] Minimize the following loss function:

[0021] ;

[0022] where, is the actual storage demand, is the predicted value, is the number of samples.

[0023] Preferably, the S3 comprises the following steps:

[0024] The state space, action space and reward function of the reinforcement learning model are set, and the Q-learning algorithm is used for storage resource scheduling;

[0025] The state space is defined as the current usage state of the storage device, and the action space is defined as the scheduling operation of the storage resource;

[0026] The reward function is defined as the optimized use efficiency of the storage resource, and the storage resource allocation strategy is adjusted through the Q-learning algorithm.

[0027] Preferably, the Q-learning algorithm for storage resource scheduling comprises the following steps:

[0028] Initialize the Q table, set the initial Q value to zero, and update the Q value according to the feedback storage resource usage;

[0029] The Q value is updated using the following update formula:

[0030] ;

[0031] Wherein, is the reward value, indicating the improvement of the storage resource utilization rate, is the discount factor, is the learning rate.

[0032] Preferably, the S4 comprises the following steps:

[0033] Collect the temperature, error log and I / O delay information of the storage device, and input them into the fault prediction model;

[0034] The state of the storage device is analyzed using a convolutional neural network (CNN), and the possible fault risk is predicted.

[0035] Preferably, the analysis of the state of the storage device using the convolutional neural network (CNN) comprises the following steps:

[0036] Extract the health state features of the storage device and input them into the CNN network for fault identification;

[0037] Calculate the probability of fault occurrence, use the softmax function to output the prediction result of the fault, and when the prediction probability exceeds the preset threshold, trigger the early warning.

[0038] Preferably, the S5 comprises the following steps:

[0039] According to the access frequency and importance of the data, the data is divided into hot data, cold data and expired data;

[0040] For hot data, it is stored on high-performance storage devices, for cold data, it is stored on low-cost storage devices, and expired data is periodically cleaned up or archived.

[0041] Preferably, the S5 further comprises the following steps:

[0042] According to the access mode of data, a hybrid storage architecture is used for data storage, and data is dynamically migrated between different storage levels;

[0043] The data migration strategy is optimized using linear programming to minimize storage costs and meet the maximum capacity constraints of each storage level.

[0044] The present application provides a kind of data center storage space dynamic monitoring and early warning method.It has the following beneficial effects:

[0045] 1, the present application by adopting dynamic storage space monitoring and prediction method, realizes the efficient scheduling of data center storage resources, by real-time monitoring storage device state and combining deep learning model to predict future storage demand, can accurately identify storage bottleneck, and make response in advance, ensure that storage system can still maintain good performance and stability under high load.

[0046] 2, based on reinforcement learning algorithm, the present application optimizes the allocation strategy of storage resources, improves storage efficiency by adaptively adjusting storage configuration, reduces the risk of over-provisioning, ensures optimal storage performance under resource constraints, and reduces resource waste.

[0047] 3, the present application introduces cross-level storage optimization and data migration mechanism, according to the access frequency and importance of data, automatically allocates data to appropriate storage device level, combines linear programming optimization migration strategy, can effectively reduce storage cost, and improve the utilization rate of storage resources, meet the performance requirements of different levels. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The method flowchart of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely in the drawings in the specification of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] Please refer to the drawings in the specification of the present application Figure 1The embodiment of the present application provides a kind of data center's storage space dynamic monitoring and early warning method, method includes the following steps:

[0051] S1, the running state data of data center storage equipment is collected and monitored in real time, the bottleneck or potential fault point of storage resource can be found in time by monitoring the running state of storage equipment in real time, accurate data basis is provided, and foundation is laid for subsequent storage demand prediction and resource scheduling, so as to improve the monitoring capability of data center to the state of storage resource, reduce storage resource waste, improve the availability of equipment;

[0052] S2, based on the collected historical storage data, the future storage demand is predicted by depth learning model, this step can accurately predict the trend of storage demand change, help data center to understand the shortage or excess of storage resource in advance, and then avoid the waste or shortage of resource, and simultaneously, through accurate storage demand prediction, the utilization efficiency of storage resource can be improved, and the performance bottleneck of data center caused by storage demand fluctuation is avoided;

[0053] S3, based on the prediction result of storage demand, the storage resource is dynamically scheduled by using reinforcement learning algorithm, and the storage device configuration is automatically adjusted to optimize the use of storage resource, by introducing reinforcement learning algorithm, the configuration of storage resource can be automatically adjusted according to the predicted demand, the complexity and error of manual operation are avoided, the utilization efficiency of storage resource is optimized, and the application of reinforcement learning ensures that the dynamic scheduling of storage resource can adapt to demand fluctuation in real time, and the intelligent level of data center resource scheduling is improved;

[0054] S4, when potential failure occurs in storage equipment, early warning is carried out by using fault prediction model, and automatic data migration or redundancy backup is executed by self-healing mechanism, this step can identify potential equipment failure risk in advance, reduce the suddenness of failure occurrence, through the automatic self-healing mechanism, it can be ensured that data will not be lost when storage equipment fails, and the system can continue to operate stably, overall, manual intervention is reduced, and the reliability and service availability of data center are improved;

[0055] S5, based on data access frequency and importance, cross-layer storage optimization is implemented, and data is automatically migrated to different storage levels, and storage cost and performance are optimized, through cross-layer storage optimization, data can be reasonably distributed to different storage levels, to ensure that the high-performance storage demand of hot data is met, while reducing the storage cost of cold data, this method not only improves the use efficiency of storage resource, but also effectively reduces the overall storage cost of data center.

[0056] Please refer to the accompanying Figure 1 In a preferred embodiment of the present application, S1 includes the following steps:

[0057] Real-time collection of data of temperature, I / O load, response time and storage capacity utilization of the storage device, real-time monitoring of temperature can help the data center to find the temperature abnormal situation that may cause device failure in advance, through real-time monitoring of I / O load, the performance bottleneck of the storage device can be identified in time, and the performance degradation and system crash problem under high load condition can be avoided, through real-time monitoring of the response time of the storage device, the problem can be found in time in the early stage of performance decline, and the bottleneck affecting the overall performance of the system can be avoided;

[0058] The monitoring data is input into the preset data storage system for subsequent analysis and prediction, and the monitoring data is input into the efficient storage system, which can provide rich and accurate data support for subsequent prediction and scheduling algorithm, and through long-term accumulated data, the data storage system can support efficient trend analysis, anomaly detection and prediction analysis and other tasks, and improve the intelligent management level of the data center.

[0059] Please refer to the accompanying Figure 1 In a preferred embodiment of the present application, S2 comprises the following steps:

[0060] The long short-term memory network (LSTM) model is used to train the historical storage demand data to predict the data storage demand in a future period of time, by using the LSTM model, the time dependence of the storage demand data can be effectively captured, and the future storage demand can be accurately predicted, compared with the traditional prediction method, the LSTM can process more complex time series data and identify long-term trends and periodic fluctuations, therefore, the use of the LSTM model can improve the accuracy of prediction and avoid the problem of resource waste or shortage caused by inaccurate storage demand prediction;

[0061] The trained LSTM model is evaluated, and the optimal prediction model is selected for deployment, and the mean square error (MSE) is used as a loss function to optimize the model, by detailed evaluation and optimization of the trained LSTM model, the efficiency and accuracy of the deployed prediction model in actual application can be ensured. Selecting the optimal prediction model for deployment can maximize the accuracy of storage demand prediction and provide reliable data support for subsequent storage resource scheduling and optimization.

[0062] Please refer to the accompanying Figure 1 In a preferred embodiment of the present application, using the mean square error (MSE) as a loss function to optimize the model comprises the following steps:

[0063] The error between the actual storage requirement and the model predicted requirement is calculated, and the weights of the LSTM network are optimized by the back propagation algorithm, and the error calculation is not only the basis of the model learning process, but also can help identify potential problems in the prediction process, such as bias, overfitting, etc., therefore, the calculation of error is the premise of deep learning model optimization, which helps to improve the accuracy of model prediction, the back propagation algorithm can optimize network parameters by calculating gradient, and ensure that the weights of LSTM model are gradually adjusted to the optimal value, through back propagation optimization, the prediction error of the model will be minimized, which improves the accuracy of storage requirement prediction, and provides more reliable decision support for storage resource management of data center;

[0064] Minimize the following loss function:

[0065] ;

[0066] Wherein, is the actual storage requirement, is the predicted value, is the number of samples, using MSE as the loss function can effectively guide the model to optimize its parameters, minimize the difference between the actual demand and the predicted demand, through repeated optimization of MSE, the LSTM model will be able to achieve higher prediction accuracy, thus providing more accurate support for the storage requirement prediction of data center, and avoiding the problem of resource waste or shortage.

[0067] Please refer to the attached Figure 1 In a preferred embodiment of the present application, S3 comprises the following steps:

[0068] The state space, action space and reward function of the reinforcement learning model are set, and the storage resource scheduling is carried out by using Q-learning algorithm, Q-learning algorithm is a model-free reinforcement learning algorithm, which learns the strategy of taking the best action in different states by constantly updating Q value table, Q value (quality value) measures the expected return of taking a certain action in a certain state. Q-learning algorithm can continuously improve the strategy through the balance of exploration and utilization, and finally find the optimal resource scheduling scheme;

[0069] The state space is defined as the current use state of the storage device, and the action space is defined as the scheduling operation of the storage resource;

[0070] The reward function is defined as the optimization of the use efficiency of storage resources, and the storage resource allocation strategy is adjusted through the Q-learning algorithm. By clearly defining the state space, action space and reward function, the Q-learning algorithm can provide sufficient information to help the model select the optimal resource scheduling strategy in multiple operations. A reasonable reward function can guide the model to optimize the configuration of storage resources through the reward mechanism, thereby improving the use efficiency of storage resources, reducing resource waste, and ensuring the efficient operation of the data center storage space.

[0071] Please refer to the attached Figure 1 In a preferred embodiment of the present application, the Q-learning algorithm for storage resource scheduling includes the following steps:

[0072] Initialize the Q table, set the initial Q value to zero, and update the Q value according to the feedback storage resource usage;

[0073] The Q value is updated using the following update formula:

[0074] ;

[0075] Wherein, is the reward value, indicating the improvement of storage resource utilization, is the discount factor, is the learning rate, through repeated learning and optimization, Q-learning can provide a gradually improved scheduling strategy, avoiding manual intervention and improving the use efficiency of storage resources. In addition, this algorithm can handle complex storage demand changes and adapt to changing workloads and usage environments, thereby achieving the best configuration of storage resources.

[0076] Please refer to the attached Figure 1 In a preferred embodiment of the present application, S4 includes the following steps:

[0077] Collect the temperature, error log and I / O delay information of the storage device and input them into the fault prediction model. By collecting the temperature, error log and I / O delay information of the storage device, the running status of the storage device can be monitored comprehensively, and these information can be collected in real time to provide rich data support for subsequent fault prediction, which helps to discover potential fault risks in advance and reduce the probability of system failure.

[0078] The state of the storage device is analyzed using a convolutional neural network (CNN), which is a deep learning algorithm mainly used for image processing and time series data analysis, to predict the risk of potential failures. By using CNN for failure prediction, features can be automatically extracted from multiple data sources such as temperature, error logs, and I / O delays, and potential failure risks can be identified. CNN can learn complex patterns in the data and predict the likelihood of device failure in advance. Through timely failure warning, the loss caused by device failure can be reduced, and the stability and reliability of the data center can be improved.

[0079] Please refer to the attached Figure 1 In a preferred embodiment of the present application, the analysis of the state of the storage device using a convolutional neural network (CNN) includes the following steps:

[0080] The health status features of the storage device are extracted and input into the CNN network for failure identification. By extracting health status features from multiple dimensions such as temperature, delay, and error rate, the running status of the storage device can be comprehensively understood. CNN can automatically identify complex patterns in these features and provide high-quality input data for subsequent failure prediction. Automatic feature extraction avoids the tedious process of manual feature extraction, improving the accuracy and efficiency of failure prediction.

[0081] The probability of failure is calculated, and the softmax function is used to output the prediction result of failure. When the prediction probability exceeds the preset threshold, an early warning is triggered. The definition of the softmax function is:

[0082] ;

[0083] where, is the original score output by the model (i.e., the output of CNN), is the probability of the corresponding category, is the base of the natural logarithm. By using the softmax function to calculate the probability of failure, the system can quantify the failure risk of the storage device and make decisions based on the actual failure probability. The preset threshold allows the warning system to be adjusted flexibly according to different fault tolerance requirements, ensuring the accuracy and timeliness of the warning trigger. This probability-based prediction and warning mechanism can improve the intelligence level of the system, helping operations personnel to identify potential failures in advance and reduce the risk of device downtime or data loss.

[0084] Please refer to the attached Figure 1 In a preferred embodiment of the present application, S5 includes the following steps:

[0085] According to the access frequency and importance of data, the data is divided into hot data, cold data and expired data, wherein the hot data refers to the data frequently accessed or modified, and the cold data refers to the data rarely accessed or long-term unmodified, by dividing the data into hot data, cold data and expired data according to the access frequency and importance, the efficient management of storage resources can be realized, the hot data is preferentially stored on the high-performance device to ensure the access speed; the cold data is stored on the low-cost device to save the storage cost; the clearance or archiving of the expired data effectively releases the storage space and avoids the occupation of valuable resources by useless data.

[0086] For hot data, it is preferentially stored on a high-performance storage device, for cold data, it is stored on a low-cost storage device, and expired data is periodically cleared or archived, storing hot data in a high-performance storage device can significantly improve data access efficiency, reduce latency and improve application response speed, storing cold data in a low-cost storage device can effectively reduce storage costs, this strategy can maximize the use of data center storage resources, ensure fast access of hot data, and still keep cold data in the system for reference without consuming high storage costs, finally, by periodically clearing or archiving expired data, a large amount of storage space can be released to avoid the occupation of storage resources by useless data.

[0087] Please refer to the accompanying Figure 1 In a preferred embodiment of the present application, S5 further comprises the following steps:

[0088] According to the data access mode, a hybrid storage architecture is used for data storage, and data is dynamically migrated between different storage levels, by using a hybrid storage architecture, the storage system can be optimized according to the data access mode and storage requirements, hot data is accessed quickly, and cold data is stored more economically, thereby realizing the optimization of storage cost and the improvement of performance, by intelligently managing the storage levels, hot data is always kept in high-performance storage devices, and cold data is stored at a lower cost. The dynamic migration mechanism enhances the flexibility and adaptability of the storage system, which can quickly respond to changes in data access mode and improve the utilization efficiency of storage resources;

[0089] Optimize data migration strategy using linear programming to minimize storage cost and meet maximum capacity constraints for each storage level, linear programming formula: let There are storage levels (such as SSD, HDD, etc.), the storage capacity of each storage level is , the storage cost is , the amount of stored data is , the goal of linear programming is to minimize storage cost:

[0090] ;

[0091] At the same time, the capacity constraints of each storage level need to be met:

[0092] ;

[0093] By solving this linear programming problem, the optimal data migration strategy can be obtained, which maximizes the reduction of storage costs under the premise of meeting performance requirements after data migration;

[0094] By considering the capacity limitations and performance constraints of the storage hierarchy, linear programming can provide an optimal solution to avoid over or under allocation of storage resources, improving the economic benefits and operational efficiency of the data center storage system.

[0095] In order to better understand the present application, the above will be described in detail in conjunction with specific embodiments.

[0096] Embodiment

[0097] Example 1: Storage space dynamic monitoring and early warning method based on deep learning prediction and reinforcement learning scheduling (basic embodiment)

[0098] In this embodiment, the system first realizes dynamic monitoring of the storage device by collecting real-time health data of the storage device (such as temperature, I / O load, response time and storage capacity usage). On this basis, the future storage demand is predicted using historical data and LSTM (Long Short Term Memory) model, so as to evaluate the storage pressure and resource bottleneck in advance. Next, the Q-learning algorithm is used for dynamic scheduling of storage resources, and the adaptive allocation of resources is realized through reinforcement learning. Finally, based on the collected device data, the convolutional neural network (CNN) model is used to predict device failure and trigger early warning before failure occurs. The optimization operation of the storage device does not involve cross-level storage or data migration strategy.

[0099] Advantages:

[0100] Accurate storage demand prediction and resource scheduling: through deep learning model prediction, the storage resource configuration is planned in advance to avoid system overload operation or resource waste.

[0101] Enhanced fault warning capability: through CNN for fault prediction, the accuracy and timeliness of fault detection can be improved, and the risk of data loss can be reduced.

[0102] Adaptive resource scheduling: Q-learning algorithm can adaptively adjust the storage resources, optimize the configuration of the storage device, and improve the reliability and performance of the system.

[0103] Embodiment 2: Storage space dynamic monitoring and early warning method with cross-tier storage optimization added

[0104] This embodiment adds cross-tier storage optimization function based on Embodiment 1. In addition to monitoring real-time data of storage devices and predicting storage demand using LSTM model, the system also classifies data into hot data, cold data and expired data based on data access frequency and importance. Hot data is preferentially stored on high-performance storage devices, while cold data is stored on low-cost storage devices, and expired data is periodically cleaned up or archived. In addition, the system uses linear programming algorithm to optimize data migration strategy, so as to dynamically migrate data between different storage tiers, optimize storage cost and performance.

[0105] Beneficial effects:

[0106] Optimized storage resource allocation: through cross-tier storage optimization, the system can intelligently allocate storage resources according to data access frequency, improve storage performance and reduce cost.

[0107] More fine-grained data storage management: hot data, high-frequency data and low-frequency data are stored in different tiers, effectively avoiding data storage redundancy and improving overall system performance.

[0108] Reduced storage cost: by storing cold data on low-cost storage devices, the system significantly reduces storage cost while meeting performance requirements.

[0109] Embodiment 3: Storage space dynamic monitoring and early warning method combined with self-healing mechanism and fault prediction

[0110] In this embodiment, in addition to storage demand prediction and resource scheduling, the system also introduces a self-healing mechanism. When the storage device fault prediction system (based on CNN fault prediction model) triggers a fault warning, the system not only sends an alarm message, but also automatically performs data migration or redundancy backup through the self-healing mechanism, migrating data from the faulty device to the healthy device to ensure the continuous operation of the system and the safety of the data. The combination of self-healing mechanism and fault prediction model can realize automatic response before the occurrence of faults.

[0111] Beneficial effects:

[0112] Enhanced fault tolerance: The addition of self-healing mechanism enables the storage system to automatically perform data migration and redundancy backup when a device fails, avoiding data loss due to hardware failure.

[0113] Real-time and automated response: The system can monitor fault risks in real time and respond automatically, reducing manual intervention after a fault occurs and reducing fault recovery time.

[0114] Enhanced system reliability: Through automated self-healing mechanisms, data centers can maintain stable operation when storage devices fail, improving overall system reliability.

[0115] Embodiment 4: Dynamic monitoring and early warning method for storage space based on global resource optimization

[0116] This embodiment combines storage resource scheduling and optimization among multiple data centers. Among multiple data centers, the system first monitors the storage device status of each data center in real time, predicts the storage demand of each data center using deep learning and reinforcement learning algorithms, and performs cross-data center storage resource scheduling through global optimization algorithms (such as global scheduling algorithms based on linear programming). This cross-data center resource optimization can dynamically adjust resources between different data centers according to load and demand, thereby achieving more efficient resource utilization and cost savings.

[0117] Benefits:

[0118] Cross-data center resource optimization: Through global resource scheduling optimization, the system can intelligently allocate storage resources among multiple data centers, improving resource utilization and reducing overall costs.

[0119] Enhanced storage efficiency and scalability: The system can dynamically adjust resource allocation to meet the storage needs of different data centers, improving the collaborative efficiency and overall scalability between data centers.

[0120] Reduced resource waste: Cross-data center scheduling and optimization effectively avoid over-provisioning or idling of storage resources, reducing system operation and maintenance costs.

[0121] Embodiment 5: Fault prediction and early warning method with fault-tolerant mechanism

[0122] In this embodiment, the system combines fault-tolerant mechanisms in the process of fault prediction and early warning. In addition to using CNN models for storage device fault prediction, the system also performs further fault-tolerant processing on the prediction results. For example, if the system predicts that the probability of a device failure reaches a preset threshold, it will replicate data in other devices through distributed storage to ensure data security. When multiple devices fail simultaneously, the system automatically switches to backup devices to avoid affecting the stability of the entire system.

[0123] Benefits:

[0124] Enhanced data protection: Fault-tolerant mechanisms can perform data backup or migration in a timely manner before faults occur, ensuring data security.

[0125] Improved system stability: When multiple devices fail, the system can automatically switch to backup devices to maintain stable operation of the system and avoid business interruptions.

[0126] Reducing loss due to failure: By fault-tolerant mechanisms, the system can maintain high availability even if a device failure occurs, reducing the loss due to failure.

[0127] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic monitoring and early warning of storage space of a data center, characterized in that, The method comprises the following steps: S1, collecting and monitoring the running state data of the data center storage device in real time; S2, predicting future storage demand based on collected historical storage demand data through a deep learning model; S3, based on the storage demand prediction result, setting the state space, action space and reward function of the reinforcement learning model, and using the Q-learning algorithm to dynamically schedule the storage resources to optimize the use of storage resources; S4, when potential failure occurs in the storage device, early warning is performed by using a fault prediction model, and automatic data migration or redundant backup is performed through a self-healing mechanism; S5, based on data access frequency and importance, data is divided into hot data, cold data and expired data, and data is automatically migrated to different storage levels; The data automatic migration adopts linear programming to optimize the data migration strategy to minimize the storage cost and meet the maximum capacity constraint of each storage level.

2. The data center storage space dynamic monitoring and early warning method of claim 1, wherein, The S1 comprises the following steps: Real-time acquisition of the running state data of the storage device, including temperature, I / O load, response time and storage capacity utilization; Input the running state data into the preset data storage system for subsequent analysis and prediction.

3. The method of claim 1, wherein the method further comprises: The S2 comprises the following steps: Use a long short-term memory network model to train historical storage demand data to predict data storage demand in the future, the training process includes using mean square error as a loss function to optimize the long short-term memory network model; Evaluate the trained long short-term memory network model and select the optimal long short-term memory network model for deployment.

4. The data center storage space dynamic monitoring and early warning method of claim 3, wherein, Using mean square error as a loss function to optimize the long short-term memory network model comprises the following steps: Calculate the error between the actual storage demand and the model prediction demand, and optimize the weight of the long short-term memory network through a back propagation algorithm; Minimize the following loss function: ； wherein, is the actual storage requirement, is the predicted value, is the number of samples.

5. The data center storage space dynamic monitoring and early warning method of claim 1, wherein, The S3 further comprises the following steps: Define the state space as the current usage state of the storage device, and define the action space as the scheduling operation of the storage resources; Define the reward function as the optimized use efficiency of the storage resources, and adjust the storage resource allocation strategy through the Q-learning algorithm.

6. The data center storage space dynamic monitoring and early warning method of claim 5, wherein, The storage resource scheduling using the Q-learning algorithm comprises the following steps: Initialize the Q table, set the initial Q value to zero, and update the Q value according to the feedback storage resource usage; Use the following update formula to update the Q value: ； wherein, is a reward value, indicating an improvement in storage resource utilization, is a discount factor, is a learning rate.

7. The method of claim 1, wherein the method further comprises: The S4 comprises the following steps: Collect the temperature, error log and I / O delay information of the storage device and input it into the fault prediction model; Use a convolutional neural network as a fault prediction model to analyze the input temperature, error log and I / O delay information of the storage device to predict the possible failure risk.

8. The data center storage space dynamic monitoring and early warning method of claim 7, wherein, Using a convolutional neural network to analyze the temperature, error log and I / O delay information of the storage device comprises the following steps: Extract the health status features of the storage device and input them into the convolutional neural network for fault identification; The failure risk is calculated using a softmax function to obtain a failure occurrence probability, and when the failure occurrence probability exceeds a preset threshold, a warning is triggered.

9. The method of claim 1, wherein the method further comprises: The S5 further includes the following steps: For hot data, it is preferentially stored on a high-performance storage device, for cold data, it is stored on a low-cost storage device, and expired data is periodically cleaned up or archived.

10. The data center storage space dynamic monitoring and early warning method of claim 1, wherein, The S5 further includes the following steps: According to the access mode of the data, a hybrid storage architecture is used for data storage, and data is dynamically migrated between different storage levels; A linear programming is used to optimize the data migration strategy to minimize the storage cost and meet the maximum capacity constraint of each storage level.

Images