Cache method and system for distributed storage system

Abstract

The present invention discloses a caching method and system for a distributed storage system, and relates to the field of storage caching technology. The system effectively identifies the usage pattern and performance change trend of cache data by analyzing and predicting using historical data. Pandas is used for data processing and statistical analysis, and Matplotlib and Seaborn are combined to create time series graphs, trend graphs, scatter plots, statistical charts, heat maps and regression graphs, which can clearly show the periodic changes in cache performance. The machine learning model constructed by integrating ARIMA model and LSTM model can accurately predict the future cache situation. This prediction capability not only reduces the risk of uneven cache utilization and increased access delay, but also improves the cache hit rate, thereby improving the response speed of the system and user experience. The prediction model is verified by evaluation indicators such as root mean square error, mean absolute error and coefficient of determination to ensure the reliability of its prediction performance.

Description

Technical Field

[0001] The present invention relates to the field of storage cache technology, and in particular to a cache method and system for a distributed storage system. Background Art

[0002] The cache of the distributed storage system is a key component to improve system performance and user experience. In large e-commerce platforms, the distributed storage system is responsible for storing and accessing massive amounts of data, while the cache system is responsible for temporarily storing frequently accessed data in memory to reduce access delays, improve system response speed and processing efficiency, and effectively utilize cache resources, which can not only significantly improve the overall performance of the system, but also reduce the load on storage devices.

[0003] In the Chinese invention application with the application publication number of 202010644027.X, a cache management method, a cache node and a distributed storage system are disclosed, and a cache management method, a cache node and a distributed storage system are provided. The method includes: pre-saving the RDMA connection information of at least one cache node, determining the storage information of the data to be cached; determining the cache node for caching the data to be cached; determining the storage node for caching the data to be cached to the cache node; sending the storage information of the data to be cached and the RDMA connection information of the determined cache node to the determined storage node, so that the determined storage node reads the data to be cached according to the storage information of the data to be cached, and caches the read data to be cached into the memory of the determined cache node through RDMA connection according to the RDMA connection information of the determined cache node. The present invention provides a cache management method, a cache node and a distributed storage system, which can improve the processing speed of the cache in the distributed storage system.

[0004] In combination with the prior art, the above application still has the following deficiencies:

[0005] Due to the lack of comprehensive evaluation and early warning mechanisms, the cache system is often not easy to respond and handle in a timely manner when cache performance anomalies occur. For example, when the cache utilization rate is too high, if the cache capacity is not increased or the cache strategy is not adjusted in time, the system access delay may increase significantly, affecting the user experience; when the cache hit rate decreases, if the cache strategy is not optimized in time, more data may need to be read from the backend storage, increasing the load on the storage device. These anomalies not only affect the stability and performance of the system, but may also lead to user churn and business losses. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a cache method and system for a distributed storage system, which solves the problems mentioned in the background technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: including a parameter acquisition module, a data analysis and prediction module, a calculation analysis module, an intelligent evaluation module and a cache optimization module;

[0008] The parameter collection module collects cache data of each distributed storage node in real time by deploying monitoring tools at each node of the distributed storage system of the e-commerce platform, integrates the collected cache data into the database, and pre-processes the collected cache data;

[0009] The data analysis and prediction module uses historical data to analyze the usage patterns and performance change trends of cache data, and then predicts the future cache status of the distributed storage system by building a machine learning model;

[0010] The calculation and analysis module is used to construct a cache utilization prediction formula, an access delay prediction formula and a cache hit rate prediction formula, and substitute the acquired cache data into the relevant formulas to calculate and obtain the cache utilization CU(t) at time t, the access delay AL(t) at time t and the cache hit rate CHR(t) at time t, and perform associated calculations to obtain a comprehensive early warning score ZHYJ;

[0011] The intelligent evaluation module is used to preset the cache threshold H and the obtained comprehensive warning score ZHYJ, and perform a first comparative evaluation to identify the cache status of the distributed storage system;

[0012] The cache optimization module is used to be triggered when an abnormality in the current system cache is identified. By introducing the cache write rate CWR and the cache read rate CRR and the obtained comprehensive warning score ZHYJ, a correlation calculation is performed to obtain the deep analysis index SDFX, and a preset warning threshold Y is performed for secondary evaluation, in-depth analysis of the cache situation of each distributed storage system, and generation of control measures.

[0013] Preferably, the parameter acquisition module includes a data acquisition unit, a data storage unit and a data preprocessing unit;

[0014] The data collection unit is used to deploy Prometheus, Telegraf and Grafana monitoring tools on each node of each distributed storage system, and collect cache data of each distributed storage system node in real time, wherein the cache data includes cache write rate CWR, cache read rate CRR, cache utilization CU, access delay AL, and cache hit rate CHR;

[0015] Prometheus is an open source system monitoring and alarm tool that supports the collection and storage of multiple indicators;

[0016] Telegraf is a plug-in tool for collecting various system performance indicators;

[0017] Grafana is an open source analytics and monitoring platform for visualizing monitoring data;

[0018] Download and install the Prometheus server and Telegraf agent in each distributed storage system, and download and install Grafana. Edit the prometheus.yml configuration file to specify the target nodes and endpoints to be monitored. Edit the telegraf.conf configuration file to set monitoring plug-ins and parameters, such as cpu, mem, and disk. Connect Prometheus as a data source through Grafana and create a dashboard to display monitoring data.

[0019] The data storage unit is used to build an InfluxDB time series database, send the collected cache data to the InfluxDB time series database through Telegraf, store the cache data, and monitor the stored cache data through Graphite;

[0020] Specific implementation steps: Automatically capture cache data through Prometheus, then send the collected cache data to the InfluxDB time series database through Telegraf, and use Grafana to read the cache data from Prometheus and InfluxDB for visualization;

[0021] The data preprocessing unit is used to detect and remove abnormal data using a statistical method for the cache data stored in the database, and then convert the cache data after taking out the abnormal data, converting the data from the original format to the format required for analysis.

[0022] Preferably, the data analysis and prediction module includes a data analysis unit and a prediction unit;

[0023] The data analysis unit is used to process, clarify and statistically analyze the cache data using Pandas based on the historical cache data, and to create time series graphs, trend graphs, scatter graphs, statistical charts, heat maps and regression graphs for the cache data using Matplotlib and Seaborn to identify periodic changes in cache performance;

[0024] The prediction unit predicts the cache performance of the distributed storage system by integrating and building a machine learning model based on the ARIMA model and the LSTM model;

[0025] The ARIMA model determines the p, d and q parameters of the model by observing the autocorrelation function ACF and the partial autocorrelation function PACF, and fits the ARIMA model using historical data;

[0026] The LSTM model converts the time series data into the LSTM input format of a multi-time step input sequence, defines the LSTM network structure and number of layers, and then uses historical cache data for training;

[0027] Use the root mean square error, mean absolute error, and determination coefficient of the trained machine learning model to evaluate the prediction performance of the machine learning model, then deploy the trained model to the production environment, update the model regularly to reflect the latest data trends, use monitoring tools to monitor the difference between the model's prediction results and actual data in real time, and adjust the model or take other measures in a timely manner.

[0028] Preferably, the calculation and analysis module includes a cache calculation unit and a comprehensive calculation unit;

[0029] The cache calculation unit includes a utilization calculation unit, a delay calculation unit and a hit rate calculation unit;

[0030] The utilization calculation unit constructs a cache utilization prediction formula, and performs calculation prediction based on the historical cache utilization CU and the current cache write rate CWR and cache read rate CRR to obtain the cache utilization CU(t) at time t;

[0031] The cache utilization CU(t) at time t is calculated and obtained by the following prediction formula:

[0032] ;

[0033] In the formula, Indicates the first adjustment coefficient, which is used to reflect the impact of cache write and read on utilization, CWR t represents the cache write rate CWR at time t, CRR t Represents the cache read rate CRR at time t.

[0034] Preferably, the delay calculation unit is used to construct an access delay prediction formula to calculate the access delay AL(t) at time t based on the currently predicted cache utilization CU(t) at time t and the cache hit rate CHR(t) at time t;

[0035] The access delay AL(t) at time t is obtained by the following prediction formula:

[0036] ;

[0037] In the formula, represents the second adjustment coefficient, reflecting the influence of the cache utilization CU(t) at time t and the cache hit rate CHR(t) at time t on the access latency AL(t) at time t.

[0038] Preferably, the hit rate calculation unit is used to construct a cache hit rate prediction formula, and calculates and predicts to obtain the cache hit rate CHR(t) at time t by combining the access delay AL(t) at time t;

[0039] The cache hit rate CHR(t) at time t is obtained by the following prediction formula:

[0040] ;

[0041] In the formula, represents the third adjustment coefficient, reflecting the impact of the access delay AL on the cache hit rate CHR(t) at time t.

[0042] Preferably, the comprehensive calculation unit obtains the cache utilization warning threshold CU by calculating the average value and standard deviation of the cache utilization CU(t) at time t, the access delay AL(t) at time t, and the cache hit rate CHR(t) at time t based on historical cache data and performance tests. h , access delay warning threshold AL h and cache hit rate warning threshold CHR h The cache utilization CU(t) at time t, the access delay AL(t) at time t, and the cache hit rate CHR(t) at time t are correlated and calculated to obtain the comprehensive warning score ZHYJ;

[0043] The comprehensive early warning score ZHYJ is calculated by the following algorithm formula:

[0044] ;

[0045] Wherein, a1, a2 and a3 represent the preset weight values ​​of cache utilization CU(t) at time t, access latency AL(t) at time t and cache hit rate CHR(t) at time t, respectively, and their specific values ​​are set by the user, and a1+a2+a3=1.

[0046] Preferably, the intelligent evaluation module is used to determine the average value and standard deviation of cache data of each distributed storage system under normal conditions based on the historical cache data of each distributed storage system, perform a first comparative evaluation of the preset cache threshold H and the obtained comprehensive warning score ZHYJ, and analyze the cache situation in each distributed storage system. The specific evaluation results are as follows;

[0047] When the comprehensive warning score ZHYJ ≥ cache threshold H, it indicates that the cache performance of the current distributed storage system is abnormal. At this time, the first warning information is generated, and the secondary evaluation is performed for in-depth analysis;

[0048] When the comprehensive warning score ZHYJ is less than the cache threshold H, it indicates that the current distributed storage system performance is normal and no warning needs to be generated.

[0049] Preferably, the cache optimization module includes a depth analysis unit and a depth evaluation unit;

[0050] The depth analysis unit is used to introduce a cache write rate warning threshold CWR h Cache read rate warning threshold CRR h , and combined with the comprehensive early warning score ZHYJ, perform related calculations to obtain the in-depth analysis index SDFX, and conduct detailed analysis of the performance of the distributed storage system;

[0051] The depth analysis indicator SDFX is calculated and obtained by the following algorithm formula;

[0052] ;

[0053] Where CWR t represents the cache write rate at time t, CRR t Indicates the cache read rate at time t, and the cache write rate warning threshold CWR h Cache read rate warning threshold CRR h , obtained by averaging the historical data of cache write rate and cache read rate;

[0054] The depth evaluation unit is used to perform a secondary comparative evaluation based on the safety limit of the historical cache data, the preset warning threshold Y and the acquired depth analysis index SDFX, and deeply analyze the potential performance of the distributed storage system. The specific evaluation scheme is as follows;

[0055] When the in-depth analysis indicator SDFX> warning threshold Y, it means that the cache operation status of the current distributed storage system is abnormal. At this time, Prometheus and Grafana are used to set up automated monitoring and warning rules. When an abnormality is identified, the second warning information is automatically sent to the administrator. At the same time, the Ansible automation tool is used to write scripts to automatically execute adjustment measures according to the warning situation. For example, when the cache utilization rate is too high, the cache capacity is automatically increased or the cache strategy is adjusted;

[0056] When the depth analysis indicator SDFX ≤ the warning threshold Y, it indicates that the cache operation status of the current distributed storage system is normal.

[0057] A caching method for a distributed storage system includes the following steps:

[0058] S1. First, a monitoring tool is deployed on each node of the distributed storage system of the e-commerce platform to collect cache data of each distributed storage node in real time, integrate the collected cache data into the database, and pre-process the collected cache data;

[0059] S2. Use historical data to analyze the usage patterns and performance change trends of cache data, and then build a machine learning model to predict the future cache status of the distributed storage system;

[0060] S3, construct a cache utilization prediction formula, an access delay prediction formula and a cache hit rate prediction formula, substitute the obtained cache data into the relevant formulas, calculate and obtain the cache utilization CU(t) at time t, the access delay AL(t) at time t and the cache hit rate CHR(t) at time t, and perform related calculations to obtain the comprehensive warning score ZHYJ;

[0061] S4, the preset cache threshold H and the obtained comprehensive warning score ZHYJ are compared and evaluated for the first time to identify the cache situation of the distributed storage system;

[0062] S5. Finally, by introducing the cache write rate CWR and the cache read rate CRR and the obtained comprehensive warning score ZHYJ, a correlation calculation is performed to obtain the deep analysis index SDFX, and the warning threshold Y is preset for secondary evaluation, the cache situation of each distributed storage system is deeply analyzed, and control measures are generated.

[0063] The present invention provides a cache method and system for a distributed storage system, which has the following beneficial effects:

[0064] (1) By deploying a parameter collection module in the distributed storage system, the system can monitor and collect cache data of each node in real time. Through the monitoring tools Prometheus, Telegraf, and Grafana, the cache data can be automatically captured and visualized, greatly improving the system's sensitivity to changes in cache performance. Real-time collection and integration of cache data helps to detect and respond to abnormal situations in a timely manner, ensuring the stable operation of the system. In addition, the data preprocessing module removes abnormal data through statistical methods, ensuring the accuracy and reliability of subsequent analysis. This not only improves the overall performance of the system, but also provides a reliable data foundation for subsequent analysis and optimization.

[0065] (2) The system effectively identifies the usage patterns and performance change trends of cache data by using historical data for analysis and prediction. Pandas is used for data processing and statistical analysis, and Matplotlib and Seaborn are used to create time series graphs, trend graphs, scatter plots, statistical charts, heat maps, and regression graphs, which can clearly show the periodic changes in cache performance. The machine learning model built by integrating the ARIMA model and the LSTM model can accurately predict future cache conditions. This prediction capability not only reduces the risk of uneven cache utilization and increased access latency, but also improves the cache hit rate, thereby improving the system's response speed and user experience. The prediction model is verified by evaluation indicators such as root mean square error, mean absolute error, and coefficient of determination to ensure the reliability of its prediction performance. Overall, the prediction and analysis capabilities enhance the system's foresight and adaptability, enabling the system to take measures in advance for optimization and adjustment.

[0066] (3) The system constructs a prediction formula for cache utilization, access latency, and cache hit rate, performs a comprehensive evaluation, and generates a comprehensive warning score ZHYJ, thereby achieving comprehensive monitoring and evaluation of cache performance. By analyzing historical cache data and performance tests, the mean and standard deviation of cache utilization, access latency, and cache hit rate are calculated, the corresponding warning thresholds are obtained, and a comprehensive warning is performed in combination with the current prediction data. When the comprehensive warning score ZHYJ exceeds the cache threshold H, it indicates that the cache performance of the current distributed storage system is abnormal. At this time, a secondary evaluation is performed for in-depth analysis. The cache optimization module introduces a cache write rate warning threshold CWR h and cache read rate warning threshold CRR h , combined with the comprehensive early warning score ZHYJ, the in-depth analysis indicator SDFX is calculated and evaluated, and corresponding control measures are generated. This multi-level evaluation and optimization mechanism ensures that the system can flexibly adjust the cache strategy when facing different loads and access modes, ensuring the stability and efficiency of the system. The introduction of automated monitoring and early warning mechanisms enables the system to promptly alarm and automatically take measures when anomalies are detected, reducing the time and cost of manual intervention and improving the reliability and maintenance efficiency of the system. By setting automated monitoring and early warning rules through Prometheus and Grafana, combined with adjustment scripts written by the Ansible automation tool, automated cache adjustment can be achieved, further improving the system's adaptability and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The figure is a schematic diagram of the cache system flow of the distributed storage system of the present invention.

[0068] Figure 2The figure is a schematic diagram of the steps of the caching method of the distributed storage system of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Example 1: Please refer to Figure 1 , the present invention provides a cache system for a distributed storage system. To achieve the above purpose, the present invention is implemented through the following technical solutions: including a parameter acquisition module, a data analysis and prediction module, a calculation analysis module, an intelligent evaluation module and a cache optimization module;

[0071] The parameter collection module deploys monitoring tools on each node of the distributed storage system of the e-commerce platform to collect cache data of each distributed storage node in real time, integrate the collected cache data into the database, and pre-process the collected cache data;

[0072] The data analysis and prediction module uses historical data to analyze the usage patterns and performance change trends of cache data, and then builds a machine learning model to predict the future cache status of the distributed storage system;

[0073] The calculation and analysis module is used to construct cache utilization prediction formulas, access delay prediction formulas and cache hit rate prediction formulas, and substitute the acquired cache data into the relevant formulas to calculate and obtain the cache utilization CU(t) at time t, the access delay AL(t) at time t and the cache hit rate CHR(t) at time t, and perform related calculations to obtain the comprehensive warning score ZHYJ;

[0074] The intelligent evaluation module is used to preset the cache threshold H and the obtained comprehensive warning score ZHYJ, and conduct the first comparative evaluation to identify the cache situation of the distributed storage system;

[0075] The cache optimization module is used to be triggered when the current system cache anomaly is identified. By introducing the cache write rate CWR and the cache read rate CRR and the obtained comprehensive warning score ZHYJ, a correlation calculation is performed to obtain the deep analysis index SDFX, and the preset warning threshold Y is used for secondary evaluation. The cache situation of each distributed storage system is deeply analyzed and control measures are generated.

[0076] In this embodiment, the system monitors and collects cache data of each node of the distributed storage system of the e-commerce platform in real time through the deployment of the parameter acquisition module, and integrates it into the database for preprocessing. This process significantly improves the availability and accuracy of cache data, ensuring the reliability of subsequent analysis and prediction. The use of real-time monitoring tools such as Prometheus, Telegraf and Grafana enables the system to quickly respond to changes in cache performance and detect and handle abnormal situations in a timely manner. This improvement not only improves the stability of the system. The data analysis and prediction module uses historical data to analyze the usage pattern and performance change trend of cache data, and predicts future cache conditions by building machine learning models such as ARIMA and LSTM. Compared with the traditional static configuration method, this data-driven dynamic prediction method can more accurately predict the system's cache requirements and performance bottlenecks, reducing the risk of uneven cache utilization and increased access delays. Through visualization tools such as Matplotlib and Seaborn, the analysis results are intuitive and easy to understand, enabling system maintenance personnel to better understand cache performance changes and formulate more effective cache strategies. The calculation and analysis module and the intelligent evaluation module calculate the cache performance index at time t by constructing the cache utilization prediction formula, the access delay prediction formula, and the cache hit rate prediction formula, and generate a comprehensive early warning score ZHYJ. When an anomaly is detected, the cache optimization module will conduct a secondary evaluation, calculate the depth analysis index SDFX by introducing the cache write rate CWR and the cache read rate CRR, and formulate corresponding control measures. Compared with traditional manual intervention methods, this automated evaluation and optimization mechanism can respond to system performance changes more quickly, reduce human errors, and improve the system's adaptability and maintenance efficiency. Overall, the system has greatly improved the performance and reliability of the distributed storage system through comprehensive evaluation and automated optimization, ensuring efficient operation under different loads and access modes.

[0077] Example 2: This example is an explanation of Example 1. Figure 1 ,Specifically: the parameter acquisition module includes a data acquisition unit, a data storage unit and a data preprocessing unit;

[0078] The data collection unit is used to deploy Prometheus, Telegraf and Grafana monitoring tools on each node of each distributed storage system to collect cache data of each distributed storage system node in real time. The cache data includes cache write rate CWR, cache read rate CRR, cache utilization CU, access latency AL, and cache hit rate CHR.

[0079] Prometheus is an open source system monitoring and alarm tool that supports the collection and storage of multiple indicators;

[0080] Telegraf is a plug-in tool for collecting various system performance indicators;

[0081] Grafana is an open source analytics and monitoring platform for visualizing monitoring data;

[0082] Download and install the Prometheus server and Telegraf agent in each distributed storage system, and download and install Grafana. Edit the prometheus.yml configuration file to specify the target nodes and endpoints to be monitored. Edit the telegraf.conf configuration file to set monitoring plug-ins and parameters, such as cpu, mem, and disk. Connect Prometheus as a data source through Grafana and create a dashboard to display monitoring data.

[0083] The data storage unit is used to build the InfluxDB time series database, send the collected cache data to the InfluxDB time series database through Telegraf, store the cache data, and monitor the stored cache data through Graphite;

[0084] Specific implementation steps: Automatically capture cache data through Prometheus, then send the collected cache data to the InfluxDB time series database through Telegraf, and use Grafana to read the cache data from Prometheus and InfluxDB for visualization;

[0085] The data preprocessing unit is used to detect and remove abnormal data using statistical methods on the cached data stored in the database, and then convert the cached data after taking out the abnormal data, converting the data from the original format to the format required for analysis.

[0086] In this embodiment, the system realizes the real-time collection and preprocessing of cache data of each node of the distributed storage system through the parameter collection module. By deploying open source monitoring tools such as Prometheus, Telegraf and Grafana, the efficient collection and visual display of system data are ensured. The combination of these tools not only improves the accuracy and real-time performance of data collection, but also provides high-quality basic data for subsequent data analysis and prediction. Compared with the traditional manual data collection method, this automated and real-time collection method significantly improves work efficiency and data quality. The data analysis and prediction module uses historical data and machine learning models to conduct in-depth analysis and accurate prediction of the usage pattern and performance change trend of cache data. Through data visualization, such as time series graphs, trend graphs, etc., the periodic changes and potential problems of cache performance are clear at a glance. Compared with the traditional static configuration method, the dynamic prediction model can more accurately predict future cache requirements, reduce the risk of uneven cache utilization and access delay, thereby improving the response speed and resource utilization of the system. The calculation analysis module and the intelligent evaluation module calculate the real-time cache performance indicators by constructing the prediction formula of cache utilization, access delay and cache hit rate, and generate a comprehensive early warning score ZHYJ. When the system detects a cache anomaly, the cache optimization module will conduct a secondary evaluation by introducing the cache write rate and cache read rate, calculating the deep analysis indicator SDFX, and formulating corresponding control measures.

[0087] Example 3: This example is explained in Example 5. Please refer to Figure 1 ,Specifically: the data analysis and prediction module includes a data analysis unit and a prediction unit;

[0088] The data analysis unit is used to process, clarify and statistically analyze cache data based on historical cache data using Pandas, and to create time series graphs, trend graphs, scatter plots, statistical charts, heat maps and regression graphs for cache data using Matplotlib and Seaborn to identify periodic changes in cache performance;

[0089] The prediction unit predicts the cache performance of the distributed storage system by integrating and building a machine learning model based on the ARIMA model and the LSTM model;

[0090] The ARIMA model determines the p, d, and q parameters of the model by observing the autocorrelation function ACF and the partial autocorrelation function PACF, and fits the ARIMA model using historical data;

[0091] The LSTM model converts the time series data into the LSTM input format of a multi-time step input sequence, defines the LSTM network structure and number of layers, and then uses historical cache data for training;

[0092] Use the root mean square error, mean absolute error, and determination coefficient of the trained machine learning model to evaluate the prediction performance of the machine learning model, then deploy the trained model to the production environment, update the model regularly to reflect the latest data trends, use monitoring tools to monitor the difference between the model's prediction results and actual data in real time, and adjust the model or take other measures in a timely manner.

[0093] In this embodiment, the system can deeply understand the performance and change trend of the cache in the distributed storage system through the precise operation of the data analysis and prediction modules. The data analysis unit uses Pandas for data processing and statistical analysis, and combines Matplotlib and Seaborn to create a variety of charts, such as time series charts, trend charts and heat maps, to effectively identify the periodic changes and potential problems of cache performance. The prediction unit builds a machine learning model through the ARIMA model and the LSTM model to accurately predict the cache performance. The ARIMA model determines parameters based on the autocorrelation function and the partial autocorrelation function, which is suitable for fitting historical data, while the LSTM model converts the time series data into a format suitable for deep learning, and ensures the accuracy and stability of the model through training and evaluation. These technical means significantly improve the prediction accuracy and real-time performance compared to traditional static analysis methods. By deploying the trained model to the production environment and updating it regularly to reflect the latest data trends, the system can quickly respond to environmental changes, thereby optimizing cache utilization and reducing access latency. The real-time monitoring and difference analysis of the monitoring tool ensure the consistency of the model prediction results with the actual data, further improving the system's adaptability and stability. In summary, these technical improvements not only enhance the system's performance optimization and data analysis capabilities, but also effectively reduce operation and maintenance costs and human errors, bringing significant efficiency and effectiveness improvements to distributed storage systems.

[0094] Example 4: This example is an explanation of Example 2. Figure 1 ,Specifically: the computing and analysis module includes a cache computing unit and a ,comprehensive computing unit;

[0095] The cache calculation unit includes a utilization calculation unit, a delay calculation unit and a hit rate calculation unit;

[0096] The utilization calculation unit constructs a cache utilization prediction formula, and performs calculation and prediction based on the historical cache utilization CU and the current cache write rate CWR and cache read rate CRR to obtain the cache utilization CU(t) at time t;

[0097] The cache utilization CU(t) at time t is calculated using the following prediction formula:

[0098] ;

[0099] In the formula, Indicates the first adjustment coefficient, which is used to reflect the impact of cache write and read on utilization, CWR t represents the cache write rate CWR at time t, CRR t Represents the cache read rate CRR at time t.

[0100] The delay calculation unit is used to construct an access delay prediction formula to calculate the access delay AL(t) at time t based on the currently predicted cache utilization CU(t) at time t and the cache hit rate CHR(t) at time t.

[0101] The access delay AL(t) at time t is obtained by the following prediction formula;

[0102] ;

[0103] In the formula, represents the second adjustment coefficient, reflecting the influence of the cache utilization CU(t) at time t and the cache hit rate CHR(t) at time t on the access latency AL(t) at time t.

[0104] The hit rate calculation unit is used to construct a cache hit rate prediction formula, and calculates and predicts the cache hit rate CHR(t) at time t by combining the access delay AL(t) at time t.

[0105] The cache hit rate CHR(t) at time t is obtained by the following prediction formula;

[0106] ;

[0107] In the formula, represents the third adjustment coefficient, reflecting the impact of the access delay AL on the cache hit rate CHR(t) at time t.

[0108] The comprehensive calculation unit calculates the average and standard deviation of the cache utilization CU(t) at time t, the access delay AL(t) at time t, and the cache hit rate CHR(t) at time t based on historical cache data and performance tests, and obtains the cache utilization warning threshold CU h , access delay warning threshold AL h and cache hit rate warning threshold CHR h The cache utilization CU(t) at time t, the access delay AL(t) at time t, and the cache hit rate CHR(t) at time t are correlated and calculated to obtain the comprehensive warning score ZHYJ;

[0109] The comprehensive early warning score ZHYJ is calculated by the following algorithm formula;

[0110] ;

[0111] Wherein, a1, a2 and a3 represent the preset weight values ​​of cache utilization CU(t) at time t, access latency AL(t) at time t and cache hit rate CHR(t) at time t, respectively, and their specific values ​​are set by the user, and a1+a2+a3=1.

[0112] In this embodiment, the system significantly improves the prediction and evaluation capabilities of the cache performance of the distributed storage system through the calculation and analysis module through refined calculation and analysis methods. The cache calculation unit of this module includes a utilization calculation unit, a delay calculation unit and a hit rate calculation unit, which respectively calculate the cache utilization CU (t) at time t, the access delay AL (t) at time t and the cache hit rate CHR (t) at time t by constructing accurate prediction formulas. The design of these units not only combines historical data and current dynamic indicators, but also considers the impact of write and read rates on cache utilization, the role of delay and hit rate on access delay, and the feedback of delay on hit rate, thereby providing a comprehensive performance evaluation basis. The improved technology of this calculation and analysis module can realize dynamic adjustment and detailed early warning mechanism. Unlike traditional technical means, these technologies can reflect the actual status of the cache system in real time through precise mathematical models and adjustment coefficients, and flexibly adjust the early warning threshold according to the actual monitoring results. This improvement not only improves the accuracy of the prediction, but also enhances the response speed of the system, so that when facing cache performance problems, it can be adjusted and optimized more quickly, thereby improving the stability and efficiency of the system. This dynamic and comprehensive performance evaluation and optimization mechanism has brought significant performance improvements and enhanced operation and maintenance efficiency to distributed storage systems.

[0113] Example 5: This example is an explanation of Example 4. Figure 1 ,Specifically: the intelligent evaluation module is used to determine the average value and standard deviation of the cache data of each ,distributed storage system under normal state based on the historical cache data of each ,distributed storage system, and the first comparative evaluation of the preset cache threshold H and the ,obtained comprehensive warning score ZHYJ is performed to analyze the cache situation in each ,distributed storage system. The specific evaluation results are as follows;

[0114] When the comprehensive warning score ZHYJ ≥ cache threshold H, it indicates that the cache performance of the current distributed storage system is abnormal. At this time, the first warning information is generated, and the secondary evaluation is performed for in-depth analysis;

[0115] When the comprehensive warning score ZHYJ is less than the cache threshold H, it indicates that the current distributed storage system performance is normal and no warning needs to be generated.

[0116] In this embodiment, the system realizes a comprehensive analysis and intelligent early warning of the cache performance of the distributed storage system by combining historical data with real-time monitoring results through an intelligent evaluation module. The core of this module is to set the cache threshold H through statistical analysis of historical cache data and conduct the first comparative evaluation with the comprehensive early warning score ZHYJ, so as to evaluate the state of the system and detect anomalies. Compared with the traditional single early warning mechanism, this multi-level evaluation method not only provides detailed cache state monitoring, but also realizes the automation and intelligence of anomaly detection. The intelligent evaluation module sets the cache threshold H by calculating the average value and standard deviation of historical data. Compared with the technical means that rely on the fixed cache threshold H in the past, this method provides a more dynamic and flexible anomaly detection mechanism. Traditional methods usually rely on fixed performance thresholds for monitoring, and often cannot adapt to the changing trends and fluctuations of cache data in the actual environment. The intelligent evaluation module sets a dynamic threshold that adapts to the current cache state through analysis based on historical data, making the judgment of abnormal state more accurate and effective. Secondly, the two-level early warning mechanism designed in this module further enhances the response capability to system anomalies. When the comprehensive early warning score ZHYJ exceeds the preset cache threshold H, the system will automatically perform in-depth analysis and perform more detailed performance diagnosis. Finally, the combination of the intelligent evaluation module and the secondary evaluation mechanism provides a scientific basis for the continuous optimization of cache performance. By introducing the cache threshold H and the early warning mechanism of deep analysis, in-depth analysis can be performed and control measures can be generated when cache performance is abnormal.

[0117] Example 6: This example is an explanation of Example 5. Please refer to Figure 1 ,Specifically: the cache optimization module includes a depth analysis unit and a depth evaluation unit;

[0118] The deep analysis unit is used to introduce the cache write rate warning threshold CWR h Cache read rate warning threshold CRR h , and combined with the comprehensive early warning score ZHYJ, perform related calculations to obtain the in-depth analysis index SDFX, and conduct detailed analysis of the performance of the distributed storage system;

[0119] The depth analysis indicator SDFX is calculated using the following algorithm formula;

[0120] ;

[0121] Where CWR t represents the cache write rate at time t, CRR t Indicates the cache read rate at time t, and the cache write rate warning threshold CWR h Cache read rate warning threshold CRR h, obtained by averaging the historical data of cache write rate and cache read rate;

[0122] The depth evaluation unit is used to perform a secondary comparative evaluation based on the safety limit of the historical cache data, the preset warning threshold Y and the acquired depth analysis index SDFX, and deeply analyze the potential performance of the distributed storage system. The specific evaluation scheme is as follows;

[0123] When the in-depth analysis indicator SDFX> warning threshold Y, it means that the cache operation status of the current distributed storage system is abnormal. At this time, Prometheus and Grafana are used to set up automated monitoring and warning rules. When an abnormality is identified, the second warning information is automatically sent to the administrator. At the same time, the Ansible automation tool is used to write scripts to automatically execute adjustment measures according to the warning situation. For example, when the cache utilization rate is too high, the cache capacity is automatically increased or the cache strategy is adjusted;

[0124] When the depth analysis indicator SDFX ≤ the warning threshold Y, it indicates that the cache operation status of the current distributed storage system is normal.

[0125] In this embodiment, the system significantly improves the performance optimization capability and management efficiency of the distributed storage system through the effective combination of the deep analysis unit and the deep evaluation unit of the cache optimization module. This module introduces a dynamic cache write rate warning threshold CWR h and cache read rate warning threshold CRR h , the in-depth analysis index SDFX is calculated through the comprehensive early warning score ZHYJ, so as to conduct a more detailed and dynamic analysis of cache performance. Compared with the traditional static performance monitoring method, this dynamic early warning and in-depth analysis method can not only reflect the fluctuation of cache performance in real time, but also automatically generate alarms for performance anomalies and provide specific optimization measures, which significantly improves the response speed and processing effect of performance problems. By utilizing the automated monitoring capabilities of Prometheus and Grafana, as well as the automated adjustment mechanism of Ansible, this module realizes real-time monitoring and intelligent regulation of the performance of distributed storage systems, thereby ensuring that the system can be adjusted and optimized quickly and effectively when facing complex cache performance problems. The implementation of this module not only improves the automation level of cache performance management, but also brings more accurate identification and efficient solution to system performance problems through refined performance analysis and optimization methods, greatly improving the stability and performance of distributed storage systems.

[0126] Example 7: Please refer to Figure 1 and Figure 2 , a caching method for a distributed storage system, comprising the following steps:

[0127] S1. First, a monitoring tool is deployed on each node of the distributed storage system of the e-commerce platform to collect cache data of each distributed storage node in real time, integrate the collected cache data into the database, and pre-process the collected cache data;

[0128] S2. Use historical data to analyze the usage patterns and performance change trends of cache data, and then build a machine learning model to predict the future cache status of the distributed storage system;

[0129] S3, construct a cache utilization prediction formula, an access delay prediction formula and a cache hit rate prediction formula, substitute the obtained cache data into the relevant formulas, calculate and obtain the cache utilization CU(t) at time t, the access delay AL(t) at time t and the cache hit rate CHR(t) at time t, and perform related calculations to obtain the comprehensive warning score ZHYJ;

[0130] S4, the preset cache threshold H and the obtained comprehensive warning score ZHYJ are compared and evaluated for the first time to identify the cache situation of the distributed storage system;

[0131] S5. Finally, by introducing the cache write rate CWR and the cache read rate CRR and the obtained comprehensive warning score ZHYJ, a correlation calculation is performed to obtain the deep analysis index SDFX, and the warning threshold Y is preset for secondary evaluation, the cache situation of each distributed storage system is deeply analyzed, and control measures are generated.

[0132] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A cache system of a distributed storage system, characterized in that: It includes parameter acquisition module, data analysis and prediction module, calculation analysis module, intelligent evaluation module and cache optimization module; The parameter collection module collects cache data of each distributed storage node in real time by deploying monitoring tools at each node of the distributed storage system of the e-commerce platform, integrates the collected cache data into the database, and pre-processes the collected cache data; The data analysis and prediction module uses historical data to analyze the usage patterns and performance change trends of cache data, and then predicts the future cache status of the distributed storage system by building a machine learning model; The calculation and analysis module is used to construct a cache utilization prediction formula, an access delay prediction formula and a cache hit rate prediction formula, and substitute the acquired cache data into the relevant formulas to calculate and obtain the cache utilization CU(t) at time t, the access delay AL(t) at time t and the cache hit rate CHR(t) at time t, and perform associated calculations to obtain a comprehensive early warning score ZHYJ; The computing and analysis module includes a cache computing unit and a comprehensive computing unit; The cache calculation unit includes a utilization calculation unit, a delay calculation unit and a hit rate calculation unit; The utilization calculation unit constructs a cache utilization prediction formula, and performs calculation prediction based on the historical cache utilization CU and the current cache write rate CWR and cache read rate CRR to obtain the cache utilization CU(t) at time t; The cache utilization CU(t) at time t is calculated and obtained by the following prediction formula: CU(t)=CU t-1 +α·(CWR t -CRR t ); In the formula, α represents the first adjustment coefficient, which is used to reflect the impact of cache write and read on utilization, CWR t represents the cache write rate CWR at time t, CRR t represents the cache read rate CRR at time t; The delay calculation unit is used to construct an access delay prediction formula to calculate the access delay AL(t) at time t based on the currently predicted cache utilization CU(t) at time t and the cache hit rate CHR(t) at time t; The access delay AL(t) at time t is obtained by the following prediction formula: Wherein, β represents the second adjustment coefficient, which reflects the influence of the cache utilization CU(t) at time t and the cache hit rate CHR(t) at time t on the access delay AL(t) at time t; The hit rate calculation unit is used to construct a cache hit rate prediction formula, and calculates and predicts the cache hit rate CHR(t) at time t by combining the access delay AL(t) at time t; The cache hit rate CHR(t) at time t is obtained by the following prediction formula: Wherein, γ represents the third adjustment coefficient, which reflects the impact of the access delay AL on the cache hit rate CHR(t) at time t; The comprehensive calculation unit obtains the cache utilization rate warning threshold CU by calculating the average value and standard deviation of the cache utilization rate CU(t) at time t, the access delay AL(t) at time t, and the cache hit rate CHR(t) at time t based on historical cache data and performance tests. h , access delay warning threshold AL h and cache hit rate warning threshold CHR h The cache utilization CU(t) at time t, the access delay AL(t) at time t, and the cache hit rate CHR(t) at time t are correlated and calculated to obtain a comprehensive warning score ZHYJ; The comprehensive early warning score ZHYJ is calculated by the following algorithm formula: Wherein, a1, a2 and a3 represent the preset weight values ​​of the cache utilization CU(t) at time t, the access delay AL(t) at time t and the cache hit rate CHR(t) at time t, respectively, and the specific values ​​are set by the user, and a1+a2+a3=1; The intelligent evaluation module is used to preset the cache threshold H and the obtained comprehensive warning score ZHYJ, and perform a first comparative evaluation to identify the cache status of the distributed storage system; The cache optimization module is used to be triggered when an abnormality in the current system cache is identified. By introducing the cache write rate CWR and the cache read rate CRR and the obtained comprehensive warning score ZHYJ, a correlation calculation is performed to obtain the deep analysis index SDFX, and a preset warning threshold Y is performed for secondary evaluation, in-depth analysis of the cache situation of each distributed storage system, and generation of control measures.

2. The cache system of the distributed storage system according to claim 1, characterized in that: The parameter acquisition module includes a data acquisition unit, a data storage unit and a data preprocessing unit; The data collection unit is used to deploy Prometheus, Telegraf and Grafana monitoring tools on each node of each distributed storage system, and collect cache data of each distributed storage system node in real time, wherein the cache data includes cache write rate CWR, cache read rate CRR, cache utilization CU, access delay AL, and cache hit rate CHR; The data storage unit is used to build an InfluxDB time series database, send the collected cache data to the InfluxDB time series database through Telegraf, store the cache data, and monitor the stored cache data through Graphite; The data preprocessing unit is used to detect and remove abnormal data using a statistical method for the cache data stored in the database, and then convert the cache data after taking out the abnormal data, converting the data from the original format to the format required for analysis.

3. The cache system of the distributed storage system according to claim 2, characterized in that: The data analysis and prediction module includes a data analysis unit and a prediction unit; The data analysis unit is used to process, clean and statistically analyze the cache data using Pandas based on the historical cache data, and to create time series graphs, trend graphs, scatter plots, statistical charts, heat maps and regression graphs for the cache data using Matplotlib and Seaborn to identify periodic changes in cache performance; The prediction unit predicts the cache performance of the distributed storage system by integrating and building a machine learning model based on the ARIMA model and the LSTM model; The ARIMA model determines the p, d and q parameters of the model by observing the autocorrelation function ACF and the partial autocorrelation function PACF, and fits the ARIMA model using historical data; The LSTM model converts the time series data into the LSTM input format of a multi-time step input sequence, defines the LSTM network structure and number of layers, and then uses historical cache data for training; The prediction performance of the machine learning model was evaluated using the root mean square error, mean absolute error, and coefficient of determination.

4. The cache system of the distributed storage system according to claim 1, characterized in that: The intelligent evaluation module is used to determine the average value and standard deviation of cache data in each distributed storage system under normal conditions based on the historical cache data of each distributed storage system, perform a first comparative evaluation on the preset cache threshold H and the obtained comprehensive warning score ZHYJ, and analyze the cache situation in each distributed storage system. The specific evaluation results are as follows; When the comprehensive warning score ZHYJ ≥ cache threshold H, it indicates that the cache performance of the current distributed storage system is abnormal. At this time, the first warning information is generated, and the secondary evaluation is performed for in-depth analysis; When the comprehensive warning score ZHYJ is less than the cache threshold H, it indicates that the current distributed storage system performance is normal and no warning needs to be generated.

5. The cache system of the distributed storage system according to claim 1, characterized in that: The cache optimization module includes a depth analysis unit and a depth evaluation unit; The depth analysis unit is used to introduce a cache write rate warning threshold CWR h Cache read rate warning threshold CRR h , and combined with the comprehensive early warning score ZHYJ, perform related calculations to obtain the in-depth analysis index SDFX, and conduct detailed analysis of the performance of the distributed storage system; The depth analysis indicator SDFX is calculated and obtained by the following algorithm formula; Where CWR t represents the cache write rate at time t, CRR t Indicates the cache read rate at time t, and the cache write rate warning threshold CWR h Cache read rate warning threshold CRR h , obtained by averaging the historical data of cache write rate and cache read rate; The depth evaluation unit is used to perform a secondary comparative evaluation based on the safety limit of the historical cache data, the preset warning threshold Y and the acquired depth analysis index SDFX, and deeply analyze the potential performance of the distributed storage system. The specific evaluation scheme is as follows; When the in-depth analysis indicator SDFX> warning threshold Y, it means that the cache operation status of the current distributed storage system is abnormal. At this time, Prometheus and Grafana are used to set up automated monitoring and warning rules. When an abnormality is identified, the second warning information is automatically sent to the administrator. At the same time, the Ansible automation tool is used to write scripts to automatically execute adjustment measures according to the warning situation; When the depth analysis indicator SDFX ≤ the warning threshold Y, it indicates that the cache operation status of the current distributed storage system is normal.

6. A caching method for a distributed storage system, applied to the caching system of the distributed storage system according to any one of claims 1 to 5, characterized in that: The following steps are involved: S1. First, a monitoring tool is deployed on each node of the distributed storage system of the e-commerce platform to collect cache data of each distributed storage node in real time, integrate the collected cache data into the database, and pre-process the collected cache data; S2. Use historical data to analyze the usage patterns and performance change trends of cache data, and then build a machine learning model to predict the future cache status of the distributed storage system; S3, construct a cache utilization prediction formula, an access delay prediction formula and a cache hit rate prediction formula, substitute the obtained cache data into the relevant formulas, calculate and obtain the cache utilization CU(t) at time t, the access delay AL(t) at time t and the cache hit rate CHR(t) at time t, and perform related calculations to obtain the comprehensive warning score ZHYJ; S4, the preset cache threshold H and the obtained comprehensive warning score ZHYJ are compared and evaluated for the first time to identify the cache status of the distributed storage system; S5. Finally, by introducing the cache write rate CWR and the cache read rate CRR and the obtained comprehensive warning score ZHYJ, a correlation calculation is performed to obtain the deep analysis index SDFX, and the warning threshold Y is preset for secondary evaluation, the cache situation of each distributed storage system is deeply analyzed, and control measures are generated.

Images