Monitoring index prediction method and device, computer equipment and readable storage medium

Abstract

The invention relates to a monitoring index prediction method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: collecting current monitoring data of the distributed storage system; performing position coding processing on the current monitoring data to obtain processed monitoring data; inputting the processed monitoring data into a first monitoring index prediction model in pre-trained monitoring index prediction models to obtain a first monitoring index prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring index prediction model in the pre-trained monitoring index prediction models to obtain a second monitoring index prediction result of the distributed storage system, obtaining a second monitoring index prediction result of the distributed storage system; and obtaining a target monitoring index prediction result of the distributed storage system according to the first monitoring index prediction result and the second monitoring index prediction result. By adopting the method, the monitoring index prediction accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a monitoring indicator prediction method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] With the continuous development of cloud computing technology, applications and services in cloud environments have experienced explosive growth; against the backdrop of cloud computing and digital transformation, distributed storage systems have become the preferred solution for storing and processing massive amounts of data.

[0003] In traditional technology, when predicting the monitoring indicators of distributed storage systems, predictions are generally made through a single model, such as using a long short-term memory model to predict the future monitoring indicators of distributed storage systems; however, when processing long time series, the long short-term memory model is prone to a decrease in prediction accuracy, resulting in a low prediction accuracy of the monitoring indicators. Summary of the invention

[0004] Based on this, it is necessary to provide a monitoring indicator prediction method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of monitoring indicator prediction in response to the above technical problems.

[0005] In a first aspect, the present application provides a monitoring indicator prediction method, comprising:

[0006] Collect current monitoring data of distributed storage systems;

[0007] Performing position encoding processing on the current monitoring data to obtain processed monitoring data;

[0008] Inputting the processed monitoring data into a first monitoring indicator prediction model in a pre-trained monitoring indicator prediction model to obtain a first monitoring indicator prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the distributed storage system;

[0009] A target monitoring indicator prediction result of the distributed storage system is obtained according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

[0010] In one embodiment, collecting current monitoring data of the distributed storage system includes:

[0011] Collecting system performance indicators of the distributed storage system and basic monitoring indicators of the server cluster associated with the distributed storage system;

[0012] According to the system performance index of the distributed storage system and the basic monitoring index of the server cluster, the current monitoring data of the distributed storage system is obtained.

[0013] In one embodiment, performing position encoding processing on the current monitoring data to obtain processed monitoring data includes:

[0014] Performing dynamic position coding processing on the current monitoring data to obtain position information of the current monitoring data;

[0015] The current monitoring data and the location information are spliced ​​together to obtain the processed monitoring data.

[0016] In one embodiment, obtaining the target monitoring indicator prediction result of the distributed storage system according to the first monitoring indicator prediction result and the second monitoring indicator prediction result includes:

[0017] Obtaining a first weight corresponding to the prediction result of the first monitoring indicator, and obtaining a second weight corresponding to the prediction result of the second monitoring indicator;

[0018] According to the first weight and the second weight, the first monitoring indicator prediction result and the second monitoring indicator prediction result are fused to obtain the target monitoring indicator prediction result of the distributed storage system.

[0019] In one embodiment, the pre-trained monitoring indicator prediction model is trained in the following manner:

[0020] Collecting sample monitoring data of a sample distributed storage system; the sample monitoring data includes system performance indicators of the sample distributed storage system and basic monitoring indicators of a sample server cluster associated with the sample distributed storage system;

[0021] Performing position coding processing on the sample monitoring data to obtain processed monitoring data corresponding to the sample monitoring data;

[0022] The sample monitoring data and the processed monitoring data corresponding to the sample monitoring data are predicted by the monitoring indicator prediction model to be trained to obtain the target monitoring indicator prediction result of the sample distributed storage system;

[0023] The actual results of the target monitoring indicators of the sample distributed storage system are obtained, and according to the difference between the predicted results of the target monitoring indicators of the sample distributed storage system and the actual results of the target monitoring indicators, the monitoring indicator prediction model to be trained is iteratively trained to obtain the pre-trained monitoring indicator prediction model.

[0024] In one embodiment, the monitoring indicator prediction model to be trained is used to perform prediction processing on the sample monitoring data and the processed monitoring data corresponding to the sample monitoring data to obtain the target monitoring indicator prediction result of the sample distributed storage system, including:

[0025] Inputting the processed monitoring data corresponding to the sample monitoring data into a first monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain a first monitoring indicator prediction result of the sample distributed storage system, and inputting the sample monitoring data into a second monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain a second monitoring indicator prediction result of the sample distributed storage system;

[0026] The first monitoring indicator prediction result and the second monitoring indicator prediction result of the sample distributed storage system are fused to obtain the target monitoring indicator prediction result of the sample distributed storage system.

[0027] In a second aspect, the present application also provides a monitoring indicator prediction device, comprising:

[0028] A data collection module is used to collect current monitoring data of the distributed storage system;

[0029] The coding processing module is used to perform position coding processing on the current monitoring data to obtain processed monitoring data;

[0030] An indicator prediction module, used for inputting the processed monitoring data into a first monitoring indicator prediction model in a pre-trained monitoring indicator prediction model to obtain a first monitoring indicator prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the distributed storage system;

[0031] A result determination module is used to obtain a target monitoring indicator prediction result of the distributed storage system according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Collect current monitoring data of distributed storage systems;

[0034] Performing position encoding processing on the current monitoring data to obtain processed monitoring data;

[0035] Inputting the processed monitoring data into a first monitoring indicator prediction model in a pre-trained monitoring indicator prediction model to obtain a first monitoring indicator prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the distributed storage system;

[0036] A target monitoring indicator prediction result of the distributed storage system is obtained according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Collect current monitoring data of distributed storage systems;

[0039] Performing position encoding processing on the current monitoring data to obtain processed monitoring data;

[0040] Inputting the processed monitoring data into a first monitoring indicator prediction model in a pre-trained monitoring indicator prediction model to obtain a first monitoring indicator prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the distributed storage system;

[0041] A target monitoring indicator prediction result of the distributed storage system is obtained according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Collect current monitoring data of distributed storage systems;

[0044] Performing position encoding processing on the current monitoring data to obtain processed monitoring data;

[0045] Inputting the processed monitoring data into a first monitoring indicator prediction model in a pre-trained monitoring indicator prediction model to obtain a first monitoring indicator prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the distributed storage system;

[0046] A target monitoring indicator prediction result of the distributed storage system is obtained according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

[0047] The above-mentioned monitoring indicator method, device, computer equipment, computer-readable storage medium and computer program product first collect the current monitoring data of the distributed storage system, and perform position encoding processing on the current monitoring data to obtain processed monitoring data, and then input the processed monitoring data into the first monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the first monitoring indicator prediction result of the distributed storage system, and input the current monitoring data into the second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the second monitoring indicator prediction result of the distributed storage system, and finally obtain the target monitoring indicator prediction result of the distributed storage system based on the first monitoring indicator prediction result and the second monitoring indicator prediction result. In this way, when predicting the monitoring indicators of the distributed storage system, comprehensive consideration of the first monitoring indicator prediction result output by the first monitoring indicator prediction model and the second monitoring indicator prediction result output by the second monitoring indicator prediction model can make the final target monitoring indicator prediction result more accurate, thereby effectively making up for the limitations of a single model in prediction accuracy, thereby improving the prediction accuracy of the monitoring indicators, and at the same time avoiding the defect that the long short-term memory model is prone to a decrease in prediction accuracy when processing long time series, resulting in a low monitoring indicator prediction accuracy; in addition, when outputting the first monitoring indicator prediction result, position encoding processing is performed on the current monitoring data, which is conducive to comprehensively considering the position information of the current monitoring data, thereby making the output first monitoring indicator prediction result more accurate, further improving the monitoring indicator prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a flow chart of a monitoring indicator prediction method in one embodiment;

[0050] Figure 2 A flowchart of the training steps of a pre-trained monitoring indicator prediction model in one embodiment;

[0051] Figure 3 A schematic diagram of a flow chart of a monitoring indicator prediction method in another embodiment;

[0052] Figure 4 A schematic diagram of the structure of a Transformer model in one embodiment;

[0053] Figure 5 A schematic diagram of the structure of a monitoring indicator prediction model in an embodiment;

[0054] Figure 6 A method for predicting monitoring indicators of a distributed storage system in an embodiment

[0055] Figure 7 It is a structural block diagram of a monitoring index prediction device in one embodiment;

[0056] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0059] With the continuous development of cloud computing technology, applications and services in cloud environments have exploded. In the context of cloud computing and digital transformation, distributed storage systems have become the preferred solution for storing and processing massive amounts of data. With the advantages of high scalability, high availability, and low cost, they provide users with data storage and access services. However, as the scale of distributed storage continues to expand, the complexity of the system increases, posing challenges to the performance monitoring and prediction capabilities of distributed storage systems. As a key factor in ensuring the stable operation of system monitoring, how to accurately and effectively predict future monitoring data in order to identify and capture potential anomalies in advance, implement a monitoring early warning mechanism, and thus reduce the frequency of system failures has become a challenge that needs to be urgently addressed in the industry.

[0060] Traditionally, distributed storage system monitoring mainly uses a post-event alarm mechanism, relying on expert experience to pre-set alarm thresholds. Alarm conditions are triggered only when system performance degrades or failures occur. This monitoring method lacks in-depth mining and analysis of massive monitoring data, resulting in alarm lags and unpredictable potential risks, which also causes inconvenience to real-time analysis and operation and maintenance.

[0061] In recent years, with the development of artificial intelligence technology, the concept of intelligent monitoring has emerged. By analyzing historical monitoring data and using deep learning and machine learning algorithms, it is possible to predict anomalies that may occur in the future, reduce the probability of accidents and reduce system maintenance costs. In the field of distributed storage, how to effectively collect and analyze monitoring indicator data to achieve more accurate data prediction has become a hot research issue. RNN (Recurrent Neural Network) predicts data in a recursive way, which is difficult to perform parallel calculations, limiting the speed of training, and it is prone to the problem of gradient disappearance, which also limits the ability of this type of model to analyze large-scale data. When predicting data, CNN (Convolutional Neural Network) is limited to the input of fixed length of the sequence, and its fixed receptive field makes it impossible to capture the dependency of the entire sequence. LSTM (Long Short-Term Memory) innovatively proposes a gating mechanism to overcome the gradient disappearance problem of RNN, but the training process is computationally intensive and requires a large amount of computing resources. When processing long time series, it still has problems of low computing efficiency and reduced accuracy. GRU (Gated Recurrent Unit) simplifies the structure of LSTM and has fewer parameters than LSTM. However, when there is not enough data and regularization processing, it is prone to overfitting problems, resulting in a decrease in generalization ability.

[0062] In summary, traditional methods are highly dependent on expert experience and have the problem of delayed alarms, and cannot effectively utilize massive monitoring data. The RNN series of deep learning prediction methods have problems such as gradient vanishing, consumption of a large amount of computing resources, and decreased accuracy in predicting long time series. Based on this, this application proposes a new monitoring indicator prediction method that can improve the accuracy of monitoring indicator prediction.

[0063] In order to more clearly explain the monitoring indicator prediction method provided in the embodiment of the present application, some terms are explained as follows:

[0064] Self-Attention Mechanism: In the field of deep learning, as an important component of the Transformer model, it allows the model to weight the importance of different time points in the sequence when processing time series data. Compared with traditional methods, it has the ability to capture long-distance dependencies and parallel computing. By combining into a multi-head attention mechanism, it can learn features from different subspaces, enhancing the efficiency and performance of the Transformer model in processing time series.

[0065] Autoregressive Network (AR Network): In applications involving complex time series data, the data dynamics may show both predictable linear trends and nonlinear behaviors that are more difficult to predict. In such scenarios, the autoregressive model, as a linear prediction model, is combined with a nonlinear prediction model to obtain a comprehensive model, which enables the model to provide more accurate predictions under a wider range of conditions.

[0066] Figure 1 is a flowchart of a monitoring indicator prediction method according to an exemplary embodiment. Figure 1 As shown, the monitoring indicator prediction method is applied to a server deployed with a distributed storage monitoring system; it is understandable that the method can also be applied to a terminal deployed with a distributed storage monitoring system, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones and tablets; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this exemplary embodiment, the method includes the following steps S101 to S104. Among them:

[0067] Step S101, collecting current monitoring data of the distributed storage system.

[0068] Among them, the current monitoring data includes the system performance indicators of the distributed storage system and the basic monitoring indicators of the server cluster associated with the distributed storage system, including multi-dimensional monitoring indicators, such as disk I / O (Input / Output), CPU (Central Processing Unit) utilization, network bandwidth, memory usage, upload and download rates, etc. In actual scenarios, the current monitoring data is a time series data.

[0069] Among them, a distributed storage system is a data storage architecture that stores data in multiple physical nodes to achieve data redundancy, load balancing and high availability; these physical nodes can be independent servers or members of a server cluster. The server cluster associated with a distributed storage system refers to the server cluster corresponding to the distributed data storage nodes of the distributed storage system.

[0070] Exemplarily, the server first determines the server cluster associated with the distributed storage system, and then collects the current system performance indicators of the distributed storage system and the current basic monitoring indicators of the server cluster associated with the distributed storage system as the current monitoring data of the distributed storage system.

[0071] For example, the server collects and integrates multiple monitoring indicators of server clusters and distributed storage systems, such as disk I / O, CPU utilization, network bandwidth, memory usage, upload and download rates, etc.; these monitoring indicators not only cover the basic monitoring indicators of server clusters, but also take into account the system performance indicators of distributed storage systems (such as storage performance indicators), forming a comprehensive system status view; by real-time monitoring and analysis of these monitoring indicators, it is possible to quickly locate problems, optimize system configuration, and prevent potential failures, thereby ensuring the efficient and stable operation of distributed storage services and improving user experience and service quality.

[0072] Step S102, performing position coding processing on the current monitoring data to obtain processed monitoring data.

[0073] The position coding process refers to a dynamic position coding process, specifically a dynamic position coding method based on sequence length.

[0074] It should be noted that, for monitoring sequence analysis, this application proposes a dynamic position encoding method based on sequence length, which can automatically adjust the encoding method according to the input length of the data (such as the current monitoring data), thereby improving the model's adaptability and generalization ability to sequences of different lengths, which is conducive to meeting complex data processing requirements.

[0075] The processed monitoring data includes the current monitoring data and the location information of the current monitoring data.

[0076] Exemplarily, the server performs position encoding processing on the current monitoring data by a dynamic position encoding method based on sequence length to embed position information in the current monitoring data, thereby obtaining processed monitoring data including the current monitoring data and the position information of the current monitoring data.

[0077] Step S103, input the processed monitoring data into the first monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the first monitoring indicator prediction result of the distributed storage system, and input the current monitoring data into the second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the second monitoring indicator prediction result of the distributed storage system.

[0078] The pre-trained monitoring indicator prediction model refers to the distributed storage system monitoring indicator prediction model, which is used to output the target monitoring indicator prediction results of the distributed storage system. It specifically includes a first monitoring indicator prediction model and a second monitoring indicator prediction model. The structure of the model can be referred to Figure 5 .

[0079] Among them, the first monitoring indicator prediction model refers to the Transformer model, which is used to output the first monitoring indicator prediction results of the distributed storage system in the future time period (such as the next few days, the next week, the next month, etc.) based on the processed monitoring data, such as the file system capacity and remaining capacity, disk read and write data rate, the number of combined read / write completions per second of the disk partition, the disk available capacity corresponding to the user data directory, the disk IO operation speed in the process, the read cache update data size, the amount of data written and dropped to disk, the amount of data to be uploaded to the cloud, etc. Its structure can be referred to Figure 4 .

[0080] Among them, the second monitoring indicator prediction model refers to the AR model (autoregressive network model), which is used to output the prediction results of the second monitoring indicator of the distributed storage system in the future time period based on the current monitoring data, such as file system capacity and remaining capacity, disk read and write data rate, the number of combined read / write completions per second of the disk partition, the available disk capacity corresponding to the user data directory, the disk IO operation speed in process, the read cache update data size, the amount of data written and dropped to disk, the amount of data to be uploaded to the cloud, etc.

[0081] It should be noted that single-step prediction can only predict data at one time node and cannot understand the changing trend of monitoring indicators, while multi-step prediction needs to solve the problems of error accumulation and prediction performance degradation. The Transformer model adopts a multi-head self-attention mechanism, which can effectively capture the long-term dependencies of distributed storage system monitoring data. The autoregressive network model is good at capturing linear relationships and can maintain good prediction accuracy for monitoring data of long columns. By taking advantage of the two models, the dynamic trend of indicators can be effectively captured, while reducing the error accumulation of multi-step predictions and effectively improving the prediction accuracy of long-distance monitoring indicators. The Transformer algorithm in deep learning is used to learn the complex associations between monitoring indicators, and the AR model is used to predict the linear trends in the monitoring sequence. The output results of the two models are weighted and fused to obtain the final prediction results of the model combining linear and nonlinear prediction algorithms, and the final distributed storage system monitoring indicator prediction model is constructed, which improves the robustness and accuracy of the model.

[0082] Exemplarily, the server inputs the processed monitoring data into the first monitoring indicator prediction model in the pre-trained monitoring indicator prediction model, performs feature extraction processing on the processed monitoring data through the first monitoring indicator prediction model to obtain a feature vector, performs a series of prediction processing on the feature vector, and obtains a first monitoring indicator prediction result of the distributed storage system; at the same time, the server inputs the current monitoring data into the second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model, performs prediction processing on the current monitoring data through the second monitoring indicator prediction model, and obtains a second monitoring indicator prediction result of the distributed storage system.

[0083] Step S104, obtaining a target monitoring indicator prediction result of the distributed storage system according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

[0084] Among them, the prediction result of the target monitoring indicator is obtained by weighted fusion of the prediction result of the first monitoring indicator and the prediction result of the second monitoring indicator, specifically including the file system capacity and remaining capacity, the disk read and write data rate, the number of combined read / write completions per second of the disk partition, the available disk capacity corresponding to the user data directory, the disk IO operation speed in process, the read cache update data size, the amount of data written and dropped to disk, the amount of data to be uploaded to the cloud, etc.

[0085] Exemplarily, the server performs weighted fusion processing on the first monitoring indicator prediction result and the second monitoring indicator prediction result to obtain the fused monitoring indicator prediction result as the target monitoring indicator prediction result of the distributed storage system. Furthermore, the server can also determine in advance the possible problems and failures of the distributed storage system in the future based on the target monitoring indicator prediction result of the distributed storage system, and perform maintenance or configuration optimization on the distributed storage system in advance.

[0086] For example, the server collects historical monitoring data of the distributed storage system, and after the dynamic position encoding based on sequence length proposed in this application, it is input into the Transformer model to obtain prediction result 1. At the same time, the AR model is used to predict each monitoring indicator to obtain prediction result 2. After the prediction results of the two are merged, the final prediction result is obtained. Through iterative training and model optimization, a trained model is obtained and deployed in the production environment. The future data of the distributed storage system monitoring indicators can be predicted through real-time monitoring data. In this way, the monitoring performance and prediction accuracy of the distributed storage system are improved, and a more accurate prediction of the future state of the system cluster is made, including resource usage trends, performance fluctuations and even potential failure points. This application effectively utilizes massive monitoring data to identify potential hardware failures and system bottlenecks in advance, help users better plan and adjust resources, arrange maintenance windows to avoid unplanned downtime, and promote efficient allocation of resources. This application helps operation and maintenance personnel to detect anomalies in a timely manner, provide early warnings for operation and maintenance and security teams, so as to detect security threats in a timely manner, reduce the risk of failures, and ensure the efficient operation of the system. Ultimately, this application can reduce service interruptions, improve business continuity, speed up data access and processing efficiency, optimize resource utilization to reduce unnecessary expenses, flexibly adapt to changes in customer needs, enhance user trust in distributed storage services, meet customer needs and improve user experience.

[0087] In the above-mentioned monitoring indicator prediction method, the current monitoring data of the distributed storage system is first collected, and the current monitoring data is position-encoded to obtain processed monitoring data, and then the processed monitoring data is input into the first monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the first monitoring indicator prediction result of the distributed storage system, and the current monitoring data is input into the second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the second monitoring indicator prediction result of the distributed storage system, and finally, based on the first monitoring indicator prediction result and the second monitoring indicator prediction result, the target monitoring indicator prediction result of the distributed storage system is obtained. In this way, when predicting the monitoring indicators of the distributed storage system, comprehensive consideration of the first monitoring indicator prediction result output by the first monitoring indicator prediction model and the second monitoring indicator prediction result output by the second monitoring indicator prediction model can make the final target monitoring indicator prediction result more accurate, thereby effectively making up for the limitations of a single model in prediction accuracy, thereby improving the prediction accuracy of the monitoring indicators, and at the same time avoiding the defect that the long short-term memory model is prone to a decrease in prediction accuracy when processing long time series, resulting in a low monitoring indicator prediction accuracy; in addition, when outputting the first monitoring indicator prediction result, position encoding processing is performed on the current monitoring data, which is conducive to comprehensively considering the position information of the current monitoring data, thereby making the output first monitoring indicator prediction result more accurate, further improving the monitoring indicator prediction accuracy.

[0088] In an exemplary embodiment, the above step S101, collecting the current monitoring data of the distributed storage system, specifically includes the following contents: collecting system performance indicators of the distributed storage system, and basic monitoring indicators of the server cluster associated with the distributed storage system; according to the system performance indicators of the distributed storage system and the basic monitoring indicators of the server cluster, obtaining the current monitoring data of the distributed storage system.

[0089] Among them, the basic monitoring indicators of the server cluster associated with the distributed storage system include CPU load conditions, including total utilization rate, user state time, IO waiting time, system time, etc.; memory status, involving total memory, used memory and available memory, etc.; network card upload and download traffic, total number of error packets sent per second by each interface and interface, etc.; TCP (Transmission Control Protocol) connection status, including number of packets and retransmission rate, etc.; as well as statistical information such as disk available space, system load and process.

[0090] Among them, the system performance indicators of the distributed storage system specifically include file system capacity and remaining capacity, disk read and write data rate, the number of combined read / write completions per second of the disk partition, the available disk capacity corresponding to the user data directory, the disk IO operation speed in process, the read cache update data size, the amount of data written and stored on disk, the amount of data to be uploaded to the cloud, and other indicators reflecting system performance.

[0091] It should be noted that, first, the current monitoring data covers the basic monitoring indicators of the server cluster; second, in view of the characteristics of distributed storage, it also pays attention to system performance indicators; these indicators together constitute a comprehensive monitoring system to ensure the efficient and stable operation of distributed storage services.

[0092] Exemplarily, the server collects the current system performance indicators of the distributed storage system and the current basic monitoring indicators of the server cluster associated with the distributed storage system; then, the current system performance indicators of the distributed storage system and the current basic monitoring indicators of the server cluster are combined to obtain multi-dimensional monitoring indicators of the distributed storage system as the current monitoring data of the distributed storage system.

[0093] In this embodiment, the current monitoring data of the distributed storage system is obtained based on the system performance indicators of the distributed storage system and the basic monitoring indicators of the server cluster, which is conducive to comprehensive consideration of the system performance indicators of the distributed storage system and the basic monitoring indicators of the server cluster when predicting future monitoring indicators, thereby improving the accuracy of monitoring indicator prediction of the distributed storage system.

[0094] In an exemplary embodiment, the above step S102, performs position coding processing on the current monitoring data to obtain processed monitoring data, specifically includes the following contents: performs dynamic position coding processing on the current monitoring data to obtain the position information of the current monitoring data; splices the current monitoring data and the position information to obtain the processed monitoring data.

[0095] Among them, the current monitoring data is a time series data.

[0096] The location information of the current monitoring data is used to indicate the location information of the elements in the current monitoring data.

[0097] Exemplarily, the server identifies the sequence length of the current monitoring data, and performs dynamic position encoding processing on the current monitoring data according to the sequence length of the current monitoring data to obtain the location information of the current monitoring data; the current monitoring data and the location information of the current monitoring data are concatenated to obtain data containing the current monitoring data and the location information of the current monitoring data as the processed monitoring data.

[0098] In this embodiment, the current monitoring data is dynamically position encoded to obtain the location information of the current monitoring data, and the current monitoring data and the location information of the current monitoring data are spliced ​​to obtain processed monitoring data, which is beneficial to improving the accuracy of the subsequent first monitoring indicator prediction results obtained based on the processed monitoring data, thereby improving the accuracy of the monitoring indicator prediction.

[0099] In an exemplary embodiment, the above step S104 obtains the target monitoring indicator prediction result of the distributed storage system based on the first monitoring indicator prediction result and the second monitoring indicator prediction result, and specifically includes the following contents: obtaining the first weight corresponding to the first monitoring indicator prediction result, and obtaining the second weight corresponding to the second monitoring indicator prediction result; according to the first weight and the second weight, the first monitoring indicator prediction result and the second monitoring indicator prediction result are integrated to obtain the target monitoring indicator prediction result of the distributed storage system.

[0100] The first weight is used to indicate the importance of the prediction result of the first monitoring indicator.

[0101] The second weight is used to indicate the importance of the prediction result of the second monitoring indicator.

[0102] Exemplarily, the server obtains the weight corresponding to the first monitoring indicator prediction model as the first weight corresponding to the first monitoring indicator prediction result, and obtains the weight corresponding to the second monitoring indicator prediction model as the second weight corresponding to the second monitoring indicator prediction result; then, according to the first weight and the second weight, the first monitoring indicator prediction result and the second monitoring indicator prediction result are merged to obtain the target monitoring indicator prediction result of the distributed storage system, that is, the target monitoring indicator prediction result = first weight × first monitoring indicator prediction result + second weight × second monitoring indicator prediction result.

[0103] In this embodiment, the first monitoring indicator prediction result and the second monitoring indicator prediction result are fused according to the first weight and the second weight to obtain the target monitoring indicator prediction result of the distributed storage system; in this way, comprehensive consideration of the first monitoring indicator prediction result and the second monitoring indicator prediction result is conducive to improving the accuracy of monitoring indicator prediction.

[0104] In an exemplary embodiment, Figure 2 As shown, the monitoring indicator prediction method provided by the present application also includes a training step of a pre-trained monitoring indicator prediction model, specifically including the following steps S201 to S204. Among them:

[0105] Step S201, collecting sample monitoring data of a sample distributed storage system; the sample monitoring data includes system performance indicators of the sample distributed storage system and basic monitoring indicators of a sample server cluster associated with the sample distributed storage system.

[0106] Step S202: perform position coding processing on the sample monitoring data to obtain processed monitoring data corresponding to the sample monitoring data.

[0107] Step S203, using the monitoring indicator prediction model to be trained, predicting the sample monitoring data and the processed monitoring data corresponding to the sample monitoring data, and obtaining the target monitoring indicator prediction result of the sample distributed storage system.

[0108] Step S204, obtaining the actual results of the target monitoring indicators of the sample distributed storage system, and iteratively training the monitoring indicator prediction model to be trained according to the difference between the target monitoring indicator prediction results of the sample distributed storage system and the actual results of the target monitoring indicators to obtain a pre-trained monitoring indicator prediction model.

[0109] The processed monitoring data corresponding to the sample monitoring data includes the sample monitoring data and location information of the sample monitoring data.

[0110] Among them, the monitoring indicator prediction model to be trained includes a first monitoring indicator prediction model to be trained (such as a Transformer model) and a second monitoring indicator prediction model to be trained (such as an AR model).

[0111] Exemplarily, the server collects system performance indicators of the sample distributed storage system and basic monitoring indicators of the sample server cluster associated with the sample distributed storage system, combines the system performance indicators of the sample distributed storage system and the basic monitoring indicators of the sample server cluster associated with the sample distributed storage system, and obtains sample monitoring data of the sample distributed storage system; then, position encoding is performed on the sample monitoring data through a dynamic position encoding method based on sequence length to embed position information in the sample monitoring data, thereby obtaining processed monitoring data containing the sample monitoring data and the position information of the sample monitoring data, as the processed monitoring data corresponding to the sample monitoring data; then, the sample monitoring data and the processed monitoring data corresponding to the sample monitoring data are predicted by the monitoring indicator prediction model to be trained. Perform prediction processing to obtain the target monitoring indicator prediction results of the sample distributed storage system; finally, obtain the actual results of the target monitoring indicators of the sample distributed storage system, and calculate the loss value, such as the cross entropy loss value, based on the difference between the target monitoring indicator prediction results of the sample distributed storage system and the actual results of the target monitoring indicators, and use the loss value to adjust the model parameters of the monitoring indicator prediction model to be trained to obtain the monitoring indicator prediction model after parameter adjustment as the trained monitoring indicator prediction model, and continue to repeat the above training process until the loss value obtained from the target monitoring indicator prediction results of the sample distributed storage system output by the trained monitoring indicator prediction model is less than the preset threshold, then stop model training, and use the trained monitoring indicator prediction model as the pre-trained monitoring indicator prediction model.

[0112] In this embodiment, the monitoring indicator prediction model to be trained is iteratively trained using sample monitoring data from a sample distributed storage system to obtain a pre-trained monitoring indicator prediction model, which can make the subsequent target monitoring indicator prediction results obtained through the pre-trained monitoring indicator prediction model more accurate, thereby improving the monitoring indicator prediction accuracy of the distributed storage system.

[0113] In an exemplary embodiment, the above step S203, through the monitoring indicator prediction model to be trained, performs prediction processing on the sample monitoring data and the processed monitoring data corresponding to the sample monitoring data to obtain the target monitoring indicator prediction result of the sample distributed storage system, specifically including the following contents: inputting the processed monitoring data corresponding to the sample monitoring data into the first monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain the first monitoring indicator prediction result of the sample distributed storage system, and inputting the sample monitoring data into the second monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain the second monitoring indicator prediction result of the sample distributed storage system; fusing the first monitoring indicator prediction result and the second monitoring indicator prediction result of the sample distributed storage system to obtain the target monitoring indicator prediction result of the sample distributed storage system.

[0114] Exemplarily, the server inputs the processed monitoring data corresponding to the sample monitoring data into the first monitoring indicator prediction model to be trained in the monitoring indicator prediction model to be trained, performs feature extraction processing on the processed monitoring data corresponding to the sample monitoring data through the first monitoring indicator prediction model to obtain a sample feature vector, performs a series of prediction processing on the sample feature vector, and obtains a first monitoring indicator prediction result of the sample distributed storage system; at the same time, the sample monitoring data is input into the second monitoring indicator prediction model to be trained in the monitoring indicator prediction model to be trained, and performs prediction processing on the sample monitoring data through the second monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the sample distributed storage system; finally, according to the current first weight of the first monitoring indicator prediction model and the current second weight of the second monitoring indicator prediction model, the first monitoring indicator prediction result and the second monitoring indicator prediction result of the sample distributed storage system are fused to obtain the target monitoring indicator prediction result of the sample distributed storage system, that is, the target monitoring indicator prediction result of the sample distributed storage system = the current first weight × the first monitoring indicator prediction result of the sample distributed storage system + the current second weight × the second monitoring indicator prediction result of the sample distributed storage system.

[0115] In this embodiment, by training a monitoring indicator prediction model that includes a first monitoring indicator prediction model and a second monitoring indicator prediction model, the target monitoring indicator prediction results subsequently output by the monitoring indicator prediction model can be made more accurate, effectively compensating for the limitations of a single model in prediction accuracy, thereby improving the monitoring indicator prediction accuracy.

[0116] Input the processed monitoring data corresponding to the sample monitoring data into the first monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain the first monitoring indicator prediction result of the sample distributed storage system, and input the sample monitoring data into the second monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain the second monitoring indicator prediction result of the sample distributed storage system; fuse the first monitoring indicator prediction result and the second monitoring indicator prediction result of the sample distributed storage system to obtain the target monitoring indicator prediction result of the sample distributed storage system

[0117] In an exemplary embodiment, Figure 3 As shown, another monitoring indicator prediction method is provided, which is described by taking the method applied to a server as an example, and includes the following steps S301 to S312. Among them:

[0118] Step S301, collecting sample monitoring data of a sample distributed storage system; the sample monitoring data includes system performance indicators of the sample distributed storage system and basic monitoring indicators of a sample server cluster associated with the sample distributed storage system.

[0119] Step S302: perform position coding processing on the sample monitoring data to obtain processed monitoring data corresponding to the sample monitoring data.

[0120] Step S303, input the processed monitoring data corresponding to the sample monitoring data into the first monitoring indicator prediction model in the monitoring indicator prediction model to be trained, and obtain the first monitoring indicator prediction result of the sample distributed storage system, and input the sample monitoring data into the second monitoring indicator prediction model in the monitoring indicator prediction model to be trained, and obtain the second monitoring indicator prediction result of the sample distributed storage system.

[0121] Step S304: fusing the first monitoring indicator prediction result and the second monitoring indicator prediction result of the sample distributed storage system to obtain a target monitoring indicator prediction result of the sample distributed storage system.

[0122] Step S305, obtaining the actual results of the target monitoring indicators of the sample distributed storage system, and iteratively training the monitoring indicator prediction model to be trained according to the difference between the target monitoring indicator prediction results of the sample distributed storage system and the actual results of the target monitoring indicators to obtain a pre-trained monitoring indicator prediction model.

[0123] Step S306: collecting system performance indicators of the distributed storage system and basic monitoring indicators of the server cluster associated with the distributed storage system.

[0124] Step S307: obtaining current monitoring data of the distributed storage system according to the system performance indicators of the distributed storage system and the basic monitoring indicators of the server cluster.

[0125] Step S308, performing dynamic position coding processing on the current monitoring data to obtain the position information of the current monitoring data.

[0126] Step S309, concatenating the current monitoring data and the location information to obtain processed monitoring data.

[0127] Step S310, input the processed monitoring data into the first monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the first monitoring indicator prediction result of the distributed storage system, and input the current monitoring data into the second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the second monitoring indicator prediction result of the distributed storage system.

[0128] Step S311, obtaining a first weight corresponding to a first monitoring indicator prediction result, and obtaining a second weight corresponding to a second monitoring indicator prediction result.

[0129] Step S312: According to the first weight and the second weight, the first monitoring indicator prediction result and the second monitoring indicator prediction result are merged to obtain the target monitoring indicator prediction result of the distributed storage system.

[0130] In the above-mentioned monitoring indicator prediction method, when predicting the monitoring indicators of the distributed storage system, the first monitoring indicator prediction result output by the first monitoring indicator prediction model and the second monitoring indicator prediction result output by the second monitoring indicator prediction model are comprehensively considered, so that the final target monitoring indicator prediction result can be made more accurate, thereby effectively making up for the limitations of a single model in prediction accuracy, thereby improving the prediction accuracy of the monitoring indicator, and at the same time avoiding the defect that the long short-term memory model is prone to a decrease in prediction accuracy when processing long time series, resulting in a low monitoring indicator prediction accuracy; in addition, when outputting the first monitoring indicator prediction result, the current monitoring data is positionally encoded, which is conducive to comprehensively considering the position information of the current monitoring data, thereby making the output first monitoring indicator prediction result more accurate, further improving the monitoring indicator prediction accuracy.

[0131] In order to more clearly illustrate the monitoring indicator prediction method provided by the embodiment of the present application, the monitoring indicator prediction method is specifically described below with a specific embodiment. In an exemplary embodiment, the present application also proposes a distributed storage system monitoring indicator prediction method, such as Figure 6As shown in the figure, by collecting multi-dimensional monitoring indicators such as server cluster and distributed storage system performance indicators, pre-processing the data based on the dynamic position coding method, using the Transformer deep learning algorithm and autoregressive network data analysis technology for prediction, and weighted fusion of the two prediction results, the real-time monitoring and intelligent prediction of the distributed storage system monitoring indicators are completed to achieve accurate prediction of system performance and early warning of potential failures, thereby improving the stability and reliability of the distributed storage system, reducing maintenance costs, and improving user experience. The specific process steps are as follows:

[0132] 1.1 Monitoring data collection

[0133] Monitoring data collection includes two parts: defining monitoring indicators and data collection.

[0134] 1.1.1. Define monitoring indicators

[0135] This application defines a series of key performance indicators, which aim to comprehensively capture the operating status of the distributed storage system. First, the monitoring indicators cover the basic monitoring of the server; second, for the characteristics of distributed storage, the system performance indicators are also focused on. These indicators together constitute a comprehensive monitoring system to ensure the efficient and stable operation of the distributed storage service.

[0136] 1.1.2 Data Collection

[0137] Distributed storage software is used to collect data such as Linux system and user reading and writing to the cloud, and key performance indicators are obtained from them to obtain raw data. The distributed storage software pushes data to the monitoring system every 20 seconds, uses 30 days of server cluster monitoring data as raw data, and divides the training set, validation set, and test set in a ratio of 8:1:1. The prediction model is trained on the server cluster monitoring data of the distributed storage system. After training, the model parameters are saved and predictions are made using the current real monitoring data.

[0138] 1.2 Data Preprocessing

[0139] There is a lot of monitoring data, and the input data is a 2D structure of M×N, where M is the number of time points for each indicator training (such as T1, T2, T3, T4, etc.), with a total of N columns (N is the number of types of monitored indicators). Monitoring indicators are essentially time series. Since the self-attention mechanism in the Transformer does not consider the order of each sequence, position encoding is required to input the position information of the sequence into the Transformer so that the Transformer understands the temporal nature of the indicator sequence. The common position encoding is the sine / cosine function position encoding, and its formula is as follows:

[0140]

[0141] Among them, pos is the position of the element in the sequence, i is the dimension index of the embedding vector, and d m Embedding dimension for the model.

[0142] As a static position coding, for sequences of different lengths, the periodicity of this type of coding cannot be changed. When the input sequence length does not match the coding design, it will lead to information loss or redundancy, and cannot adapt to monitoring sequence data of different lengths. In order to better dynamically adjust the representation of position information according to the characteristics of the monitoring sequence, this application proposes a dynamic position coding for the monitoring sequence, using the sequence length L as the position coding adjustment factor, the formula is as follows:

[0143]

[0144] When the monitoring sequence data length is short, the position encoding frequency will increase, otherwise it will decrease, so that the encoding can adapt to monitoring sequence data of different lengths, more accurately reflect the relative position information of the sequence, and improve the performance robustness and generalization ability of the algorithm.

[0145] 1.3 Model construction and training

[0146] 1.3.1 Transformer Algorithm

[0147] When processing sequence data, RNN and its variants (such as LSTM and GRU) must make predictions in chronological order. The data at the previous moment will affect the calculation at the next moment, resulting in serial calculation. Unlike the RNN series of algorithms, Transformer has full parallelization capabilities and can simultaneously consider the relationship between all time points in the sequence. It does not rely heavily on the training of the data at the previous moment, fully utilizes hardware resources and greatly reduces training time, solving the serial calculation limitation of the RNN series of algorithms caused by the increase in sequence length. In addition, RNN algorithms need to store complete historical information of monitoring indicators, while Transformer only needs to store attention weights, reducing memory consumption.

[0148] refer to Figure 4 Transformer overcomes the limitations of traditional sequence models by building an encoder-decoder structure based on the self-attention mechanism. The self-attention mechanism layer can focus on multiple positions in the sequence, solve the long-distance dependency problem of monitoring data, and pass it to the feedforward neural network to increase the model's fitting ability. The residual network structure is used to solve the gradient disappearance problem, and the normalization operation is used to accelerate the convergence of the model. In order to avoid leaking the true training value in the decoder, a mask matrix is ​​used for the input of the self-attention layer, so that the decoder only uses past information to predict the future.

[0149] Transformer is suitable for capturing long-distance features of monitoring indicators. It uses the self-attention mechanism to calculate the correlation between long-distance features to avoid the attenuation of the effectiveness of the captured features as the time distance increases. It pays more attention to the features of any two time nodes of the input monitoring data. The formula of the self-attention mechanism is as follows:

[0150]

[0151] After dynamically encoding the monitoring sequence data, it is input into the Transformer and converted into a vector using the weight matrix W. q W k W v Get the corresponding qkv, and combine them to get the QKV vector, where Q is the query vector of the current data, K is the key vector of the data features at other times, and V is the value vector at each time. The self-attention mechanism performs a dot product by transposing Q and K to obtain the original eigenvalues ​​of the monitoring indicator sequence, which reflects the relationship strength between the positions in the monitoring data sequence. In order to prevent the product from being too large and causing the gradient to approach 0, the original eigenvalue is reduced by dividing the dimension, and Softmax is used for normalization. Finally, the dot product is performed with the value vector, which has the effect of weighting the value vector of each time node to obtain the final monitoring data features.

[0152] Transformer also uses a multi-head attention mechanism to enhance feature extraction capabilities, that is, by capturing information from different representation subspaces. It consists of multiple self-attention mechanisms, and the formula is as follows:

[0153] Multi-Head(Q,K,V)=Concat(head1,...,head h )W O ,

[0154] head i =Self-Attention(Q i ,K i ,V i )

[0155] The temporal correlation of monitoring data is learned from h different feature subspaces, the features are cascaded through weights, the QKV vector matrix is ​​calculated in parallel, combined into a multi-dimensional matrix, and then linearly transformed to map it to the original monitoring data dimension to obtain the output result, which enhances the expressiveness of the monitoring indicator prediction model and its adaptability to data diversity.

[0156] 1.3.2 Autoregressive Network

[0157] As a typical representative of linear prediction, the autoregressive model predicts future values ​​by fitting the data of historical monitoring indicators. Its main principle is to predict the estimated value P at the next time point through multiple linear regression of the previous n time points of the monitoring data. t , the error term is the random interference ε at the current moment t , the model formula is as follows:

[0158] P t =φ0+φ1P t-1 +φ2P t-2 +...+φ m P t-m +ε t

[0159] 1.3.3 Fusion

[0160] Transformer can perform time series prediction for multiple variables, and AR model is a time prediction model for single variable. Therefore, in this method, the monitoring data is input into AR model in parallel for prediction respectively. After weighted fusion of the prediction results and Transformer model prediction results, the final monitoring index prediction is achieved, and the combination of linear and nonlinear prediction models, as well as the combination of single variable and multivariate time series prediction, is realized, which effectively improves the prediction accuracy and creates a more robust and general prediction model. The specific model is as follows: Figure 5 shown.

[0161] In summary, this application first obtains the monitoring index data of the distributed storage system through monitoring data collection, and outputs the prediction result 1 through the autoregressive network. At the same time, the collected monitoring data is dynamically encoded to embed the position information into the data, and the prediction result 2 is output through the Transformer model. The prediction result 1 and the prediction result 2 are weighted and fused to obtain the final prediction result. The flow chart of this method is as follows: Figure 6 shown.

[0162] 1.4. Model training and evaluation optimization

[0163] The training process of the distributed storage system monitoring indicator prediction model is as follows:

[0164] 1.4.1. Execute the forward propagation process of the network

[0165] (1) The training data is sent to the embedding layer for dynamic position encoding to maintain the relative position information in the monitoring sequence. At the same time, the monitoring data is sent to the autoregressive network to obtain the prediction result 1.

[0166] (2) Encoder

[0167] The output data of the dynamic position encoding layer passes through a series of encoding layers to mine the deep information of the monitoring data. Each layer includes a multi-head attention module and a feedforward neural network model. The multi-head attention module is used to capture the dependency of the input monitoring sequence to enhance the monitoring indicator prediction model's ability to capture dynamic changes in data. The feedforward neural network model can further transform features and enrich the data representation capabilities.

[0168] (3) Decoder

[0169] The decoder consists of multiple layers, each of which includes a masked multi-head self-attention model, an encoder-decoder attention module, and a feedforward neural network module. The masked multi-head self-attention model prevents the decoder from seeing future information by masking the training truth. The encoder-decoder attention module connects the encoder and the decoder to fuse the encoder's output information with the current decoding state, allowing the decoder to refer to the encoder's output. The feedforward neural network module further transforms the processed features to generate more accurate and rich output features.

[0170] (4) Output layer

[0171] The decoder output generates a probability distribution through the linear output layer and the Softmax function, and the prediction result 2 is obtained through the cross entropy loss function. The prediction result 1 obtained by the autoregressive network and the prediction result 2 of the Transformer model are weighted and fused to obtain the final distributed storage system monitoring indicator prediction result.

[0172] 1.4.2. Execute the back propagation process of the network

[0173] Back propagation aims to optimize the model weights and minimize the loss function by calculating the gradient of the loss function with respect to the model weights and updating the weights in the opposite direction of the gradient. This algorithm uses the root mean square error (RMSE), y i and They represent the predicted value and true value of the monitoring indicator respectively, and the formula is as follows:

[0174]

[0175] The loss between the predicted value and the expected value of the monitoring prediction model is calculated and back-propagated along the decoder and encoder. During this process, the Adam optimizer is used to update the model weights according to the calculated gradient. This process is repeated until all the monitoring indicator training data are sent to the model to complete the training. The learning rate is updated and one training is completed. When the specified number of training cycles is reached, the training ends and the trained model is saved.

[0176] 1.4.3 Model Optimization

[0177] The cross-validation method is used to evaluate the generalization ability of the model and improve the accuracy of predicting future monitoring indicators. The model parameters are adjusted through the random search method to find the configuration with the best prediction performance.

[0178] 1.5 Deployment and Monitoring

[0179] Deploy the trained model to the production environment where the distributed storage monitoring system is located to achieve real-time monitoring and prediction. In order to maintain the accuracy of the prediction of monitoring indicators, the performance of the model can be evaluated regularly and updated in a timely manner according to business needs and indicator updates.

[0180] The above-mentioned prediction method of distributed storage system monitoring indicators can achieve the following technical effects:

[0181] (1) Multi-dimensional monitoring indicators: Compared with conventional hardware monitoring indicators, this application also adds distributed storage system indicators, such as user read and write data volume, cloud and download data volume, upload and download speed and other system performance indicators, enriching the types of monitoring data. It can not only detect the operating status of the server cluster, but also record the performance of the distributed storage system, providing a data basis for identifying system performance bottlenecks and providing data support for subsequent resource adjustments and code optimization.

[0182] (2) A dynamic position coding algorithm is proposed for monitoring data: This application provides a dynamic coding method based on data length. Compared with the traditional sine-cosine position coding method, the dynamic coding method can automatically adjust the coding method according to the length of the data input, so that the prediction model is more adaptable, the calculation efficiency is higher, the position retention is more complete, and it can better balance the relationship between the complexity of the model and the amount of data.

[0183] (3) A prediction algorithm combining Transformer and AR model is proposed for monitoring indicator prediction: For monitoring indicator prediction, this application combines the Transformer model with the autoregressive model. This fusion strategy not only combines the powerful representation ability of deep learning with the stability advantage of the autoregressive model in time series analysis, but also successfully achieves comprehensive capture of long-term and short-term dependencies and covers the complex characteristics of multi-scale time series. By combining the complex feature extraction capability of Transformer with the robust prediction mechanism of AR model, the constructed prediction model shows higher robustness when facing diversified monitoring data and can effectively resist abnormal fluctuations in the data. In addition, this combination method utilizes the nonlinear prediction advantages of deep learning and the linear prediction stability of statistical models, effectively making up for the limitations of a single model in prediction accuracy, thereby achieving a more accurate prediction effect compared to a single model.

[0184] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0185] Based on the same inventive concept, the embodiment of the present application also provides a monitoring indicator prediction device for implementing the monitoring indicator prediction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more monitoring indicator prediction device embodiments provided below can refer to the limitations of the monitoring indicator prediction method above, and will not be repeated here.

[0186] In an exemplary embodiment, Figure 7 As shown, a monitoring indicator prediction device is provided, including: a data acquisition module 710, a coding processing module 720, an indicator prediction module 730 and a result determination module, wherein:

[0187] The data collection module 710 is used to collect the current monitoring data of the distributed storage system;

[0188] The encoding processing module 720 is used to perform position encoding processing on the current monitoring data to obtain processed monitoring data.

[0189] The indicator prediction module 730 is used to input the processed monitoring data into the first monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the first monitoring indicator prediction result of the distributed storage system, and to input the current monitoring data into the second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain the second monitoring indicator prediction result of the distributed storage system.

[0190] The result determination module 740 is used to obtain a target monitoring indicator prediction result of the distributed storage system according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

[0191] In an exemplary embodiment, the data acquisition module 710 is also used to collect system performance indicators of the distributed storage system and basic monitoring indicators of the server cluster associated with the distributed storage system; based on the system performance indicators of the distributed storage system and the basic monitoring indicators of the server cluster, the current monitoring data of the distributed storage system is obtained.

[0192] In an exemplary embodiment, the encoding processing module 720 is also used to perform dynamic position encoding processing on the current monitoring data to obtain the position information of the current monitoring data; and to splice the current monitoring data and the position information to obtain processed monitoring data.

[0193] In an exemplary embodiment, the result determination module 740 is also used to obtain a first weight corresponding to a first monitoring indicator prediction result, and to obtain a second weight corresponding to a second monitoring indicator prediction result; the first monitoring indicator prediction result and the second monitoring indicator prediction result are fused according to the first weight and the second weight to obtain a target monitoring indicator prediction result of the distributed storage system.

[0194] In an exemplary embodiment, the monitoring indicator prediction device provided by the present application also includes a model training module, which is used to collect sample monitoring data of a sample distributed storage system; the sample monitoring data includes system performance indicators of the sample distributed storage system, and basic monitoring indicators of a sample server cluster associated with the sample distributed storage system; the sample monitoring data is position-encoded to obtain processed monitoring data corresponding to the sample monitoring data; the sample monitoring data and the processed monitoring data corresponding to the sample monitoring data are predicted through the monitoring indicator prediction model to be trained to obtain a target monitoring indicator prediction result of the sample distributed storage system; the actual result of the target monitoring indicator of the sample distributed storage system is obtained, and according to the difference between the target monitoring indicator prediction result of the sample distributed storage system and the actual result of the target monitoring indicator, the monitoring indicator prediction model to be trained is iteratively trained to obtain a pre-trained monitoring indicator prediction model.

[0195] In an exemplary embodiment, the model training module is also used to input the processed monitoring data corresponding to the sample monitoring data into the first monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain the first monitoring indicator prediction result of the sample distributed storage system, and input the sample monitoring data into the second monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain the second monitoring indicator prediction result of the sample distributed storage system; the first monitoring indicator prediction result and the second monitoring indicator prediction result of the sample distributed storage system are fused to obtain the target monitoring indicator prediction result of the sample distributed storage system.

[0196] Each module in the above monitoring index prediction device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0197] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as current monitoring data, target monitoring index prediction results, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a monitoring index prediction method is implemented.

[0198] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0199] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0200] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0201] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0202] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0203] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0204] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A monitoring index prediction method, characterized in that: The method comprises: Collect current monitoring data of distributed storage systems; Performing position encoding processing on the current monitoring data to obtain processed monitoring data; Inputting the processed monitoring data into a first monitoring indicator prediction model in a pre-trained monitoring indicator prediction model to obtain a first monitoring indicator prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the distributed storage system; A target monitoring indicator prediction result of the distributed storage system is obtained according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

2. The method according to claim 1, characterized in that The collecting of current monitoring data of the distributed storage system includes: Collecting system performance indicators of the distributed storage system and basic monitoring indicators of the server cluster associated with the distributed storage system; According to the system performance index of the distributed storage system and the basic monitoring index of the server cluster, the current monitoring data of the distributed storage system is obtained.

3. The method according to claim 1, characterized in that The performing position encoding processing on the current monitoring data to obtain processed monitoring data includes: Performing dynamic position coding processing on the current monitoring data to obtain position information of the current monitoring data; The current monitoring data and the location information are spliced ​​together to obtain the processed monitoring data.

4. The method according to claim 1, characterized in that: The obtaining, according to the first monitoring indicator prediction result and the second monitoring indicator prediction result, a target monitoring indicator prediction result of the distributed storage system includes: Obtaining a first weight corresponding to the prediction result of the first monitoring indicator, and obtaining a second weight corresponding to the prediction result of the second monitoring indicator; According to the first weight and the second weight, the first monitoring indicator prediction result and the second monitoring indicator prediction result are fused to obtain the target monitoring indicator prediction result of the distributed storage system.

5. The method according to any one of claims 1 to 4, characterized in that: The pre-trained monitoring indicator prediction model is trained in the following way: Collecting sample monitoring data of a sample distributed storage system; the sample monitoring data includes system performance indicators of the sample distributed storage system and basic monitoring indicators of a sample server cluster associated with the sample distributed storage system; Performing position coding processing on the sample monitoring data to obtain processed monitoring data corresponding to the sample monitoring data; The sample monitoring data and the processed monitoring data corresponding to the sample monitoring data are predicted by the monitoring indicator prediction model to be trained to obtain the target monitoring indicator prediction result of the sample distributed storage system; The actual results of the target monitoring indicators of the sample distributed storage system are obtained, and according to the difference between the predicted results of the target monitoring indicators of the sample distributed storage system and the actual results of the target monitoring indicators, the monitoring indicator prediction model to be trained is iteratively trained to obtain the pre-trained monitoring indicator prediction model.

6. The method according to claim 5, characterized in that The monitoring indicator prediction model to be trained is used to perform prediction processing on the sample monitoring data and the processed monitoring data corresponding to the sample monitoring data to obtain the target monitoring indicator prediction result of the sample distributed storage system, including: Inputting the processed monitoring data corresponding to the sample monitoring data into a first monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain a first monitoring indicator prediction result of the sample distributed storage system, and inputting the sample monitoring data into a second monitoring indicator prediction model in the monitoring indicator prediction model to be trained to obtain a second monitoring indicator prediction result of the sample distributed storage system; The first monitoring indicator prediction result and the second monitoring indicator prediction result of the sample distributed storage system are fused to obtain the target monitoring indicator prediction result of the sample distributed storage system.

7. A monitoring index prediction device, characterized in that: The device comprises: A data collection module is used to collect current monitoring data of the distributed storage system; The coding processing module is used to perform position coding processing on the current monitoring data to obtain processed monitoring data; An indicator prediction module, used for inputting the processed monitoring data into a first monitoring indicator prediction model in a pre-trained monitoring indicator prediction model to obtain a first monitoring indicator prediction result of the distributed storage system, and inputting the current monitoring data into a second monitoring indicator prediction model in the pre-trained monitoring indicator prediction model to obtain a second monitoring indicator prediction result of the distributed storage system; A result determination module is used to obtain a target monitoring indicator prediction result of the distributed storage system according to the first monitoring indicator prediction result and the second monitoring indicator prediction result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images