Cluster monitoring method and electronic device

Abstract

The application discloses a cluster monitoring method and an electronic device, and relates to the technical field of computers. The cluster monitoring method comprises the following steps: when monitoring a monitoring index of a current cluster, predicting an abnormal probability of the monitoring index; combining a state of the monitoring index and an associated index associated with the monitoring index to evaluate a risk parameter representing a potential risk of the monitoring index; comprehensively monitoring the abnormal probability of the monitoring index itself and the risk parameter obtained by the combined action of the monitoring index and the associated index; matching a monitoring mode corresponding to the risk parameter as a target mode for monitoring the monitoring index; allowing to assign a differentiated monitoring mode according to the real state of each monitoring index; and dynamically adjusting the monitoring mode according to the risk parameter obtained by the evaluation. The technical problem of fixed cluster monitoring mode is solved, and the technical effect of adaptively adjusting the monitoring mode according to the state of the monitoring index is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a cluster monitoring method and electronic device. Background Technology

[0002] In clusters composed of electronic devices such as servers, storage devices, and personal computers, it is often necessary to monitor cluster metrics in order to promptly restore the cluster when these metrics indicate anomalies. However, the monitoring patterns for these metrics are usually relatively static, or simply involve increasing or decreasing the monitoring frequency based on current data, leading to a rigid cluster monitoring model. Summary of the Invention

[0003] This application provides a cluster monitoring method and an electronic device to at least solve the problem of fixed cluster monitoring modes in related technologies.

[0004] This application provides a cluster monitoring method, which includes: obtaining a monitoring threshold corresponding to the current cluster configuration as a target threshold; monitoring the current cluster's monitoring indicators within a preset period to obtain a monitoring sequence, and evaluating the anomaly probability corresponding to the difference between the monitoring sequence and the target threshold; fusing the anomaly probability of the monitoring indicators and an importance factor to obtain a first parameter, calculating a second parameter using the risk parameters of the associated indicators of the monitoring indicators, evaluating the risk parameters of the monitoring indicators corresponding to the first parameter and the second parameter; and matching the monitoring mode corresponding to the risk parameter as the target mode for monitoring the monitoring indicators.

[0005] This application also provides an electronic device, which includes: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the steps of any of the above-described cluster monitoring methods.

[0006] This application addresses the issue of fixed cluster monitoring modes by predicting the probability of anomalies in the monitored indicators and assessing risk parameters characterizing their potential risks by combining the status of the monitored indicators with those of their associated indicators. This allows for the comprehensive analysis of the anomaly probability of the monitored indicator itself and its interaction with associated indicators to obtain risk parameters. A monitoring mode corresponding to these risk parameters is then matched as the target mode for monitoring the indicators. This approach adapts to the actual state of each monitored indicator, allowing for differentiated monitoring modes, and enables dynamic adjustment of the monitoring mode based on the assessed risk parameters. Therefore, it solves the technical problem of fixed cluster monitoring modes and achieves the technical effect of adaptively adjusting the monitoring mode based on the status of the monitored indicators. Attached Figure Description

[0007] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of the cluster monitoring method of this application; Figure 2 This is a flowchart illustrating an embodiment of the cluster monitoring method of this application; Figure 3 This is a flowchart illustrating another embodiment of the cluster monitoring method of this application; Figure 4 This is a schematic diagram of the process of training a prediction model according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the cluster monitoring system of this application; Figure 6 This is a schematic diagram of another embodiment of the cluster monitoring system of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0010] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0011] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] The specific application environment architecture or specific hardware architecture on which the cluster monitoring method depends is described here.

[0013] Please see Figure 1 , Figure 1This is a schematic diagram illustrating an application scenario of an embodiment of the cluster monitoring method of this application.

[0014] In one embodiment, the cluster monitoring method can be designed for business operating environments such as distributed computing scenarios.

[0015] The current cluster 10 serves as the core business unit, consisting of multiple nodes with independent computing, storage, or scheduling capabilities. It can be specifically applied to scenarios such as cloud computing data centers, artificial intelligence model training clusters, and distributed database clusters. Each node is interconnected through the network to achieve parallel processing of tasks and resource collaboration to support business requests.

[0016] The cluster monitoring system 20 can be deployed within the current cluster 10 and establish data connections with each node that constitutes the current cluster 10.

[0017] The interactive device 30 can be considered as a requesting device or maintenance personnel. The interactive device 30 can interact with the current cluster 10 and the cluster monitoring system 20.

[0018] For example, cluster 10 can currently handle actual business processing tasks, such as data computation, task distribution, and data storage. Each node completes its assigned business tasks according to the cluster scheduling strategy. Simultaneously, each node feeds back to the cluster monitoring system 20, or the cluster monitoring system monitors the current cluster's indicators. In this way, the cluster monitoring system 20 can perform anomaly probability assessment, risk parameter calculation, and monitoring mode matching on the current cluster 10's monitoring indicators, achieving intelligent monitoring of the cluster's status. That is, the cluster monitoring system 20 implements cluster monitoring methods to monitor and manage the current cluster 10.

[0019] The embodiments of this application provide a cluster monitoring method. The working principle of the cluster monitoring method is described in detail below in conjunction with the execution flow of the cluster monitoring method.

[0020] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the cluster monitoring method of this application.

[0021] S101: Obtain the monitoring threshold corresponding to the current cluster configuration as the target threshold.

[0022] In this embodiment, the cluster configuration file can be read or the current cluster configuration can be identified to determine the monitoring threshold corresponding to the current cluster configuration and set it as the target threshold.

[0023] The current cluster consists of multiple nodes with independent computing, storage and scheduling capabilities. The cluster configuration includes, but is not limited to, the number of cluster nodes, node hardware specifications, types of services carried, service level agreement requirements, etc. Different configurations of the cluster correspond to different monitoring threshold standards. The target threshold serves as a benchmark reference for cluster status assessment, providing a quantitative benchmark for cluster monitoring and assessment.

[0024] S102: Monitor the current cluster's monitoring indicators within a preset period to obtain a monitoring sequence, and evaluate the anomaly probability corresponding to the difference between the monitoring sequence and the target threshold.

[0025] In this embodiment, the operating status of each node in the current cluster is continuously monitored according to a preset period, and a monitoring sequence of the corresponding monitoring indicators is collected. Optionally, the monitoring sequence may refer to the monitoring data of multiple monitoring indicators at the same time, forming a monitoring sequence; or, the monitoring sequence may refer to the data sequence formed by the monitoring data of the same monitoring indicator within a preset period according to the time sequence. In the following text, the time sequence monitoring data of the same monitoring indicator will be used as the monitoring sequence for description.

[0026] In response to acquiring the monitoring sequence, the difference between each indicator value in the monitoring sequence and the corresponding target threshold can be calculated. Based on the magnitude of the difference and the fluctuation trend, the probability of anomaly of the monitoring indicator can be assessed. Optionally, the larger the difference between the monitoring sequence and the target threshold, the more drastic the fluctuation is considered, and the higher the probability of anomaly; the smaller the difference, the more stable the fluctuation is considered, and the lower the probability of anomaly, so as to achieve a preliminary quantitative assessment of the monitoring indicators and the risk of anomalies in the cluster's operating status.

[0027] S103: The first parameter is obtained by integrating the anomaly probability and importance factor of the monitoring indicators. The second parameter is calculated using the risk parameters of the related indicators of the monitoring indicators. The risk parameters of the monitoring indicators corresponding to the first parameter and the second parameter are evaluated.

[0028] In this embodiment, in response to obtaining the anomaly probability of the monitoring indicator, the anomaly probability is fused with the importance factor of the corresponding monitoring indicator to obtain the first parameter; wherein, the importance factor is used to characterize the degree of impact of the monitoring indicator on the stability of cluster operation and business continuity. The historical risk parameters of the related indicators of the monitoring indicator can also be obtained and calculated to obtain the second parameter; related indicators refer to other monitoring indicators that have business logic or resource dependencies on the current monitoring indicator.

[0029] Thus, risk parameters for monitoring indicators are obtained through a comprehensive evaluation based on the first and second parameters, thereby achieving multi-dimensional and link-based quantification of the risks of cluster monitoring indicators.

[0030] S104: Match the monitoring mode corresponding to the risk parameters as the target mode for monitoring indicators.

[0031] In this embodiment, in response to the risk parameters of the monitoring indicator obtained from the evaluation, a corresponding monitoring mode can be matched according to the magnitude of the risk parameter, and the monitoring mode can be determined as the target mode for monitoring the indicator.

[0032] For example, when the risk parameter is below a preset low threshold, a regular monitoring mode can be matched, which extends the monitoring cycle and reduces the data collection frequency, thus reducing the monitoring system's consumption of cluster business resources. When the risk parameter is in a preset middle range, an enhanced monitoring mode can be matched, which maintains the regular monitoring frequency while increasing the monitoring dimensions of related indicators to achieve early detection of potential risks. When the risk parameter is above a preset high threshold, an emergency monitoring mode can be matched, which shortens the monitoring cycle, increases the data collection frequency, tracks indicator changes in real time, and triggers anomaly alarm mechanisms. By adaptively matching the target monitoring mode, the monitoring accuracy and timeliness under high-risk conditions of the cluster can be ensured while reducing monitoring resource consumption under normal conditions, thereby improving the intelligence and efficiency of cluster monitoring and control.

[0033] In other words, this embodiment uses the target threshold corresponding to the current cluster configuration as a benchmark, evaluates the probability of indicator anomalies through monitoring sequences within a preset period, and comprehensively quantifies indicator risks by combining indicator importance factors and associated indicator risk parameters, ultimately adaptively matching the corresponding monitoring mode. This can improve the accuracy of identifying abnormal risks in cluster operation, help ensure the stable operation of cluster services, and dynamically adjust monitoring strategies according to risk levels, reducing excessive consumption of monitoring resources, thereby effectively improving the intelligence level and operational efficiency of cluster monitoring.

[0034] In this embodiment, when monitoring the current cluster's monitoring indicators, the anomaly probability of the monitoring indicators is predicted. The risk parameters characterizing the potential risk are evaluated by combining the status of the monitoring indicators and their associated indicators. That is, the risk parameters can be obtained by comprehensively considering the anomaly probability of the monitoring indicators themselves and their combined effect with associated indicators. The monitoring mode corresponding to the risk parameters is matched as the target mode for monitoring the monitoring indicators. This allows for differentiating monitoring modes to be assigned based on the actual status of each monitoring indicator. Furthermore, the monitoring mode can be dynamically adjusted based on the evaluated risk parameters to achieve adaptive adjustment of the monitoring mode in combination with the status of the monitoring indicators.

[0035] Please see Figure 3 , Figure 3 This is a flowchart illustrating another embodiment of the cluster monitoring method of this application.

[0036] S201: Obtain the monitoring threshold corresponding to the current cluster configuration as the target threshold.

[0037] In this embodiment, the cluster configuration of the current cluster can be obtained, and the monitoring threshold corresponding to the cluster configuration can be used as the target threshold. The cluster configuration includes at least one of the services, nodes, and containers configured in the current cluster.

[0038] Optionally, a preset rated monitoring threshold can be used as the target threshold, and / or the target threshold can be dynamically adjusted based on the current operating status of the cluster.

[0039] Specifically, a set of rated baselines can be obtained. This set stores rated monitoring thresholds adapted to different cluster configurations. The target threshold is then selected from the rated baselines that match the current cluster's configuration. In other words, a set of rated baselines can be pre-maintained, categorized by cluster type, service level, node size, etc., to store rated monitoring thresholds adapted to different cluster configurations. When determining the target threshold, the actual configuration of the current cluster can be used as the query condition. The set of rated baselines is traversed, and the corresponding entries are precisely matched. The retrieved rated monitoring threshold is then directly used as the target threshold, ensuring that the threshold is compatible with the cluster's basic attributes.

[0040] The principle of dynamic adjustment of the target threshold is explained below. The current target threshold of the current cluster can be periodically used as the current threshold at preset intervals. Monitoring data for a preset forward interval is used as window data to predict and update the current threshold, resulting in a new target threshold and its fluctuation range. The preset interval is shorter than the preset forward interval to ensure a reasonable periodic update frequency and to guarantee the sufficiency and effectiveness of historical data used for prediction, thus promoting a dynamic threshold that aligns with the real-time operating trend of the current cluster.

[0041] S202: Construct a correlation map between monitoring indicators.

[0042] In this embodiment, the correlation between monitoring indicators can be assessed. This correlation includes direct and indirect correlations. Directly correlated monitoring indicators have a direct influence on each other, while indirectly correlated monitoring indicators influence each other through other monitoring indicators.

[0043] Constructing a correlation graph among monitoring indicators that matches their relationships allows for the retrieval of directly related monitoring indicators as their associated indicators when querying the correlation graph. This improves the efficiency of retrieval of associated indicators. In other words, monitoring indicators interact directly with their directly related indicators, while indirectly related indicators influence each other through intermediate indicators. Building a corresponding correlation graph based on these relationships clearly outlines the correlation network of each monitoring indicator, enabling visualization and systematic management of these relationships. Furthermore, querying the correlation graph allows for the rapid location of directly related indicators as their associated indicators, improving retrieval efficiency, enhancing the clarity of monitoring indicator relationships, reducing retrieval complexity, and further improving the systematic nature and accuracy of cluster monitoring.

[0044] Optionally, based on the correlation obtained from the assessment, a correlation graph can be constructed between monitoring indicators that match the correlation relationship, with each monitoring indicator as a graph node and the correlation strength as the graph edge weight.

[0045] S203: Monitor the current cluster's monitoring indicators within a preset period to obtain the monitoring sequence.

[0046] In this embodiment, monitoring data can be collected continuously for a preset period of time for each monitoring indicator in the current cluster. The collected monitoring data is then arranged and organized in chronological order to form a continuous and orderly monitoring sequence.

[0047] S204: Assess the probability of anomalies corresponding to the difference between the monitoring sequence and the target threshold.

[0048] In this embodiment, the target threshold may include a dynamic threshold baseline.

[0049] Therefore, the difference between the monitoring sequence and the dynamic threshold baseline can be used as the first factor. The dynamic standard deviation of the monitoring sequence or the dynamic threshold baseline can be used as the second factor. The ratio between the first and second factors can be used as the anomaly probability. This approach helps to reduce the calculation bias caused by the differences in the fluctuation amplitude of the indicators themselves, and improves the uniform comparability of the anomaly probabilities between monitoring indicators of different magnitudes and fluctuation characteristics. This, in turn, enhances the objectivity of reflecting the degree of anomaly in the indicators deviating from the normal range.

[0050] The following is an example of the specific formula for calculating the probability of anomalies. The specific formula for calculating the probability of anomalies can be: ; Where AnoScore represents the anomaly probability; y(t) represents the monitoring sequence; σ(t) represents the target threshold; σ(t) represents the dynamic standard deviation.

[0051] S205: The first parameter is obtained by combining the anomaly probability and importance factor of the integrated monitoring indicators.

[0052] In this embodiment, the static importance and dynamic importance of the monitoring indicators can be obtained. The importance factor includes both static and dynamic importance, with the influence period of static importance being longer than that of dynamic importance. A first parameter is obtained by weighted fusion of the anomaly probability, static importance, and dynamic importance. This approach considers both the long-term and short-term importance of the monitoring indicators in the current cluster and / or monitoring strategy, thereby improving the comprehensiveness of the first parameter in reflecting the degree of anomaly risk of the monitoring indicators.

[0053] For example, static importance and dynamic importance can be evaluated based on the business value, components, and hierarchical labels of the monitored metrics.

[0054] The evaluation principles of static importance and dynamic importance are illustrated below with examples.

[0055] When evaluating the static importance of monitoring metrics, the value is determined based on the inherent configuration information of the current cluster. Static importance is related to the long-term operational attributes of the cluster, and its impact period is relatively long and does not change frequently with real-time business fluctuations. Specifically, values ​​can be assigned based on the node type, business module level, resource coreness, and preset service guarantee strategies to which the monitoring metrics belong. For example, metrics related to core scheduling nodes, database services, and gateways have higher static importance, while metrics related to non-core log nodes and cache auxiliary nodes have lower static importance. This ensures that static importance stably reflects the fundamental importance of monitoring metrics within the overall cluster architecture, thus contributing to the long-term stability of the current cluster.

[0056] When assessing the dynamic importance of monitoring indicators, the value is determined based on the current real-time operating status of the cluster and short-term changes in business load. The impact period of dynamic importance is short and can be adjusted in real time according to business traffic, task concurrency, and resource usage trends. Specifically, it can be calculated based on the fluctuation range of current monitoring indicators, abnormal states of related indicators, changes in business request volume, and preset peak-period strategies. For example, during peak business periods, the dynamic importance of indicators such as request processing latency and concurrent connections will increase accordingly; when a certain indicator experiences sustained abnormal fluctuations, the dynamic importance of its related indicators will also adaptively increase. Dynamic importance determined in this way can quickly respond to real-time changes in cluster operation, making the importance factor more closely match the current actual risk scenario and reducing the lag in risk identification caused by relying solely on static configuration.

[0057] Optionally, a first weight, a second weight, and a third weight can be obtained. The first weight is higher than the second weight, and the second weight is higher than the third weight. The first weight is fitted to the anomaly probability, the second weight is fitted to the static importance, and the third weight is fitted to the dynamic importance; these three are then superimposed to obtain the first parameter. By assigning progressively decreasing first, second, and third weights to the anomaly probability, static importance, and dynamic importance respectively, and then weighting and superimposing these three to obtain the first parameter, the real-time anomaly level of the indicator itself can be prioritized in risk assessment, while also taking into account the long-term inherent importance of the indicator and the impact of short-term dynamic changes. This makes the calculation of the first parameter more closely reflect the actual operating state of the cluster, reducing the risk of misjudgment due to relying solely on anomaly values, and also reducing the weakening of the true risk due to excessive interference from secondary factors, effectively improving the accuracy and rationality of risk assessment for monitoring indicators.

[0058] The following is an example of the specific calculation formula for the first parameter. The specific calculation formula for the first parameter can be: BR=(Wa*AnoScore)+(Ws*StaImportance)+(Wd*DynImportance); Wherein, BR represents the first parameter, which can be considered the basic risk score of the monitoring indicator; Wa represents the first weight; AnoScore represents the anomaly probability; Ws represents the second weight; StaImportance represents the static importance; Wd represents the third weight; and DynImportance represents the dynamic importance.

[0059] Furthermore, the sum of the first weight, the second weight, and the third weight can be 1. For example, the first weight can be 0.7, the second weight can be 0.2, and the third weight can be 0.1.

[0060] S206: Calculate the second parameter using the risk parameters of the associated indicators of the monitoring indicators.

[0061] In this embodiment, the related indicators that are directly associated with the monitoring indicators are used as the target indicators.

[0062] Obtain the risk parameters of the target indicator, filter out the risk parameters that indicate abnormality, and superimpose them to obtain the target parameters.

[0063] The parameters of the fitted target and the preset boosting factor are used to obtain undetermined parameters. The smaller of the undetermined parameters and the preset second parameter threshold is taken as the second parameter. In other words, this embodiment reduces the risk of excessive amplification of associated risks by limiting the upper limit of the second parameter, i.e., the second parameter threshold. This helps to ensure that the calculation of associated risk transmission is within a relatively reasonable range, thereby improving the reliability of risk assessment of monitoring indicators. The following is an example of the specific calculation formula for the second parameter, which can be: CB = min(50, Sum(RiskScore of all directly associated abnormal nodes) * BF); Where CB represents the second parameter; min() represents the minimum value function; 50 represents the threshold of the second parameter; sum() represents the summation function; RiskScore represents the risk parameter; and BF represents the preset boost factor.

[0064] Optionally, the preset boosting factor can be an adjustable parameter. Taking a preset boosting factor value of 0.2 as an example, if the first parameter value of the monitoring indicator is not high, but the risk parameters of the three related indicators it depends on are 60, 70, and 80 respectively, then the second parameter value of the monitoring indicator is 42, which can significantly highlight its potential risk.

[0065] S207: Risk parameters for assessing the monitoring indicators corresponding to the first parameter and the second parameter.

[0066] In this embodiment, the first parameter and the second parameter can be combined and directly used as risk parameters.

[0067] Alternatively, the first and second parameters can be combined as the current parameter to obtain a risk parameter threshold. The smaller of the risk parameter threshold and the current parameter is then used as the risk parameter. This means the risk parameter threshold acts as an upper limit to constrain the risk parameter, keeping it within a relatively reasonable range. This reduces the numerical length of the risk parameter involved in the monitoring pattern matching process, improving matching efficiency and facilitating monitoring pattern matching. The second parameter threshold can be less than the risk parameter threshold. The following is an example of the specific calculation formula for the risk parameter: RiskScore = min(100, BR + CB); Where RiskScore represents the risk parameter; min() represents the minimum function; 100 represents the risk parameter threshold; BR represents the first parameter; and CB represents the second parameter.

[0068] S208: Match the monitoring mode corresponding to the risk parameters as the target mode for monitoring indicators.

[0069] In this embodiment, based on the numerical range of the calculated risk parameter, the corresponding regular monitoring, enhanced monitoring, or emergency monitoring modes can be adaptively matched, and the matched monitoring mode can be determined as the target mode for subsequent continuous monitoring of the indicator, thereby realizing the dynamic adaptive adjustment of the cluster monitoring strategy.

[0070] For example, in response to a risk parameter falling within the first range, the monitoring indicator is monitored at the first frequency, triggering an anomaly alert. In response to a risk parameter falling within the second range, the monitoring indicator is monitored at the second frequency, with anomaly alerts selectively triggered. In response to a risk parameter falling within the third range, the normal frequency is used as the third frequency, and the monitoring indicator is monitored at the third frequency. In response to a risk parameter falling within the fourth range, the monitoring indicator is monitored at the fourth frequency.

[0071] Among them, the parameter values ​​of the first interval, the second interval, the third interval, and the fourth interval decrease sequentially; the first frequency, the second frequency, the normal frequency, and the fourth frequency decrease sequentially.

[0072] As exemplified above, the maximum possible value of the risk parameter of the monitoring indicator is 100. Using this as an example, we will provide detailed examples of the four monitoring modes.

[0073] When the risk parameters of the monitored indicators are in the range of 80-100, they can be considered to be in the first range, and can be monitored in real time and anomaly alerts can be triggered. This can be considered as monitoring them at the highest frequency.

[0074] When the risk parameters of the monitored indicators are in the range of 60-80, they can be considered to be in the second range. They can be monitored at a relatively high frequency, and anomaly alerts can be triggered selectively. That is, anomaly alerts can be triggered or not, and the triggering of anomaly alerts can also be controlled based on the changing trend of the monitored indicators.

[0075] When the risk parameters of the monitored indicators are in the range of 30-60, they can be considered to be in the third range, and the monitoring of the indicators can be carried out at the regular frequency.

[0076] When the risk parameter of the monitored indicator is in the range of 0-30, it can be considered to be in the fourth range, so its monitoring frequency can be reduced to the lowest level.

[0077] Furthermore, when the risk parameter of the monitoring indicator is continuously at its lowest value, such as 0, the monitoring of the monitoring indicator can be put into hibernation. The risk parameter of the monitoring indicator can be periodically evaluated at a preset hibernation interval, such as 24 hours. This is equivalent to waking it up and detecting it, so as to evaluate the subsequent monitoring mode based on the detection results.

[0078] Please see Figure 4 , Figure 4 This is a schematic diagram of a process for training a prediction model according to an embodiment of this application.

[0079] S301: Obtain the monitoring data stream of the monitoring indicators and write the monitoring data stream to be written instruction to the message queue.

[0080] In this embodiment, monitoring data streams corresponding to various monitoring indicators can be collected. To ensure the orderly writing of data and reduce the risk of blocking the main monitoring process, the write instructions corresponding to the monitoring data streams can be encapsulated and written to a message queue. This allows for buffering of high-concurrency data through an asynchronous queue mechanism, reducing data loss or write delays caused by excessive instantaneous data volume.

[0081] S302: Access the message queue to execute instructions.

[0082] In this embodiment, a message queue can be accessed, and instructions within the message queue can be retrieved sequentially in a certain order and executed accordingly. For example, instructions to be written can be retrieved and written, and other types of instructions may also exist in the current message queue. By decoupling the data acquisition and data storage stages through the message queue, even if there are fluctuations in the monitored data stream, it is beneficial to ensure the stability of instruction execution and improve the robustness of cluster monitoring and prediction model training in high-concurrency scenarios.

[0083] S303: Group and store the monitoring data stream as historical data according to the monitoring indicators and indicator labels, and write it into the time series database.

[0084] In this embodiment, the monitoring data stream can be classified and aggregated according to the monitoring indicator name and corresponding indicator label, and then structured into structured historical data by dimension and written in batches to a time-series database for storage. The time-series database can be considered as a storage structure optimized for time series data, which can support high-speed writing, efficient querying, and historical trend backtracking.

[0085] S304: Determine if the training cycle has been reached.

[0086] In this embodiment, step S304 can be executed when it is determined that the training cycle has been reached; step S302 can be executed when it is determined that the training cycle has not been reached.

[0087] It's easy to understand that the training cycle is a pre-set model update interval used to periodically optimize and iterate the target threshold. Timing logic and time windows can be used to determine if the current moment meets the training trigger conditions. If the cycle hasn't been reached, the message queue continues to be consumed and data is written; if the cycle is reached, the threshold update process is initiated, balancing historical data accumulation with model iteration.

[0088] S305: Perform time series decomposition and prediction to obtain a new target threshold and its fluctuation range.

[0089] In this embodiment, corresponding historical data can be read from a time-series database, and the time-series data can be decomposed into components such as trend terms, periodic terms, and random terms. Numerical extrapolation is then performed based on a time-series prediction model, and a new target threshold is calculated by combining the current cluster's operating patterns with historical fluctuation ranges. Simultaneously, the upper and lower fluctuation ranges of the target threshold are generated. This time-series decomposition and prediction allows the target threshold to adapt to the cluster load change trend, thereby improving the rationality of the monitoring benchmark.

[0090] S306: Model parameters and prediction results of the serialization prediction model.

[0091] In this embodiment, information such as the model parameters, time series decomposition results, new target threshold, and floating range of the prediction model can be serialized and uniformly converted into a storable and transmittable format for storage. The serialized content can be used for subsequent model recovery, threshold loading, log traceability, and effect verification, which helps to ensure that the threshold update process is traceable and reusable, thereby improving the maintainability of system iteration.

[0092] In layman's terms, this embodiment maintains historical data within a rolling time window as the indicator data stream. For example, the time window could be 40 days, 45 days, or 50 days. Periodic detection is performed on the indicator data stream to obtain its main period, which could be 24 hours or 7 days (i.e., 168 hours). Thus, the duration of each training cycle, such as 6 hours, 7 hours, or 8 hours, allows the indicator data stream within the time window to be used to train the prediction model. This trained model then provides the expected value and expected fluctuation range for each time point within a future period, such as 6 hours, i.e., the new target threshold. and its floating range; wherein, the floating range can be represented by the first extreme value of the threshold. and the second extreme value of the threshold Endpoint value representation, first extreme value of the threshold The second extreme value above the threshold Furthermore, the model parameters and prediction results of the trained and updated prediction model can be serialized and stored in a time-series database.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0094] Embodiments of this application also provide a cluster monitoring system.

[0095] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the cluster monitoring system of this application.

[0096] In one embodiment, the cluster monitoring system may include an indicator module 21, an analysis module 22, and a scheduling module 23, which are interconnected.

[0097] The indicator module 21 is used to obtain the monitoring threshold corresponding to the current cluster configuration as the target threshold.

[0098] The analysis module 22 is used to monitor the monitoring indicators of the current cluster within a preset period to obtain the monitoring sequence, evaluate the anomaly probability corresponding to the difference between the monitoring sequence and the target threshold, integrate the anomaly probability of the monitoring indicators and the importance factor to obtain the first parameter, calculate the second parameter using the risk parameters of the associated indicators of the monitoring indicators, and evaluate the risk parameters of the monitoring indicators corresponding to the first parameter and the second parameter.

[0099] The scheduling module 23 is used to match the monitoring mode corresponding to the risk parameters as the target mode for monitoring indicators.

[0100] In layman's terms, the indicator module 21 can feed back the raw monitoring data of the monitoring indicators and the abnormal signals that represent the probability of abnormality to the analysis module 22. Based on this, the analysis module 22 evaluates the risk parameters and feeds back the risk parameters to the scheduling module 23, so that the scheduling module 23 can adapt and control the monitoring mode.

[0101] Please see Figure 6 , Figure 6 This is a schematic diagram of another embodiment of the cluster monitoring system of this application.

[0102] In one embodiment, the cluster monitoring system may further include a configuration detection module 24. The configuration detection module 24 may be connected to the indicator module 21.

[0103] The configuration detection module 24 can identify the cluster configuration of the current cluster. For example, the configuration detection module 24 can automatically identify the services, nodes, and containers of the current cluster, and push / match baseline configurations of metrics that match the cluster configuration as initial target thresholds. The baseline configuration of metrics can be a set of baseline thresholds that have been verified and can effectively ensure the reliability of the cluster. Optionally, the configuration detection module 24 can also assign metric tags to the monitoring metrics. Metric tags can include at least one of the following: business value, component, or layer.

[0104] The following combination Figure 6 The example illustrates the structure of a cluster monitoring system and provides a detailed explanation of its working principle.

[0105] Specifically, the indicator module 21 may include a data acquisition unit 211, a threshold evaluation unit 212, and an anomaly detection unit 213.

[0106] The data acquisition unit 211 can collect raw indicator data of the monitoring indicators. The threshold evaluation unit 212 can be used to evaluate the target threshold of each monitoring indicator. Optionally, the threshold evaluation unit 212 can integrate the prediction model described above, and can use algorithms such as machine learning to calculate a dynamic baseline as the target threshold for each monitoring indicator. At the same time, the dynamic baseline can learn the change cycle of the monitoring indicators and adapt accordingly. The anomaly detection unit 213 can calculate the degree of deviation of the current monitoring sequence of the monitoring indicator from the dynamic baseline and output its anomaly probability.

[0107] The analysis module 22 may include a perception analysis unit 221, a correlation analysis unit 222, and a risk scoring unit 223.

[0108] The perception and analysis unit 221 acts as a context-aware engine, performing context-aware analysis on time, business, topology, etc. The correlation analysis unit 222 analyzes the correlation between monitoring indicators to construct and maintain a correlation graph, thereby linking monitoring indicators and improving the reliability of monitoring indicator analysis. The risk scoring unit 223 performs correlation analysis on isolated detection indicators through the correlation graph, assessing their anomaly risk in conjunction with the current overall cluster operating status. This allows for the assessment of risk parameters for monitoring indicators as described above, quantifying the degree of attention required for each indicator. Optionally, the risk parameter can range from 0 to 100.

[0109] The scheduling module 23 may include a mode selection unit 231, a resource management unit 232, and a mode switching unit 233.

[0110] The mode selection unit 231 can select the appropriate monitoring mode based on the risk parameters. The resource management unit 232 can allocate corresponding resources based on the monitoring mode of the monitoring indicators; that is, the resource management unit 232 can adjust the monitoring resources for each monitoring indicator while balancing the overall load of the current cluster and / or the cluster monitoring system. The mode switching unit 233 actually executes the monitoring model switching steps, switching the current monitoring mode of the monitoring indicator to a monitoring mode that matches its risk parameters.

[0111] For a description of the features in the embodiment corresponding to the cluster monitoring system, please refer to the relevant description of the embodiment corresponding to the cluster monitoring method, which will not be repeated here.

[0112] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described cluster monitoring method embodiments.

[0113] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described cluster monitoring method embodiments at runtime.

[0114] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0115] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described cluster monitoring method embodiments.

[0116] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described cluster monitoring method embodiments.

[0117] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] The above provides a detailed description of the cluster monitoring method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A cluster monitoring method, characterized in that, The cluster monitoring method includes: Obtain the monitoring threshold corresponding to the current cluster configuration as the target threshold; The monitoring indicators of the current cluster within a preset period are monitored to obtain a monitoring sequence, and the anomaly probability corresponding to the difference between the monitoring sequence and the target threshold is evaluated. The first parameter is obtained by integrating the anomaly probability and importance factor of the monitoring indicator; the second parameter is calculated using the risk parameters of the associated indicators of the monitoring indicator; and the risk parameters of the monitoring indicator corresponding to the first parameter and the second parameter are evaluated. The monitoring mode corresponding to the risk parameter is used as the target mode for monitoring the monitoring indicator.

2. The cluster monitoring method according to claim 1, characterized in that, The step of fusing the anomaly probability and importance factor of the monitoring indicator to obtain the first parameter includes: acquiring the static importance and dynamic importance of the monitoring indicator; wherein, the importance factor includes the static importance and the dynamic importance, and the influence period duration of the static importance is longer than that of the dynamic importance; weightedly fusing the anomaly probability, the static importance, and the dynamic importance to obtain the first parameter; and / or, The evaluation of the anomaly probability corresponding to the difference between the monitoring sequence and the target threshold includes: using the difference between the monitoring sequence and the dynamic threshold baseline as a first factor; wherein the target threshold includes the dynamic threshold baseline; using the dynamic standard deviation of the monitoring sequence or the dynamic threshold baseline as a second factor; and using the ratio between the first factor and the second factor as the anomaly probability.

3. The cluster monitoring method according to claim 2, characterized in that, The weighted fusion of the anomaly probability, the static importance, and the dynamic importance includes: Obtain a first weight, a second weight, and a third weight; wherein the first weight is higher than the second weight, and the second weight is higher than the third weight. The first weight is fitted to the anomaly probability, the second weight is fitted to the static importance, and the third weight is fitted to the dynamic importance; the three are then superimposed.

4. The cluster monitoring method according to claim 1, characterized in that, The calculation of the second parameter using the risk parameters of the correlation indicators of the monitoring indicators includes: The related indicators that are directly associated with the monitoring indicators are used as target indicators. Obtain the risk parameters of the target indicator, filter out the risk parameters that represent anomalies, and superimpose them to obtain the target parameters; The target parameters are fitted with a preset boosting factor to obtain undetermined parameters, and the smaller of the undetermined parameters and the preset second parameter threshold is taken as the second parameter.

5. The cluster monitoring method according to claim 1 or 4, characterized in that, Before calculating the second parameter using the risk parameters of the correlation indicators of the monitoring indicators, the following steps are included: Assess the correlation between the monitoring indicators; wherein the correlation includes direct correlation and indirect correlation, the directly correlated monitoring indicators have a direct influence relationship, and the indirectly correlated monitoring indicators influence each other through other monitoring indicators; Construct a correlation graph among the monitoring indicators that matches the correlation relationship, so that querying the correlation graph will retrieve the monitoring indicators that are directly associated with the monitoring indicators as their associated indicators.

6. The cluster monitoring method according to claim 1, characterized in that, The risk parameters for evaluating the monitoring indicators corresponding to the first parameter and the second parameter include: The first parameter and the second parameter are combined to form the current parameter; Obtain a risk parameter threshold, and use the smaller of the risk parameter threshold and the current parameter as the risk parameter.

7. The cluster monitoring method according to claim 1, characterized in that, The step of obtaining the monitoring threshold corresponding to the current cluster configuration as the target threshold includes: Obtain the cluster configuration of the current cluster; wherein, the cluster configuration includes at least one of the services, nodes, and containers configured in the current cluster; The monitoring threshold corresponding to the cluster configuration is used as the target threshold.

8. The cluster monitoring method according to claim 7, characterized in that, The step of using the monitoring threshold corresponding to the cluster configuration as the target threshold includes: Obtain a rated baseline set; wherein, the rated baseline set is used to store rated monitoring thresholds adapted to different cluster configurations; query the rated monitoring threshold in the rated baseline set that matches the cluster configuration of the current cluster as the target threshold; and / or, The target threshold of the current cluster is periodically used as the current threshold at preset time intervals, and the monitoring data of the preset forward time is used as window data to predict and update the current threshold to obtain a new target threshold and its fluctuation range; wherein, the preset time is shorter than the preset forward time.

9. The cluster monitoring method according to claim 1, characterized in that, The monitoring mode that matches the risk parameter as the target mode for monitoring the monitoring indicator includes: In response to the risk parameter being in the first range, the monitoring indicator is monitored at a first frequency to trigger an anomaly alert; In response to the risk parameter being in the second range, the monitoring indicator is monitored at a second frequency, and anomaly alerts can be selectively triggered. In response to the risk parameter being in the third interval, the normal frequency is used as the third frequency, and the monitoring indicator is monitored at the third frequency. In response to the risk parameter being in the fourth interval, the monitoring indicator is monitored at the fourth frequency; Specifically, the parameter values ​​of the first interval, the second interval, the third interval, and the fourth interval decrease sequentially; the first frequency, the second frequency, the conventional frequency, and the fourth frequency decrease sequentially.

10. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer programs; A processor, configured to implement the steps of the cluster monitoring method as described in any one of claims 1 to 9 when executing the computer program.

Images