Method for evaluating the health status of software after changes through AI

Abstract

The present application proposes a method for evaluating the health status of software after a software change by means of AI, which relates to the field of data analysis technology, wherein the method includes: obtaining a KPI time series to be detected, and performing a median difference test on the obtained KPI time series; if a data point is detected to be higher than the threshold of the median difference test, unsupervised anomaly detection is performed on the curve of the KPI time series to obtain the anomaly score corresponding to each point; the anomaly score is used as the input of the extreme value theory to determine the dynamic threshold, and the curve is judged whether there is an abnormal point based on the comparison between the anomaly score of the point to be detected on the curve and the threshold corresponding to this point; if the curve has an abnormal point, the spatial regression algorithm is used to evaluate the curve to determine the type of anomaly. The present application using the above scheme can quickly and accurately determine whether the anomaly is caused by a software change, thereby completing the evaluation of the service change, greatly improving the efficiency of the service change evaluation.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a method and device for evaluating the health status of software after a software change by means of AI. Background Art

[0002] Microservice architecture and agile development have resulted in a complex service tree structure, which in turn leads to a large number of service changes every day. With the dramatic increase in business volume, public clouds undergo hundreds of service changes every day. Failed service changes have become one of the most important factors threatening the stability of cloud services. Operation and maintenance personnel manually perform a large number of service changes and rely solely on manual experience to judge the abnormalities of service software changes, which will greatly affect the efficiency and stability of software changes. In order to ensure the stability of the service, it is necessary to quickly and accurately evaluate whether the service change is successful so that a service rollback operation can be triggered immediately when the change fails to avoid greater losses caused by failed changes.

[0003] The current prior art mainly includes:

[0004] (1) OS anomaly detection and diagnosis

[0005] Diagnose hardware failures such as routers and disks by analyzing log data;

[0006] (2) Observe the trend of online indicator data and rely on manual experience to judge service anomalies

[0007] Operation and maintenance personnel stare at the screen with their naked eyes, observing the changes in indicator curves during and after the service change, and rely on manual experience to determine whether the service change is abnormal.

[0008] Disadvantages of existing technology:

[0009] 1) Operation and maintenance personnel only rely on online data analysis and manual experience to determine whether service changes are abnormal

[0010] This method does not use historical data, and cannot compare current indicator changes with historical data change trends. It is also easy to make misjudgments by relying solely on the manual experience of operation and maintenance personnel.

[0011] 2) Not robust to multiple types of KPI curves

[0012] In the detection results, only one type of curve is detected well, and it is impossible to achieve good detection status for multiple types of curves.

[0013] 3) It is necessary to set a specific threshold for each indicator curve

[0014] The setting of thresholds is heavily dependent on the personal experience of the operation and maintenance engineer and the specific business scenario at the moment. In addition, these set thresholds need to be constantly modified as the business volume grows, the business scenario changes, or the machine environment changes.

[0015] 4) Anomaly detection is only performed on the changed services without further verification of whether the anomaly is caused by software changes. Anomaly detection uses the machine indicator and business indicator data of the service changes for analysis and judgment, and does not compare the indicator data of the services without software changes under the same conditions before and after the change. It is impossible to distinguish whether the anomaly is caused by software changes or other reasons.

[0016] 5) Use the same detection scheme for different actual change scenarios

[0017] Since the release methods of common change scenarios such as rolling release, canary release, and blue-green release are different, the drainage methods are different during the service change process, and the same detection solution cannot achieve the best detection effect. Summary of the invention

[0018] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0019] To this end, the first purpose of this application is to propose a method for evaluating the health status of software after changes through AI. Based on the current service change scenario and the historical data generated by the service, with machine learning algorithm as the core, it analyzes the key performance indicators of the service and the machine, provides further intelligent anomaly detection, and quickly and accurately determines whether the anomaly is caused by software changes, thereby completing the evaluation of the service change, so that the operation and maintenance personnel can roll back the service or perform the next operation in time according to the feedback of the algorithm results when operating the service change, changing the situation of manual screen monitoring, and largely liberating the manual energy of the operation and maintenance personnel, avoiding greater losses caused by failed changes, and greatly improving the efficiency of service change evaluation.

[0020] The second purpose of this application is to propose a device for evaluating the health status of software after changes through AI.

[0021] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a method for evaluating the health status of software after change by means of AI, comprising: obtaining a KPI time series to be detected, and performing a median difference test on the obtained KPI time series;

[0022] If a data point is detected to be higher than the median difference test threshold, unsupervised anomaly detection is performed on the KPI time series curve to obtain the anomaly score corresponding to each point;

[0023] The anomaly score is used as the input of extreme value theory to determine the threshold, and the curve is judged whether there is an abnormal point based on the comparison between the threshold and the anomaly score of each point on the curve;

[0024] If the curve has abnormal points, the spatial regression algorithm is used to evaluate the curve and determine the type of abnormality.

[0025] The method for evaluating the health status of software after the change by AI in the embodiment of the present application, based on the historical indicator data generated by the service in the current service change scenario, takes the machine learning algorithm as the core, trains the offline data to generate the service model, and then performs intelligent anomaly detection on the online data. Through algorithm calculation, it avoids the need to directly set the threshold for each curve, and enhances the robustness of the algorithm. The service with software change is used as the experimental group, and the service without software change under the same conditions and the same component is used as the control group. The statistical dependency structure between the experimental group and the control group before the change is learned to better predict the difference between the experimental group and the control group after the change, and quickly and accurately determine whether the anomaly is caused by the software change, thereby completing the evaluation of the service change, so that the operation and maintenance personnel can roll back the service or perform the next operation in time according to the feedback of the algorithm results when operating the service change, changing the manual screen watching situation, liberating the manual energy of the operation and maintenance personnel to a large extent, avoiding greater losses caused by failed changes, and greatly improving the efficiency of service change evaluation. In addition, this patent designs corresponding change evaluation schemes for different actual change scenarios, so that better detection effects can be achieved in different change scenarios.

[0026] Optionally, in one embodiment of the present application, performing a median difference test on the acquired KPI time series includes:

[0027] Calculate the median difference coefficient of each point on the curve of the KPI time series, where the median difference coefficient of each point is obtained by subtracting the median of the previous sequence and the median of the next sequence of the point and finding the absolute value;

[0028] A threshold is set to be compared with the median difference coefficient at each point on the curve.

[0029] Optionally, in one embodiment of the present application, unsupervised anomaly detection is performed on a curve of a KPI time series, including:

[0030] Preprocess the KPI time series data to generate time window data;

[0031] Through the MCMC method, reconstructed data is generated based on the distribution of other non-missing point data in the time window data, and the reconstructed data is used to replace the corresponding missing data;

[0032] Input the padded time window data into the trained conditional variational autoencoder and output the anomaly score corresponding to each point.

[0033] Among them, data preprocessing includes:

[0034] Fill missing points in the KPI time series data with 0 and make the intervals of the time series data the same;

[0035] Normalize the time series after padding.

[0036] Optionally, in one embodiment of the present application, the anomaly score corresponding to each point is expressed as:

[0037]

[0038] Among them, x represents the data of the input time window, y represents the time information of the input time window, and z represents the prior distribution of the low-dimensional latent space. Represents the conditional distribution q Φ Find the expectation under the condition of sampling z in (z|x, y), logp θ (x|z, y) represents the logarithm of the probability density of the conditional distribution P(x|z, y). represents the logarithmic probability density of the input x in the reconstructed distribution of the decoder output.

[0039] Optionally, in one embodiment of the present application, training a conditional variational autoencoder model includes:

[0040] Obtain normal KPI time series, perform data preprocessing on the normal KPI time series, and generate time window data to form a training data set;

[0041] Using the training data set, the model is continuously trained based on the M-ELBO loss function and the SGVB optimization algorithm until the model converges and the training is completed.

[0042] Optionally, in one embodiment of the present application, the M-ELBO loss function is expressed as:

[0043]

[0044] Among them, x represents the data of the input time window, x i represents a point in the input time window, y represents the time information of the input time window, and z represents the prior distribution of the low-dimensional latent space. Represents the conditional distribution Find the expectation under the condition of sampling z, logp(x i|z, y) represents the logarithm of the probability density of the conditional distribution P(x|z, y), logp(z|y) represents the logarithm of the probability density of the conditional distribution P(z|y), Represents the logarithm of the probability density of the conditional distribution P(z|x, y), α i Indicates whether the i-th point in the time window is an outlier, β indicates whether the i-th point in the time window is an outlier, and W indicates the length of the time window.

[0045] Optionally, in one embodiment of the present application, the anomaly score is used as an input of extreme value theory to determine a threshold, including:

[0046] Set the peak threshold to filter out the peak value in the time series before the point to be detected, where the peak value in the sequence is the part that exceeds the peak threshold;

[0047] Calculate the initial abnormal threshold based on the sequence data, where the sequence data includes the total length of the sequence, the peak threshold, the number of peaks, and the risk probability, where the risk probability is the hyperparameter to be adjusted;

[0048] The initial anomaly threshold is updated using the data of the KPI time series to be detected to obtain the final threshold.

[0049] Optionally, in one embodiment of the present application, a spatial regression algorithm is used to evaluate the anomaly, including:

[0050] Step S1: A sampling control group is simply randomly selected from the control group, wherein the control group is the relevant indicators in the machine where the same service is deployed as the machine where the software change occurs and the software change does not occur, and the sampling control group data includes the time series matrix before and after the change;

[0051] Step S2: learning linear regression parameters based on the time series matrix of the control group before the change and the time series vector of the experimental group to be tested before the change;

[0052] Step S3: Use the learned parameters and the time series matrix of the control group before and after the change to predict the time series vector to be tested before and after the change;

[0053] Step S4: Repeat steps S1, S2, and S3 to obtain a predicted time series vector group;

[0054] Step S5: Calculate the median of the values ​​of different prediction time vectors at the same time point for the prediction time series vector group to obtain a new prediction vector, and then make a difference between the time series before and after the change to be tested and the new prediction vector to obtain a difference sequence before and after the change;

[0055] Step S6: using a robust rank test to compare the difference sequence before the change and the difference sequence after the change to obtain a significant difference score;

[0056] Step S7: Set a threshold value and compare it with the significant difference score. If it exceeds the threshold value, it is determined to be an anomaly caused by software change; otherwise, it is an anomaly caused by other reasons.

[0057] Optionally, in one embodiment of the present application, a median difference test is performed on the acquired KPI time series to determine that all data points in the sequence are below the threshold of the median difference test; or based on the comparison of the threshold with the anomaly score of each point on the curve, it is determined that the curve does not contain abnormal points, and it is determined that the software is in a healthy state after the change.

[0058] To achieve the above purpose, the second embodiment of the present invention proposes a device for evaluating the health status of software after change by AI, including a median difference inspection module, an abnormal score determination module, an abnormal point detection module, and an evaluation module, wherein:

[0059] The median difference test module is used to obtain the KPI time series to be tested and perform a median difference test on the obtained KPI time series;

[0060] The anomaly score determination module is used to perform unsupervised anomaly detection on the KPI time series curve if a data point is detected to be higher than the median difference test threshold, and obtain the anomaly score corresponding to each point;

[0061] The outlier detection module is used to determine the threshold by taking the anomaly score as the input of extreme value theory, and to judge whether the curve has an anomaly point by comparing the threshold with the anomaly score of each point on the curve;

[0062] The evaluation module is used to evaluate the curve using a spatial regression algorithm if there are abnormal points on the curve to determine the type of abnormality.

[0063] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0065] Figure 1 A flowchart of a method for evaluating the health status of software after a software change by means of AI provided in Embodiment 1 of the present application;

[0066] Figure 2 This is a schematic diagram of the median difference test of an embodiment of the present application;

[0067] Figure 3A schematic diagram of the Bagel algorithm structure of an embodiment of the present application;

[0068] Figure 4 A schematic diagram of a time information encoding process according to an embodiment of the present application;

[0069] Figure 5 A schematic diagram of the Bagel algorithm training process in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of the SPOT detection process of an embodiment of the present application;

[0071] Figure 7 The relative changes of the experimental group and the control group of the present application example;

[0072] Figure 8 The figure is a schematic diagram of the overall algorithm flow of the embodiment of the present application.

[0073] Fig. 9 This is a structural diagram of an online algorithm deployment detection system according to an embodiment of the present application;

[0074] Fig.10 A schematic diagram of the structure of a device for evaluating the health status of software after a software change by means of AI provided in an embodiment of the present application; DETAILED DESCRIPTION

[0075] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0076] The following describes the method and device for evaluating the health status of software after modification by means of AI in an embodiment of the present application with reference to the accompanying drawings.

[0077] Figure 1 A flowchart of a method for evaluating the health status of software after a software change by means of AI provided in Example 1 of the present application.

[0078] like Figure 1 As shown, the method for evaluating the health status of software after the change by AI includes the following steps:

[0079] Step 101, obtaining a KPI time series to be tested, and performing a median difference test on the obtained KPI time series;

[0080] Step 102: If a data point is detected to be higher than the threshold of the median difference test, unsupervised anomaly detection is performed on the curve of the KPI time series to obtain an anomaly score corresponding to each point;

[0081] Step 103, using the anomaly score as an input of extreme value theory to determine a threshold, and judging whether the curve has an abnormal point by comparing the threshold with the anomaly score of each point on the curve;

[0082] Step 104: If the curve has an abnormal point, a spatial regression algorithm is used to evaluate the curve to determine the type of abnormality.

[0083] The method for evaluating the health status of software after a software change by means of AI in the embodiment of the present application, based on the historical data generated by the service in the current service change scenario and the service, takes the machine learning algorithm as the core, analyzes the key performance indicators of the service and the machine, provides further intelligent anomaly detection, and quickly and accurately determines whether the anomaly is caused by the software change, thereby completing the evaluation of the service change, allowing the operation and maintenance personnel to roll back the service or perform the next operation in time according to the feedback of the algorithm results when operating the service change, changing the situation of manual screen monitoring, freeing up the manual energy of the operation and maintenance personnel to a large extent, avoiding greater losses caused by failed changes, and greatly improving the efficiency of service change evaluation.

[0084] Furthermore, in the embodiment of the present application, a median difference test is performed on the acquired KPI time series, including:

[0085] The Bagel algorithm is sensitive to jitter and noise during detection, but does not pay attention to jitter and noise in KPIs when performing drastic changes in KPIs during change assessment. Based on the fact that the median of a set of data will not be affected by extreme values, a median difference test is proposed by combining the sliding window and the median. Figure 2 As shown, the main idea is to traverse the KPI time series from front to back. The median difference coefficient of each point is the median of the previous sequence and the median of the next sequence. The difference is calculated and the absolute value is obtained.

[0086] The time window of the median difference test is an empirical value. If the time window is too small, the noise cannot be filtered out. If the time window is too large, anomalies with a long duration may be filtered out, resulting in missed detection. The threshold of the median difference test may be different for different indicators, and is also an empirical value. However, it is different from setting a fixed threshold directly for the indicator to determine whether the indicator is abnormal. The threshold of the median difference test is an estimate of the maximum change amplitude of the indicator. If the change amplitude does not exceed this threshold and does not last long enough, it will be directly filtered out by the median difference test. Therefore, this threshold can generally be set relatively small. It is worth mentioning that the median difference test is only used to filter out jitter and noise, and plays an auxiliary role. When the indicator is a fixed type or there are a large number of indicators, the default threshold of 0 can be used.

[0087] Specifically, in the embodiment of the present application, the Bagel algorithm is a relatively new single-indicator anomaly detection algorithm based on an unsupervised deep learning generative model, which is based on the Donut algorithm, another unsupervised deep learning generative model. It specifically solves the problem of detecting time-related anomalies and avoids the problem of false positives of periodic jitter (spike) in the curve. Since it is necessary to detect multiple types of KPI curves in change assessment, especially when the curve has obvious periodicity, the unsupervised learning ability of the Bagel algorithm is needed to assist in detecting anomalies in periodic curves, such as Figure 3 As shown in the figure, the Bagel algorithm mainly consists of three parts: data preprocessing, training, and prediction.

[0088] Variational Autoencoder and Conditional Variational Autoencoder:

[0089] Deep Bayesian networks combine deep learning and probabilistic graphical models (PGMs), which expand the capabilities of probabilistic graphical models and model the relationship between random variables and neural networks. Variational inference is a very useful way to solve the posterior probability problem of distributions derived from neural networks, so it is often used for training and prediction of deep Bayesian networks.

[0090] Variational autoencoders and conditional variational autoencoders are both typical deep Bayesian networks. Variational autoencoders model the relationship between two random variables x and z, while conditional variational autoencoders model the relationship between random variables x and z under condition y, which can be expressed as p(x, z|y). The reason for choosing conditional variational autoencoders in the Bagel algorithm is that the time information of the conditional variables is particularly important in KPI anomaly detection.

[0091] Therefore, the Bagel algorithm is used to perform unsupervised anomaly detection on the curve of the KPI time series, including:

[0092] Preprocess the KPI time series data to generate time window data;

[0093] During the detection process, missing points in the time window may have a negative impact on the detection effect. In order to reduce the detection bias caused by missing points, the MCMC method is used to generate reconstructed data based on the distribution of other non-missing point data in the time window data, and the reconstructed data is used to replace the corresponding missing data;

[0094] Input the padded time window data into the trained conditional variational autoencoder and output the anomaly score corresponding to each point.

[0095] Among them, data preprocessing includes:

[0096] Since the actual collected data often has some missing points, we first need to fill the missing points with 0 and make the intervals of the time series data the same, which is generally 1 minute or 5 minutes in the experiment.

[0097] Since the KPI data of different indicators vary greatly, the entire sequence needs to be standardized. The method is to subtract the mean of the entire sequence from each point in the sequence and then divide it by the standard deviation.

[0098] Since the experimental data to be studied is a KPI time series, and the conditional variational autoencoder is not a time series model, the original KPI data needs to be processed in the form of a sliding window of the KPI series as the input of the model, and the conditional variable y represents the time information of each input time window. In order to emphasize the periodicity, the timestamp corresponding to each time window is decomposed into the following components: minutes, hours, and days of the week. The components to be extracted from the time information are encoded into a binary 01 vector using the one-hot method. Figure 4 Shows how to encode temporal information.

[0099] Specifically, in the embodiment of the present application, the reconstruction term in ELBO is used as a reference for the basis of detection, and the anomaly score corresponding to each point is expressed as:

[0100]

[0101] Among them, x represents the data of the input time window, y represents the time information of the input time window, and z represents the prior distribution of the low-dimensional latent space. Represents the conditional distribution q Φ Find the expectation under the condition of sampling z in (z|x, y), logp θ (x|z, y) represents the logarithm of the probability density of the conditional distribution P(x|z, y). represents the logarithmic probability density of the input x in the reconstructed distribution of the decoder output.

[0102] This anomaly score can be understood as the negative reconstruction probability of the detection data. The larger the reconstruction probability, the less likely the detected curve is to be an anomaly. On the contrary, the smaller the probability is, the less likely it is to be an anomaly. Therefore, the anomaly score is selected as the logarithm of the negative reconstruction probability, where the logarithm is used to amplify the value of the anomaly score when the reconstruction probability is small.

[0103] Specifically, in the embodiment of the present application, the conditional variational autoencoder model should learn the normal pattern with time information in the KPI time series. The model uses an enhanced version based on the ELBO loss function, the M-ELBO loss function, which enables the conditional variational autoencoder to reduce the impact of abnormal points and missing points on the loss function in a training window. Therefore, this enables the model to learn and reconstruct normal patterns from the time window where abnormalities may exist. The M-ELBO loss function is listed as follows:

[0104]

[0105] For an input time window x, assume that the time window of the annotation information corresponding to it is a binary vector α, if and only if x i When it is an abnormal point or missing point, α i is 1. β represents the proportion of normal points in a time window. Since the posterior probability of x is assumed to be a diagonal Gaussian distribution, its log-likelihood can be rewritten as By adding α i and logp θ (x i |z, y) can remove the contribution of abnormal points and missing points to the loss function, and then remove their influence on the training effect. In order to maximize the advantage of the model, random abnormal data injection is also performed during the training process, that is, the data in the time window is randomly changed to 0, and it is changed to 1. This is also data enhancement in a sense. Under the condition of unsupervised learning, the normal mode in the curve can be learned as little as possible without relying on annotations. Figure 5 As shown in the figure, the detailed training steps of Bagel are shown. The entire training consists of two parts: encoder and decoder.

[0106] For the encoder, the main task is to transform the KPI numerical information and time information within the time window into a low-dimensional latent space. Since neural networks can easily overfit the time information, it is necessary to pass the Y-dimensional vector of time information through a random inactivation layer to discard part of the content to achieve better training results. After that, the W-dimensional vector representing the KPI numerical information within the time window and the Y-dimensional vector representing the time information are directly concatenated to form the input data of the encoder. Since the model assumes that the prior probability distribution of the z variable in the latent space is a standard normal distribution, its posterior probability distribution is a diagonal Gaussian distribution. The output of the encoder is two K-dimensional vectors, representing the mean μ of the posterior probability of the z variable. z and standard deviation σ z . We obtain the posterior probability distribution of the z variable in the latent space, and then sample z~N(μ z , σ z ), and get another K-dimensional vector z. It needs to be concatenated with the Y-dimensional vector of time information after the random inactivation layer as the input of the decoder. The decoder will also output two vectors, which have the same shape as the W-dimensional vector representing the KPI numerical information before entering the encoder, representing the mean μ of each point in the reconstructed window. x and standard deviation σ x At this point, the data has completed a forward propagation through the encoder and decoder of the model, and then the model is continuously trained based on the M-ELBO loss function and the SGVB optimization algorithm.

[0107] Specifically, in the embodiment of the present application, the M-ELBO loss function is expressed as:

[0108]

[0109] Among them, x represents the data of the input time window, x i represents a point in the input time window, y represents the time information of the input time window, and z represents the prior distribution of the low-dimensional latent space. Represents the conditional distribution Find the expectation under the condition of sampling z, logp(x i |z, y) represents the logarithm of the probability density of the conditional distribution P(x|z, y), logp(z|y) represents the logarithm of the probability density of the conditional distribution P(z|y), Represents the logarithm of the probability density of the conditional distribution P(z|x, y), α i Indicates whether the i-th point in the time window is an outlier, β indicates whether the i-th point in the time window is an outlier, and W indicates the length of the time window.

[0110] Specifically, in the embodiment of the present application, the anomaly score is used as an input of the extreme value theory to determine the threshold, including:

[0111] 1) Peak Over Threshold (POT) Method

[0112] The Peak Over Threshold (POT) method relies on the Pickands-Balkema-de Haan Theorem (also known as the Second Theorem of Extreme Value Theory) given below.

[0113] The extreme value distribution of the cumulative distribution F approaches the extreme value distribution, as shown in Equation 6.3, if and only if there exists a function, and for all x∈R, when 1+γx>0, t→τ, then:

[0114]

[0115] More clearly, as shown in Equation 6.4, it can be expressed as when 1+γx>0, t→τ, we have:

[0116]

[0117] This result shows that the part exceeding the threshold t (denoted as Xt) may follow a generalized Pareto distribution (GPD) with parameters γ and σ. GPD also requires a third parameter, the location μ, but it is empty in this design. Instead of fitting the extreme values ​​of X to an extreme value distribution (EVD), the POT method attempts to fit Xt to a Pareto distribution.

[0118] If you get and The estimated value of , then the quantile can be calculated by formula 6.5:

[0119]

[0120] Where t is the peak threshold, q is the probability of anomaly, n is the total length of the sequence, and N t is the number of peaks, that is, when X i >X at t i The number of z q It can be regarded as the threshold of abnormality. After using logarithmic maximum likelihood estimation, we need to maximize Equation 6.6.

[0121]

[0122] where Y i Yes X i The excess relative to t, that is, when X i >t,Y i =X i -t.

[0123] 2) Building an anomaly detector based on the POT method

[0124] Extreme value theory provides a method for estimating z through the POT method. q The method makes P(X>z q )<q does not make any assumptions about the distribution of X, nor does it require any clear understanding of the distribution of X. Figure 6 As shown in Figure 1, this method will be used to build a flow outlier detector. The entire detector is divided into two parts: calibration and detection. First, the calibration step is based on the first n observations of the entire sequence X1,...,X n Calculate the anomaly threshold z q Then in the subsequent detection part, z is updated with the incoming data q And use it as the threshold for judging abnormal points.

[0125] a. Initialization Steps

[0126] The whole process of POT algorithm is based on the sequence X1,...,X n , and a certain risk probability q. The goal is to calculate a probability that verifies P(x>z q )<the initial abnormal threshold z of q q The main idea is to set a higher peak threshold t (actually this is an empirical value) to filter out the peak (the part exceeding the peak threshold t), and fit the sequence of the exceeding values ​​to GPD (generalized Pareto distribution). In this way, the distribution of extreme values ​​can be inferred and the abnormal threshold z can be calculated. q .

[0127] b. SPOT determines outliers in the data

[0128] The Stream POT (SPOT) algorithm first obtains the initial peak threshold t and abnormal threshold z through the POT method q Then traverse the remaining sequences, and store two lists, one sequence Y t Used to store the part exceeding the peak threshold (X i -t), another sequence A is used to store abnormal values, that is, the value that exceeds z during the traversal process. q When X i Exceeding the abnormal threshold z q , add it to sequence A; otherwise, if it does not exceed the abnormal threshold but exceeds the peak threshold t, then X i -t join sequence Y t , then the abnormal threshold z will be updated q ; When X i When the peak threshold is not exceeded, continue to traverse backwards. After traversing the remaining sequence, when traversing to each point, the z at that time q That is, the threshold value is dynamically determined for the input sequence.

[0129] Specifically, in the embodiment of the present application, a spatial regression algorithm is used to evaluate the anomaly, including:

[0130] The algorithm uses spatial regression between the experimental group indicator data before the change and the control group indicator data to learn the statistical dependency structure. The structure learns the ability to use one set of data to predict another set of data. The control group data is used to predict the experimental group data, and the predicted values ​​of the experimental group data before and after the change are obtained. The difference between the prediction and the actual value of the experimental group before and after the change is then calculated. Figure 7 Three examples are given of the relative changes between the experimental group and the control group before and after the change.

[0131] The differences between the two were compared using a robust-rank-order-test to see whether the changes resulted in significant changes in the experimental and control groups.

[0132] When selecting the control group, a simple random sampling method is used to select k from N control groups. Comparison of multiple prediction differences enhances the robustness of spatial regression, making the impact of a few indicator changes in the control group less likely to have a negative impact on the prediction. If there are multiple experimental groups to be analyzed, the above steps need to be repeated.

[0133] 1) Training and prediction

[0134] X b , X a is the time series matrix before and after the change of the control group, in which each column represents a time series vector of the control group, Y b (j), Y a (j) represents the time series vector of the jth experimental group. K samples are randomly sampled from N control groups to generate a new control group sampling matrix The linear regression parameter β is learned based on the sampling control group time series matrix and the experimental group time series vector before the change, as shown in Formula 6.7.

[0135]

[0136] Then use the learned parameters and the time series matrix of the sampled control group before and after the change to predict the time series vector of the experimental group before and after the change, as shown in Formula 6.8.

[0137]

[0138] 2) Compare the differences

[0139] First, multiple predicted vectors need to be aggregated, usually using the median method, that is, taking the median of different predicted sequences at the same time to obtain an aggregated predicted sequence. Then, the difference between the time series before and after the actual experimental group change and the aggregated predicted time series is taken, as shown in Formula 6.9.

[0140]

[0141] Then we need to use the robust-rank-order-tests method to compare the sequences. and difference.

[0142] 3) Robustness rank test

[0143] For two sequences, the length of X sequence is m, and the length of Y sequence is n. Sequence U(YX i ) represents the number of elements in the X sequence whose corresponding position is greater than the number of elements in the Y sequence. U(YX) is U(YX i ). Similarly, U(XY i ) Each element represents the number of elements in the Y sequence whose corresponding position is greater than the number of elements in the X sequence. In addition, Equation 6.10 defines two other variables:

[0144]

[0145] The final significant difference score of the robust rank test is shown in formula 6.11:

[0146]

[0147] Regarding judging the significant difference between two sequences, when m and n are both small, the difference can be judged according to the appendix of the table. When m and n are large, the significance score approximately follows the standard normal distribution.

[0148] The solution of this application includes two major parts: drastic change detection and evaluation of whether the anomaly is caused by software changes. The main process is as follows: the input KPI time series is first subjected to a median difference test. If they are all lower than the threshold of the median difference test, then this curve has no anomaly; otherwise, the curve of the original sequence is subjected to unsupervised anomaly detection using the Bagel algorithm to obtain an anomaly score sequence corresponding to each point; the anomaly score is used as the input of the extreme value theory to determine the threshold, and the comparison between the threshold and the anomaly score of the point is used to determine whether the curve has an anomaly. If the curve has an abnormal point, it is necessary to further combine the grayscale release configuration data, that is, the KPI time series of the control group, and make a comprehensive judgment through the spatial regression algorithm. If the confidence values ​​of the experimental group and the control group data are relatively large, that is, there is a significant difference between the two groups of data, then the anomaly of the indicator is caused by the change; otherwise, it is an anomaly caused by other reasons. Figure 8 The flowchart of the algorithm is shown in FIG.

[0149] In summary, the algorithm is the core part of intelligent evaluation of services. This design uses each curve corresponding to each change as the detection granularity. The core of the algorithm is to use an unsupervised anomaly detection algorithm based on machine learning theory to detect single-indicator anomalies. The spatial regression algorithm is used to determine whether the drastic change in the indicator curve is caused by the change. The specific method is to input a period of single-indicator time series data during offline training, and preprocess the data to obtain a normalized two-dimensional data matrix. Then, the model is trained offline to learn the normal mode in the data set and obtain the score of each data. Finally, the abnormal threshold is obtained through extreme value theory. During online detection, the data is preprocessed first, and the trained model is used for data detection. The obtained abnormal score is compared with the threshold to obtain the final abnormal result. If the detection result is abnormal, the spatial regression algorithm is used for evaluation to determine whether the abnormality is caused by software changes.

[0150] The design of this application is to deploy on the change detection platform and provide detection tasks for service changes. In order to deploy the algorithm online and meet the compatibility and applicability of each platform, the algorithm is uploaded to the algorithm platform, and changes are detected at each region, cloud service, component, and microservice level. According to the user's configuration information, the service detection system starts the offline training task on the algorithm platform to complete the training to generate the model, and calls the model service in the change detection system to perform drastic change detection on the data to be detected. According to the detection results and the actual change scenario, a comprehensive judgment of spatial regression is performed to complete the intelligent evaluation. When the drastic change detection and spatial regression results are abnormal during the service deployment change process, the deployment will be suspended and the operation and maintenance personnel will be notified. The drastic change detection task runs through the service change to within two hours after the change, and the time can be set according to user needs, so as to integrate the intelligent change evaluation into the actual service change work. The overall structure diagram is as follows Fig. 9 As shown, including:

[0151] Change deployment system: Change tasks can be initiated, and change decisions can be selected in this system: continue with the change or roll back.

[0152] Change detection system: During the entire design process, the change detection system can initiate offline training tasks and online detection tasks respectively, and feedback the detection results for the services that have changed.

[0153] Monitoring system: used to store 7 days of online data and real-time indicator data.

[0154] Object storage service: used to store offline data sets of change services and model parameters generated by training.

[0155] Algorithm platform: used to store algorithm codes and complete operations from training to outputting models.

[0156] Detection system implementation process:

[0157] 1) The user first clicks a button on the change detection front end to select configuration information such as algorithms, indicators, and training;

[0158] 2) The user changes the service on the change deployment platform. The change deployment platform starts the change task at the beginning of the service change and calls the change detection platform. The change detection platform obtains more detailed information about the service by calling the CMDB platform and the object storage service to obtain the historical data of the service based on the valid parameters selected by the user on the front end. It starts the offline training task for the service and generates a service model. At the same time, the change detection platform starts the online detection task, obtains the online data of the service in real time by calling the monitoring system, and transmits it to the service model to obtain the indicator detection results.

[0159] If the indicator detection results are abnormal, the change detection platform will call the algorithm platform again to use the change assessment algorithm, call the CMDB platform to obtain the information of the machines under the same conditions and the same components as the current service change as the control group, call the object storage platform to obtain the historical data of the experimental group and the control group before the change, and call the monitoring system to obtain the online data of the experimental group and the control group after the change to perform software change assessment to determine whether the abnormality is caused by the software change. The results of the last two judgments are displayed on the front-end interface. For detailed results, users can query the records in the database of the change detection platform.

[0160] This technical solution has been deployed and used on the change detection platform and applied to service change detection. The performance of the above mechanism has been verified using a simulation environment and a fault injection platform. The F1-score on these two data sets reached 95.66% and 97.39% respectively. In the actual service change result feedback, the accuracy rate reached more than 85%, and the recall rate reached more than 90%. It can detect drastic changes in the KPIs of hundreds of components and complete the monitoring of hundreds of thousands of indicators within 15 minutes. It does not require manual parameter adjustment by operation and maintenance personnel, avoids setting specific thresholds for each curve, and greatly improves the efficiency of service change evaluation.

[0161] In order to implement the above embodiment, the present application also proposes a device for evaluating the health status of software after the software is changed by AI.

[0162] Fig.10 A schematic diagram of the structure of a device for evaluating the health status of software after a software change by means of AI provided in an embodiment of the present application.

[0163] like Fig.10 As shown, the device for evaluating the health status of software after the change by AI includes a median difference inspection module, an abnormal score determination module, an abnormal point detection module, and an evaluation module, wherein:

[0164] The median difference test module is used to obtain the KPI time series to be tested and perform a median difference test on the obtained KPI time series;

[0165] The anomaly score determination module is used to perform unsupervised anomaly detection on the KPI time series curve if a data point is detected to be higher than the median difference test threshold, and obtain the anomaly score corresponding to each point;

[0166] The outlier detection module is used to determine the threshold by taking the anomaly score as the input of extreme value theory, and to judge whether the curve has an anomaly point by comparing the threshold with the anomaly score of each point on the curve;

[0167] The evaluation module is used to evaluate the curve using a spatial regression algorithm if there are abnormal points on the curve to determine the type of abnormality.

[0168] It should be noted that the above explanation of the method embodiment for evaluating the health status of software after a change by means of AI is also applicable to the device of this embodiment for performing anomaly detection and change evaluation on the health status after an actual software change scenario by means of AI, and will not be repeated here.

[0169] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0170] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0171] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0172] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0173] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0174] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0176] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for evaluating the health status of software after a change by means of AI, characterized in that: The following steps are involved: Obtain the KPI time series to be tested, and perform a median difference test on the obtained KPI time series; If a data point is detected to be higher than the threshold of the median difference test, unsupervised anomaly detection is performed on the curve of the KPI time series to obtain an anomaly score corresponding to each point; The anomaly score is used as the input of extreme value theory to determine the threshold, and the curve is judged whether there is an abnormal point based on the comparison between the threshold and the anomaly score of each point on the curve; If the curve has abnormal points, use the spatial regression algorithm to evaluate the curve and determine the type of abnormality; The median difference test on the acquired KPI time series includes: Calculate the median difference coefficient of each point on the curve of the KPI time series, wherein the median difference coefficient of each point is obtained by subtracting the median of a sequence before the point and the median of a sequence after the point and calculating the absolute value; A threshold is set to be compared with the median difference coefficient at each point on the curve.

2. The method according to claim 1, characterized in that The performing unsupervised anomaly detection on the curve of the KPI time series includes: Performing data preprocessing on the KPI time series to generate time window data; Generate reconstructed data based on the distribution of other non-missing point data in the time window data by using the MCMC method, and use the reconstructed data to replace the corresponding missing data; Input the padded time window data into the trained conditional variational autoencoder and output the anomaly score corresponding to each point. Wherein, the data preprocessing includes: Fill missing points in the KPI time series data with 0 and make the intervals of the time series data the same; Normalize the time series after padding.

3. The method according to claim 2, characterized in that The anomaly score corresponding to each point is expressed as: in, represents the data of the input time window, Represents the time information of the input time window, represents the prior distribution of the low-dimensional latent space, Represents the conditional distribution Medium Sampling Under the condition of seeking expectation, Represents the logarithm of the probability density of the conditional distribution P(x|z,y), represents the logarithmic probability density of the input x in the reconstructed distribution of the decoder output.

4. The method according to claim 3, characterized in that Training the conditional variational autoencoder model includes: Obtain a normal KPI time series, perform data preprocessing on the normal KPI time series, and generate time window data to form a training data set; Using the training data set, the model is continuously trained based on the M-ELBO loss function and the SGVB optimization algorithm until the model converges and the training is completed.

5. The method according to claim 4, characterized in that The M-ELBO loss function is expressed as: in, represents the data of the input time window, represents a point in the input time window, Represents the time information of the input time window, represents the prior distribution of the low-dimensional latent space, Represents the conditional distribution Medium Sampling Under the condition of seeking expectation, Represents the logarithm of the probability density of the conditional distribution P(x|z,y), The logarithm of the probability density representing the conditional distribution P(z|y) is The logarithm of the probability density representing the conditional distribution P(z|x,y) is obtained. Indicates whether the i-th point in the time window is an abnormal point. Represents the proportion of normal points in a time window, Indicates the length of the time window.

6. The method according to claim 1, characterized in that The method of using the anomaly score as an input of extreme value theory to determine the threshold includes: Setting a peak threshold to filter out the peak value in the time series before the point to be detected, wherein the peak value in the sequence is the part exceeding the peak threshold; Calculate the initial abnormal threshold value according to the sequence data, wherein the sequence data includes the total length of the sequence, the peak threshold value, the number of peaks, and the risk probability, and the risk probability is the hyperparameter to be adjusted; The initial abnormality threshold is updated using the data of the KPI time series to be detected to obtain a final threshold.

7. The method according to claim 1, characterized in that The use of a spatial regression algorithm to evaluate anomalies includes: Step S1: A sampling control group is simply randomly selected from the control group, wherein the control group is a related indicator in a machine that has the same service deployed as the machine with the software change and has not undergone the software change, and the sampling control group data includes a time series matrix before and after the change; Step S2: learning linear regression parameters based on the time series matrix of the control group before the change and the time series vector of the experimental group to be tested before the change; Step S3: Use the learned parameters and the time series matrix of the control group before and after the change to predict the time series vector to be tested before and after the change; Step S4: Repeat steps S1, S2, and S3 to obtain a predicted time series vector group; Step S5: Calculate the median of the values ​​of different prediction time vectors at the same time point for the prediction time series vector group to obtain a new prediction vector, and then make a difference between the time series before and after the change to be tested and the new prediction vector to obtain a difference sequence before and after the change; Step S6: Use the robust rank test to compare the difference sequence before and after the change to obtain a significant difference score, where for an X sequence of length m and a Y sequence of length n, the sequence Each element in represents the number of elements in the X sequence whose corresponding position is greater than the number of elements in the Y sequence. for The mean of the series Each element in represents the number of elements in the Y sequence whose corresponding position is greater than the number of elements in the X sequence. for The mean of the X and Y sequences is: ， ； Step S7: setting a threshold value and comparing it with the significant difference score; if it exceeds the threshold value, determining that the abnormality is caused by software change; otherwise, determining that the abnormality is caused by other reasons.

8. The method according to claim 1, characterized in that Perform a median difference test on the acquired KPI time series to determine whether all data points in the series are below the median difference test threshold; or compare the threshold with the anomaly score of each point on the curve to determine that the curve does not contain anomalies, thus confirming that the software is in a healthy state after the change.

9. A device for evaluating the health status of software after a software change by means of AI, characterized in that: It includes median difference test module, abnormal score determination module, abnormal point detection module and evaluation module, among which: The median difference test module is used to obtain the KPI time series to be tested and perform a median difference test on the obtained KPI time series; The anomaly score determination module is used to perform unsupervised anomaly detection on the curve of the KPI time series to obtain an anomaly score corresponding to each point if a data point is detected to be higher than a threshold of the median difference test; The outlier detection module is used to use the anomaly score as an input of extreme value theory to determine a threshold, and to determine whether the curve has an outlier based on a comparison between the threshold and the anomaly score of each point on the curve; The evaluation module is used to evaluate the curve using a spatial regression algorithm to determine the type of abnormality if there are abnormal points on the curve; The median difference test on the acquired KPI time series includes: Calculate the median difference coefficient of each point on the curve of the KPI time series, wherein the median difference coefficient of each point is obtained by subtracting the median of a sequence before the point and the median of a sequence after the point and calculating the absolute value; A threshold is set to be compared with the median difference coefficient at each point on the curve.

Images