Application of anomaly detection methods, devices, equipment and media

Abstract

This disclosure provides an application anomaly detection method, relating to the fields of artificial intelligence, information security, fintech, or other related fields. The method includes: acquiring application indicator data of an application at a specified time point, the application indicator data including time-series data generated during the application's operation; inputting the application indicator data to be detected into an anomaly detection model to obtain a first anomaly result, the first anomaly result including an application anomaly result or an application normal result; issuing an alarm signal when the first anomaly result is the application anomaly result; the anomaly detection model is pre-trained X times using unlabeled historical indicator data as training samples, wherein at least one training iteration includes obtaining labels using the uncertainty value of a portion of the unlabeled historical indicator data and performing supervised training. This disclosure also provides an application anomaly detection device, equipment, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence, information security, fintech, or other related fields, and more specifically, to an application of anomaly detection methods, apparatus, equipment, media, and program products. Background Technology

[0002] During application operation, abnormal behaviors may occur, affecting the availability and reliability of application services. Related technologies can utilize artificial intelligence algorithms to develop anomaly detection models for detecting abnormal behavior. These models typically involve steps such as acquiring historical application metrics data, data preprocessing, manual labeling, model building, model training, model deployment, and implementing anomaly detection in a production environment.

[0003] In the process of realizing the inventive concept disclosed herein, the inventors discovered that the amount of historical indicator data is large, and manual labeling consumes a lot of time and economic costs, resulting in an excessively long training phase. Consequently, the overall time required for abnormal behavior detection increases, making it difficult to efficiently deploy the model and quickly perform abnormal behavior detection in the production environment. Summary of the Invention

[0004] In view of the above problems, this disclosure provides methods, apparatus, equipment, media and program products for anomaly detection.

[0005] One aspect of this disclosure provides an application anomaly detection method, comprising: acquiring application indicator data of an application at a time point to be detected, the application indicator data including time series data generated during the operation of the application; inputting the application indicator data to be detected into an anomaly detection model to obtain a first anomaly result, the first anomaly result including an application anomaly result or an application normal result; issuing an alarm signal when the first anomaly result is the application anomaly result; the anomaly detection model is obtained by pre-training unlabeled historical indicator data as training samples X times, wherein at least one training includes obtaining labels using the uncertainty value of some unlabeled historical indicator data and performing supervised training, the uncertainty value characterizing the degree of uncertainty of the corresponding unlabeled historical indicator data, and X being greater than or equal to 1.

[0006] According to an embodiment of this disclosure, before inputting the application indicator data to be detected into the anomaly detection model, the method further includes: determining first sub-indicator data within a predetermined time period before the time point to be detected; determining Q second sub-indicator data within Q historical time periods, wherein any two time periods in the Q historical time periods and the predetermined time period belong to the same time period position on different dates, and Q is an integer greater than or equal to 1; sorting the first sub-indicator data and the Q second sub-indicator data in chronological order to obtain the application indicator data to be detected.

[0007] According to an embodiment of this disclosure, the application is deployed on multiple servers, and the application services running between any two servers are different. Each server has application indicator data to be detected. The step of inputting the application indicator data to be detected into an anomaly detection model to obtain a first anomaly result includes: concatenating the application indicator data to be detected for each of the multiple servers according to the calling order of the application services running on each of the multiple servers to obtain model input data; and processing the model input data using the anomaly detection model to obtain the first anomaly result.

[0008] According to embodiments of this disclosure, the step of obtaining labels and performing supervised training using the uncertainty value of some unlabeled historical indicator data includes: obtaining a second anomaly result for each of S unlabeled first historical indicator data, the second anomaly result including an application anomaly probability value and an application normal probability value, where S is an integer greater than or equal to 1; obtaining an uncertainty value based on the degree of difference between the application anomaly probability value and the application normal probability value of each first historical indicator data; obtaining a label for the first historical indicator data whose uncertainty value is greater than a preset threshold, the label including an application anomaly label or an application normal label; and supervising the training of the anomaly detection model based on labeled first historical indicator data.

[0009] According to embodiments of this disclosure, after supervised training of the anomaly detection model based on labeled first historical indicator data, the step of using unlabeled historical indicator data as training samples for X training iterations further includes: collecting K unlabeled second historical indicator data, where K is an integer greater than or equal to 1; calculating the similarity between each second historical indicator data and each labeled first historical indicator data for each second historical indicator data; obtaining the label of each second historical indicator data based on the labeled first historical indicator data with the highest similarity; and continuing to train the anomaly detection model based on the labeled second historical indicator data.

[0010] According to an embodiment of this disclosure, before obtaining the second abnormal result for each of the S unlabeled first historical indicator data, the step of using the unlabeled historical indicator data as training samples for X training iterations further includes: clustering the N unlabeled first historical indicator data to obtain M clusters, where N is an integer greater than or equal to 2, M is an integer less than N, and N is greater than S; for each cluster, at least one first historical indicator data is extracted, and the S first historical indicator data includes all the first historical indicator data extracted from the M clusters.

[0011] According to embodiments of this disclosure, extracting at least one first historical indicator data for each cluster includes: randomly extracting a portion of the first historical indicator data from each cluster; and / or obtaining a label for the first historical indicator data whose uncertainty value is greater than a preset threshold includes: determining the label based on the cluster to which the first historical indicator data whose uncertainty value is greater than the preset threshold belongs.

[0012] According to an embodiment of this disclosure, obtaining the second abnormal result of each of the S unlabeled first historical indicator data includes: processing the S first historical indicator data using a classification model to obtain the second abnormal result of each of them, wherein the classification model is pre-trained using labeled third historical indicator data.

[0013] According to an embodiment of this disclosure, obtaining the uncertainty value based on the degree of difference between the application anomaly probability value and the application normal probability value of each first historical indicator data includes: calculating the absolute value of the difference between the application anomaly probability value and the application normal probability value; and taking the negative of the absolute value of the difference as the uncertainty value.

[0014] Another aspect of this disclosure provides an application anomaly detection device, comprising: a data acquisition module for acquiring application indicator data of an application at a time point to be detected, the application indicator data including time series data generated during the operation of the application; a data input module for inputting the application indicator data to be detected into an anomaly detection model to obtain a first anomaly result, the first anomaly result including an application anomaly result or an application normal result; and an alarm module for issuing an alarm signal when the first anomaly result is the application anomaly result; the anomaly detection model is obtained by pre-training unlabeled historical indicator data as training samples X times, wherein at least one training includes obtaining labels using the uncertainty value of a portion of the unlabeled historical indicator data and performing supervised training, the uncertainty value characterizing the degree of uncertainty of the corresponding unlabeled historical indicator data, and X being greater than or equal to 1.

[0015] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0016] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0017] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0018] The above one or more embodiments have the following beneficial effects: By acquiring time-series data generated during application operation at the point to be detected as input to the anomaly detection model, and issuing timely alarm signals upon detecting anomalies, the speed of application anomaly detection can be improved. Furthermore, by pre-training unlabeled historical indicator data as training samples X times, and obtaining labels for the historical indicator data based on uncertainty values ​​for supervised training, the anomaly detection model can be obtained and finally deployed in actual anomaly detection scenarios. This allows for the filtering and reduction of sample labeling based on uncertainty values, further obtaining labels and conducting supervised training, enabling rapid anomaly behavior detection in the production environment, and timely issuance of alarm signals upon anomaly detection. Attached Figure Description

[0019] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 This diagram illustrates an application scenario of the anomaly detection method according to an embodiment of the present disclosure.

[0021] Figure 2 A flowchart illustrating supervised training according to an embodiment of the present disclosure is shown schematically;

[0022] Figure 3 A flowchart illustrating the continued training of the model according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 4 A flowchart illustrating the process of obtaining application metric data to be detected according to an embodiment of the present disclosure is shown in the schematic diagram.

[0024] Figure 5 A flowchart illustrating the process of obtaining a first abnormal result according to an embodiment of the present disclosure is shown schematically;

[0025] Figure 6 A flowchart illustrating a clustering process according to an embodiment of the present disclosure is shown schematically;

[0026] Figure 7 A flowchart illustrating an application anomaly detection method according to another embodiment of this disclosure is shown schematically;

[0027] Figure 8A and Figure 8B A schematic diagram illustrating the structure of an application anomaly detection device according to an embodiment of the present disclosure is shown; and

[0028] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing an application anomaly detection method according to an embodiment of the present disclosure. Detailed Implementation

[0029] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0032] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0033] Taking cloud applications as an example, cloud applications may exhibit abnormal behaviors that affect service availability and reliability, and even cause economic losses. Abnormal behaviors may be caused by disk failures, memory leaks, network fluctuations, etc., and are reflected in changes in performance metrics. Furthermore, the massive amounts of monitoring data generated by cloud applications mean that a deviation in one component can become an anomaly, causing the entire system to fail. Due to the complexity of cloud systems, the data characteristics of the same metric differ across applications and scenarios, making timely and accurate anomaly detection challenging in the context of big data.

[0034] For example, in machine learning-based anomaly detection algorithms, unsupervised learning can be used for model training. While unsupervised learning doesn't require labeled data and is less costly, the presence of noisy data leads to a higher false positive rate, necessitating management oversight of the application's status. Supervised learning, on the other hand, requires labeled data, involving the annotation of massive datasets, resulting in significant costs. Therefore, issues such as inaccurate data, lack of labeled data, and imbalanced sample classes limit the efficiency of AI anomaly detection models in production environments.

[0035] Some embodiments of this disclosure provide an application anomaly detection method. This method acquires time-series data generated during application operation at the point in time to be detected, using it as input to an anomaly detection model. Upon detecting an anomaly, it promptly issues an alarm signal, thereby improving the speed of application anomaly detection. Furthermore, unlabeled historical indicator data is used as training samples for X training iterations. Labels are obtained from the historical indicator data based on uncertainty values ​​for supervised training to obtain the anomaly detection model, which is then deployed to a real-world anomaly detection scenario. This allows for the filtering and reduction of sample labeling based on uncertainty values. Further labeling and supervised training enable rapid anomaly behavior detection in the production environment, facilitating timely alarm signal issuance upon anomaly detection.

[0036] Anomaly detection aims to identify data objects that deviate from expectations, thus enabling timely and effective identification of abnormal application behavior based on collected application performance metrics. Once an anomaly is identified, it can promptly notify support managers and display the anomaly metrics, improving emergency response timeliness.

[0037] Figure 1 The illustration schematically depicts an application scenario of the anomaly detection method according to an embodiment of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples to illustrate the application of the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure. However, they do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios.

[0038] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0041] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0042] Server 105 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, network services, and middleware services.

[0043] In applications such as client applications and web applications (abbreviated as APP), the client (i.e., the front-end) and the server (i.e., the back-end) can communicate data via network packets. For example, in the APP client, the parameters that the server needs to obtain from the client are assembled and sent to the server by calling a network request method. It should be noted that the application anomaly detection method provided in this embodiment of the invention can be executed by terminal devices 101, 102, 103 or server 105. Correspondingly, the application anomaly detection device provided in this embodiment of the invention can generally be set in terminal devices 101, 102, 103 or server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0045] The following will be based on Figure 1 The described scene, through Figures 2-7 The application anomaly detection method of this disclosure will be described in detail.

[0046] In some embodiments, the application anomaly detection method includes: acquiring application indicator data of the application at the time point to be detected, the application indicator data including time series data generated during the application's operation; inputting the application indicator data to be detected into an anomaly detection model to obtain a first anomaly result, the first anomaly result including an application anomaly result or an application normal result; issuing an alarm signal when the first anomaly result is an application anomaly result; the anomaly detection model is obtained by pre-training unlabeled historical indicator data as training samples X times, wherein at least one training includes obtaining labels using the uncertainty value of some unlabeled historical indicator data and performing supervised training, the uncertainty value characterizing the degree of uncertainty of the corresponding unlabeled historical indicator data, and X being greater than or equal to 1.

[0047] For example, application metric data can come from one or more applications. When it comes from a single application, the metric data from that application can be used to determine if any anomalies have occurred. When it comes from multiple applications, the interaction between the metric data from these applications can be used to collaboratively determine if any applications are experiencing anomalies.

[0048] When an application malfunctions, such as encountering problems or failures (e.g., network failures, data center failures, software vulnerabilities), performance metrics will also show corresponding abnormal changes (e.g., sudden increases or decreases). For example, the application metric data or historical metric data involved in this disclosure embodiment may include one or more performance metrics such as user access volume, query request volume, query success volume, CPU utilization, storage utilization, and network resource utilization.

[0049] Before performing anomaly detection, an anomaly detection model can be pre-built, which can be a neural network model based on deep learning. This model can then perform anomaly detection based on application metric data. Therefore, the anomaly detection model can learn the relationships between different performance metrics, and the correlation characteristics of the trend changes of one or more performance metrics over time.

[0050] For example, the neural network model may include any one or more combinations of Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory (LSTM), but is not limited to these. It may also be other network models, such as fully connected neural networks (DNN). Those skilled in the art can design the network structure of the anomaly detection model according to actual needs.

[0051] If an application encounters an anomaly, the first abnormal result will be considered an application abnormal result; otherwise, a normal result will be considered an application normal result, indicated by the message "Application Abnormal" or "Application Normal". When an application encounters an abnormal result, the type of anomaly can also be specified. Anomaly types may include, but are not limited to, the following: system anomalies, anomalies triggered by network attacks, database loss, and data report errors.

[0052] According to embodiments of this disclosure, time-series data generated during application operation is acquired at the point to be detected and used as input to the anomaly detection model. When an anomaly is detected, an alarm signal is issued promptly, improving the speed of application anomaly detection. Furthermore, during the training phase, uncertainty values ​​can be used to filter and reduce the number of labeled samples. Further labeling and supervised training enable rapid anomaly behavior detection in the production environment, allowing for timely alarm signal issuance upon anomaly detection.

[0053] Figure 2 A flowchart illustrating supervised training according to an embodiment of this disclosure is shown schematically.

[0054] like Figure 2 As shown, referring to model training method 200, the anomaly detection model is configured to be pre-trained through the following operations:

[0055] In operation S210, the second anomaly results of each of the S unlabeled first historical indicator data are obtained. The second anomaly results include the applied anomaly probability value and the applied normal probability value, where S is an integer greater than or equal to 1.

[0056] In some embodiments, the first abnormal result can be obtained through manual annotation, or it can be determined through log information corresponding to each first historical indicator data.

[0057] In some embodiments, a classification model can be used to process S first historical indicator data to obtain their respective second anomaly results. The classification model is pre-trained using labeled third historical indicator data.

[0058] The classification model can be built based on the same or different artificial intelligence algorithms as the anomaly detection model. The third historical indicator data can include manually labeled historical data. For example, the classification model can be built based on at least one of the following algorithms: Recurrent Neural Network (RNN), Isolation Forest, Exponentially Weighted Moving-Average (EWMA), and Random Forest.

[0059] Model training method 200 incorporates a query strategy through active learning to select appropriate samples for labeling. For example, a classification model built on a random forest algorithm is used as the base classifier to execute the query strategy during the active learning process, calculating the probability values ​​of data points as normal or abnormal. Based on these probability values, the uncertainty value of the data points is calculated, improving the accuracy of the query strategy.

[0060] In operation S220, the uncertainty value is obtained based on the degree of difference between the application anomaly probability value and the application normal probability value of each first historical indicator data. The degree of difference can be characterized by methods such as difference, percentage, squared difference, standard deviation, etc. In some embodiments, the degree of difference can be directly used as the uncertainty value.

[0061] Uncertainty values ​​measure the degree of uncertainty regarding a given historical indicator. For example, standard deviation can be used to represent the uncertainty of a prediction, with higher values ​​indicating greater uncertainty.

[0062] In some embodiments, obtaining the uncertainty value based on the degree of difference between the application anomaly probability value and the application normal probability value for each first historical indicator data includes: calculating the absolute value of the difference between the application anomaly probability value and the application normal probability value. The negative of the absolute value of the difference is taken as the uncertainty value.

[0063] For example, the uncertainty value Uncertain can be obtained using the following formula 1, where Prob abnormal The value representing the probability of an anomaly in the current data:

[0064] Uncertain = -|(1-Prob) abnormal )-Prob abnormal | Formula 1

[0065] According to embodiments of this disclosure, a larger absolute value of the difference indicates that the model's prediction result is more biased, meaning the classification model has obtained enough information to directly use the anomaly results as labels. Conversely, a smaller absolute value of the difference indicates that the classification model cannot obtain enough information, thus requiring a dedicated label acquisition operation. By labeling a small amount of diverse data with specific characteristics, the efficiency of subsequent training and the accuracy of the anomaly detection model are improved.

[0066] In operation S230, the labels of the first historical indicator data with an uncertainty value greater than a preset threshold are obtained. The labels include application abnormal labels or application normal labels.

[0067] The preset threshold can be flexibly set according to the data processing capability, for example, it can be set to -0.1, which is just an example.

[0068] For example, if the uncertainty value is higher than a preset threshold, the data is considered to have a large amount of information, and experts will manually label the data. Alternatively, it can be connected to a dedicated automated labeling platform for automatic labeling.

[0069] In operation S240, the anomaly detection model is trained under supervision based on labeled first historical indicator data.

[0070] According to the embodiments of this disclosure, the first unlabeled historical indicator data is first classified to obtain application anomaly probability value and application normal probability value, and then the uncertainty value is obtained based on the degree of difference between the two. Then, the label of the first historical indicator data with uncertainty value greater than a preset threshold is obtained, thereby realizing the effect of actively learning the training availability of the first historical indicator data and improving the accuracy of training data and training efficiency.

[0071] Figure 3 A flowchart illustrating the continued training of the model according to an embodiment of the present disclosure is shown schematically.

[0072] After training the anomaly detection model based on labeled first historical indicator data, such as Figure 3 As shown, this embodiment includes:

[0073] In operation S310, collect K unlabeled second historical indicator data, where K is an integer greater than or equal to 1.

[0074] In operation S320, for each second historical indicator data, the similarity between it and each labeled first historical indicator data is calculated.

[0075] For example, historical indicator data may be a matrix composed of one or more performance indicators, and similarity calculation can be performed using methods such as Euclidean distance or Manhattan distance. In some embodiments, different performance indicators can be assigned corresponding weights. The weight allocation method can be linear allocation or softmax, etc.

[0076] In some embodiments, based on the Pearson similarity calculation formula, the unlabeled data (K unlabeled second historical indicator data) X and the first historical indicator data Y with an uncertainty value higher than a preset threshold are similarly calculated, as shown in Formula 2, where cov(X, Y) is the covariance of X and Y, and σ is the standard deviation.

[0077]

[0078] In operation S330, the label of each second historical indicator data is obtained based on the first historical indicator data with the highest similarity.

[0079] If the unlabeled data has the highest similarity to one of the labeled first historical indicator data, and the similarity is greater than a certain value (such as 0.9), then the label is directly obtained, thereby reducing the cost of expert annotation and also reducing steps such as clustering processing and running query strategies.

[0080] While operating S340, the anomaly detection model continues to be trained based on labeled second historical indicator data.

[0081] In real-world scenarios, the S unlabeled first historical indicator data points may represent only a portion of the data, with the remaining unlabeled data being continuously input. Reprocessing the substandard data to obtain second anomaly results and uncertainty values ​​would reduce efficiency and generate redundant information.

[0082] Therefore, K second historical indicator data points are selected from the unlabeled set once or multiple times. Following the similarity calculation steps described above, these second historical indicator data points are labeled, and then added to the training set. This process is used to train the model completed in operation S240. After a predetermined number of iterations, the final detection model is obtained. This reduces the amount of sample processing and computational resources, improving training efficiency.

[0083] In some embodiments, the K second historical indicator data can come from the application indicator data to be detected mentioned above. That is, the anomaly detection model can be continuously trained iteratively based on the time series data generated during the operation of the application, so as to be able to detect newly emerging anomalies and improve accuracy.

[0084] Figure 4 A flowchart illustrating the process of obtaining application metric data to be detected according to an embodiment of this disclosure is shown.

[0085] Before inputting the application metric data to be detected into the anomaly detection model, such as Figure 4 As shown, the application index data to be detected obtained in this embodiment includes:

[0086] In operation S410, the first sub-indicator data within a predetermined time period before the time point to be detected is determined.

[0087] In operation S420, determine Q second sub-indicator data within Q historical time periods. The Q historical time periods and the predetermined time period belong to the same time position within different numbers of days, where Q is an integer greater than or equal to 1.

[0088] Sub-metric data includes one or more performance metrics such as user visits, query request volume, successful query volume, CPU utilization, storage utilization, and network resource utilization. Data within a specified time window (i.e., the length of the time period) can be selected as the sub-metric data to be monitored. The length of the time period can be determined based on monitoring needs, data periodicity, and other factors.

[0089] In operation S430, the first sub-indicator data and Q second sub-indicator data are sorted according to time sequence to obtain the application indicator data to be tested.

[0090] For example, provide data points at fixed time intervals for the current day, as well as data points from yesterday, the day before yesterday, and last week for the corresponding time periods, forming a time series. For a single server, it is necessary to provide data from the current 3 hours, typically 180 points, and 180 points from yesterday, the day before yesterday, three days ago, and seven days ago for the corresponding time periods, totaling 900 data points.

[0091] According to embodiments of this disclosure, routine monitoring can quickly detect application anomalies using application indicator data with time-series information, thereby enabling better resource scheduling, anomaly repair, and other tasks, providing users with a stable user experience. The anomaly detection model can be used to perform time-series prediction based on the time-series information of application indicator data. Time-series prediction refers to prediction based on the distribution patterns of data over time.

[0092] Figure 5 A flowchart illustrating the process of obtaining a first abnormal result according to an embodiment of the present disclosure is shown.

[0093] The application is deployed on multiple servers, and the application services running on any two servers are different. Each server has application metric data to be monitored.

[0094] like Figure 5 As shown, this embodiment is one example of operating S110, including:

[0095] When operating the S510, the application metric data to be detected from each server is concatenated according to the calling order of the application services running on each server to obtain the model input data. The application metric data from multiple servers are then integrated so that the subsequent anomaly detection model can analyze the overall system operation.

[0096] Applications are installed and run on multiple servers to achieve load balancing, high availability, and other objectives. An application service refers to a specific function or service of an application running on a server. The call order refers to the relationship between application services called by multiple servers; that is, an application service on one server may call an application service on another server.

[0097] When operating the S520, the anomaly detection model is used to process the model input data to obtain the first anomaly result.

[0098] For example, when an application triggers an anomaly detection, it sends a request to the monitoring system to obtain 900 data points from α application servers, ultimately resulting in a set of 900*α data points.

[0099] According to embodiments of this disclosure, application indicator data from multiple servers can be integrated and anomaly detected, thereby enabling timely discovery and handling of potential problems. By concatenating the application indicator data from multiple servers to form model input data, and processing it using an anomaly detection model, comprehensive monitoring and anomaly detection of the entire system can be achieved, improving the system's reliability and stability.

[0100] Figure 6 A flowchart illustrating a clustering process according to an embodiment of the present disclosure is shown schematically.

[0101] Before obtaining the second outlier results for each of the S unlabeled first historical indicator data, such as Figure 6 As shown, this embodiment includes:

[0102] In operation S610, the N unlabeled first historical indicator data are clustered to obtain M clusters, where N is an integer greater than or equal to 2, M is an integer less than N, and N is greater than S.

[0103] For example, in a dataset, clustering algorithms can be used to divide the first historical index data into different clusters, with the first historical index data within each cluster having similar characteristics.

[0104] In operation S620, for each cluster, at least one first historical indicator data is extracted, and the S first historical indicator data include all the first historical indicator data extracted from the M clusters.

[0105] For example, applications typically generate more normal data than abnormal data. To improve detection efficiency, the DBSCAN algorithm is used. Input parameters include: the original dataset, the neighborhood radius defined by Eps (density), and the threshold defined by MmPtS (core point). The original data points are divided into three categories: if an object contains more than MmPtS points within its radius Eps, it is a core point. If an object contains fewer than MmPtS points within its radius Eps, but falls within the neighborhood of a core point, it is a boundary point. If an object is neither a core point nor a boundary point, it is a noise point.

[0106] The specific process is as follows: Select data points p (i.e., the first historical indicator data) sequentially from the dataset. If, for parameters Eps and MinPts, the selected data point p is a core point, then find all data points densityably reachable from p, forming a cluster. If the selected data point p is an edge point, select another data point. Repeat the above selection steps until all data points have been processed, ultimately obtaining the partitioned clusters.

[0107] In some embodiments, for each cluster, retrieving at least one first historical indicator data includes: randomly retrieving a portion of the first historical indicator data from each cluster. And / or, obtaining labels for first historical indicator data with uncertainties greater than a preset threshold includes: determining labels based on the cluster to which the first historical indicator data with uncertainties greater than the preset threshold belongs.

[0108] Data of the same batch size are selected from different clusters and used as input to the pooling-based active learning algorithm. Based on the clustering results in the data preprocessing step, data points of the same batch size are randomly selected from each cluster and input into the selection strategy of the active learning algorithm, thereby reducing the number of computations.

[0109] Associating labels with their respective clusters improves the efficiency of automatic or manual labeling, which is helpful for subsequent data processing and analysis, allowing for special handling or focused attention on data points with significant uncertainty.

[0110] According to embodiments of this disclosure, the accuracy of training samples can be improved. By obtaining at least one first historical indicator data from each cluster and randomly selecting a portion of the data, the representativeness of the data samples is ensured, enabling more accurate data analysis and processing. Furthermore, by sampling data based on clustering results, the amount of data requiring computation and processing is reduced, lowering computational complexity and resource consumption, thereby reducing computational load. Simultaneously, by setting an uncertainty threshold and determining the labels for data points with significant uncertainty, it is possible to identify data points requiring special processing or focused attention, reducing the number of labels and improving training efficiency.

[0111] Figure 7 A flowchart illustrating an application anomaly detection method according to another embodiment of this disclosure is shown schematically.

[0112] Taking an anomaly detection scenario in cloud applications as an example, the performance metric data is U, and the test set is T. For example... Figure 7 As shown, the application anomaly detection method in this embodiment includes:

[0113] In the S710 operation, unlabeled samples are added to the performance index data U. The DBSCAN algorithm is used to cluster U, and the same number of data points are selected from each cluster and labeled to construct a labeled sample set L1. A random forest classifier is trained using the labeled sample set L1.

[0114] When operating the S720, the trained random forest classifier is used to perform the query strategy described above on some unlabeled samples in the performance index data U, resulting in uncertainty, which is then sorted in descending order. Then, experts label the valuable samples at the top of the sorted list, resulting in the labeled sample set L2.

[0115] For other samples that do not meet the performance index data U, the sorted samples and labeled samples are calculated based on the Pearson correlation coefficient. Based on the similarity results, it is considered whether to select the data to be handed over to experts for labeling, resulting in the labeled sample set L3. The similarity is calculated in batches, and the labeled sample set Li is obtained by sequentially increasing the similarity.

[0116] When operating the S730, the model is trained iteratively multiple times using each labeled sample set Li. This can be done through transfer learning training based on previous training, or by retraining.

[0117] The final model is obtained by operating the S740.

[0118] According to embodiments of this disclosure, the active learning algorithm significantly reduces the number of labeled samples compared to supervised learning, thus saving costs. Compared to unsupervised learning, the model is less affected by outlier data.

[0119] Based on the above-described application anomaly detection method, this disclosure also provides an application anomaly detection device. The following will be combined with... Figure 8A The device is described in detail in Figure B.

[0120] Figure 8A Figure B schematically illustrates a structural block diagram of an application anomaly detection device according to an embodiment of the present disclosure.

[0121] like Figure 8A As shown, the application anomaly detection device 800 of this embodiment includes a data acquisition module 810, a data input module 820, and an alarm module 830.

[0122] The data acquisition module 810 is used to acquire application indicator data of the application at the time point to be detected. The application indicator data includes time series data generated during the operation of the application.

[0123] The data input module 820 is used to input the application indicator data to be detected into the anomaly detection model to obtain the first anomaly result, which includes the application anomaly result or the application normal result.

[0124] In some embodiments, the data input module 820 can perform operations S510 to S520, which will not be described in detail here.

[0125] The alarm module 830 is used to issue an alarm signal when the first abnormal result is an application abnormal result;

[0126] The anomaly detection model is pre-trained X times using unlabeled historical indicator data as training samples. At least one training session includes obtaining labels using the uncertainty value of some unlabeled historical indicator data and performing supervised training. The uncertainty value represents the degree of uncertainty of the corresponding unlabeled historical indicator data, and X is greater than or equal to 1.

[0127] In other embodiments, the anomaly detection device 800 may further include a model training module 840. The model training module 840 includes a history classification unit 841, an uncertainty value calculation unit 842, a label annotation unit 843, and a model training unit 844. The anomaly detection model is configured to be pre-trained by the model training module 840.

[0128] The historical classification unit 841 can perform operation S210 to obtain the second anomaly result for each of the S unlabeled first historical indicator data. The second anomaly result includes an applied anomaly probability value and an applied normal probability value, where S is an integer greater than or equal to 1.

[0129] The uncertainty value calculation unit 842 can perform operation S220 to obtain the uncertainty value based on the degree of difference between the application abnormal probability value and the application normal probability value of each first historical index data.

[0130] The labeling unit 843 can perform operation S230 to obtain labels for the first historical indicator data whose uncertainty value is greater than a preset threshold. The labels include application abnormal labels or application normal labels.

[0131] In some embodiments, the labeling unit 843 may perform operations S610 to S620, which will not be described in detail here.

[0132] The model training unit 844 can perform operation S240 to train an anomaly detection model based on labeled first historical indicator data.

[0133] In some embodiments, the model training unit 844 may also perform operations S310 to S340, which will not be described in detail here.

[0134] In some embodiments, the application anomaly detection device 800 may further include a data determination module, which can perform operations S410 to S430, which will not be described in detail here.

[0135] The application anomaly detection device 800 includes components for performing the above-mentioned functions. Figures 2-7 The modules of each step in any of the described embodiments.

[0136] It should be noted that the implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the device embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of each corresponding step in the method embodiments, and will not be repeated here.

[0137] According to embodiments of this disclosure, any multiple modules among the data acquisition module 810, data input module 820, alarm module 830, and model training module 840 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module.

[0138] According to embodiments of this disclosure, at least one of the data acquisition module 810, data input module 820, alarm module 830, and model training module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the data acquisition module 810, data input module 820, alarm module 830, and model training module 840 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0139] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing an application anomaly detection method according to an embodiment of the present disclosure.

[0140] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0141] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0142] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0143] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0144] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0145] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0146] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0147] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0148] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0149] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0151] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0152] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An application anomaly detection method, comprising: Obtain application indicator data of the application at the time point to be detected, wherein the application indicator data includes time series data generated during the operation of the application; Input the application indicator data to be detected into the anomaly detection model to obtain the first anomaly result, which includes application anomaly results or application normal results. When the first abnormal result is the application abnormal result, an alarm signal is issued; The anomaly detection model is obtained by training unlabeled historical indicator data as training samples X times in advance. At least one training session includes obtaining labels using the uncertainty value of some unlabeled historical indicator data and performing supervised training. The uncertainty value represents the degree of uncertainty of the corresponding unlabeled historical indicator data, and X is greater than or equal to 1. The training method of the anomaly detection model introduces a query strategy in an active learning manner to select appropriate samples for labeling. The process of obtaining labels and conducting supervised training by utilizing the uncertainty value of some unlabeled historical indicator data includes: Obtain the second anomaly result for each of the S unlabeled first historical indicator data. The second anomaly result includes the applied anomaly probability value and the applied normal probability value, where S is an integer greater than or equal to 1. The uncertainty value is obtained based on the degree of difference between the application anomaly probability value and the application normal probability value of each first historical indicator data. Obtain a label for the first historical indicator data whose uncertainty value is greater than a preset threshold, the label including an application anomaly label or an application normal label; The anomaly detection model is trained under supervision based on labeled first historical indicator data.

2. The method of claim 1, wherein, The acquisition of application indicator data at the time point to be detected includes: The first sub-indicator data within a predetermined time period prior to the time point to be detected; Determine Q second sub-indicator data within Q historical time periods, wherein any two time periods in the Q historical time periods belong to the same time period position on different dates, and Q is an integer greater than or equal to 1; The first sub-indicator data and the Q second sub-indicator data are sorted in chronological order to obtain the application indicator data to be detected.

3. The method according to claim 1 or 2, wherein: The application is deployed on multiple servers, and the application services running on any two of the multiple servers are different, wherein each server has the application metric data to be detected; The step of inputting the application indicator data to be detected into the anomaly detection model to obtain the first anomaly result includes: According to the calling order of the application services run by the multiple servers, the application indicator data to be detected by the multiple servers are concatenated to obtain the model input data; The anomaly detection model is used to process the model input data to obtain the first anomaly result.

4. The method of claim 1, wherein, After supervised training of the anomaly detection model based on labeled first historical indicator data, the step of training the model X times using unlabeled historical indicator data as training samples further includes: Collect K unlabeled second historical indicator data points, where K is an integer greater than or equal to 1; For each second historical indicator data point, calculate its similarity to each labeled first historical indicator data point; The label of each second historical indicator data is obtained based on the first historical indicator data with the highest similarity. The anomaly detection model is further trained based on labeled second historical indicator data.

5. The method of claim 1 or 4, wherein, Before obtaining the second anomaly results for each of the S unlabeled first historical indicator data, the step of using the unlabeled historical indicator data as training samples for X training iterations also includes: Cluster the N unlabeled first historical index data to obtain M clusters, where N is an integer greater than or equal to 2, M is an integer less than N, and N is greater than S; For each cluster, at least one first historical indicator data is extracted, wherein the S first historical indicator data includes all first historical indicator data extracted from the M clusters.

6. The method according to claim 5, wherein: The step of retrieving at least one first historical indicator data for each cluster includes: randomly retrieving a portion of the first historical indicator data from each cluster; and / or The label for obtaining the first historical indicator data with an uncertainty value greater than a preset threshold includes: determining the label based on the cluster to which the first historical indicator data with an uncertainty value greater than a preset threshold belongs.

7. The method of claim 1, wherein, The second anomaly results obtained for each of the S unlabeled first historical indicator data include: The classification model is used to process the S first historical indicator data to obtain their respective second anomaly results. The classification model is pre-trained using labeled third historical indicator data.

8. The method of claim 1, wherein, The degree of difference between the application anomaly probability value and the application normal probability value based on each first historical indicator data is used to obtain its uncertainty value, which includes: Calculate the absolute value of the difference between the application's abnormal probability value and the application's normal probability value; The negative of the absolute value of the difference is taken as the uncertainty value.

9. An application anomaly detection device, comprising: The data acquisition module is used to acquire application indicator data of the application at the time point to be detected. The application indicator data includes time series data generated during the operation of the application. The data input module is used to input the application indicator data to be detected into the anomaly detection model to obtain the first anomaly result, which includes the application anomaly result or the application normal result. An alarm module is used to issue an alarm signal when the first abnormal result is the application abnormal result; The anomaly detection model is obtained by training unlabeled historical indicator data as training samples X times in advance. At least one training session includes obtaining labels using the uncertainty value of some unlabeled historical indicator data and performing supervised training. The uncertainty value represents the degree of uncertainty of the corresponding unlabeled historical indicator data, and X is greater than or equal to 1. The training method of the anomaly detection model introduces a query strategy in an active learning manner to select appropriate samples for labeling. The process of obtaining labels and conducting supervised training by utilizing the uncertainty value of some unlabeled historical indicator data includes: Obtain the second anomaly result for each of the S unlabeled first historical indicator data. The second anomaly result includes the applied anomaly probability value and the applied normal probability value, where S is an integer greater than or equal to 1. The uncertainty value is obtained based on the degree of difference between the application anomaly probability value and the application normal probability value of each first historical indicator data. Obtain a label for the first historical indicator data whose uncertainty value is greater than a preset threshold, the label including an application anomaly label or an application normal label; The anomaly detection model is trained under supervision based on labeled first historical indicator data.

10. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images