Abnormal application determination method and device for micro-service cluster, equipment and medium

Abstract

The embodiment of the invention provides an abnormal application determination method and device for a micro-service cluster, equipment and a medium, and the method comprises the steps: obtaining historical log data and current abnormal log data of an application, and an abnormal request for accessing the micro-service cluster through an entrance gateway, generating at least one historical log template according to the historical log data of the application, generating at least one current log template according to the current abnormal log data of the application, and comparing the historical log template with the current log template to obtain the difference degree of the application; and obtaining the target application called by the abnormal request and the difference degree of the target application, and determining the abnormal application of the micro-service cluster based on the difference degree of the target application. According to the embodiment of the invention, the possible abnormal root cause of the micro-service cluster can be obtained by collecting and intensively analyzing the log data and cooperating with the calling relationship of the micro-service cluster, so that the dependence of operation and maintenance personnel on front-line research and development personnel can be greatly reduced, and the loss caused by faults and the manpower troubleshooting cost are reduced.

Description

Technical Field

[0001] The present invention relates to the field of microservice technology, and in particular to a method, device, equipment and medium for determining abnormal applications of a microservice cluster. Background Art

[0002] In the operation and maintenance of medium and large microservice application clusters, operation and maintenance personnel usually do not have a deep understanding of the services provided by the services and the service call relationships involved in the business functions. The traditional way to deal with business failures is for operation and maintenance to notify front-line R&D personnel to handle them, but complex business scenarios usually involve multiple services and multiple teams, and multiple services call each other, making it difficult for R&D personnel to locate the failure. How to quickly locate the abnormal root cause of complex application calls to the network to reduce the duration of failures has become an operation and maintenance pain point for an enterprise.

[0003] The traditional method of handling faults is to build a link tracking system, collect Tracing data on each service, and find the abnormal call chain through the entry gateway exception when a fault occurs. However, this method relies on the construction of a relatively complex link tracking system and the transformation and adaptation of the basic framework, and the construction cost and difficulty are relatively high. Summary of the invention

[0004] In view of the above problems, the present invention aims to propose a method, device, equipment and medium for determining abnormal applications in a microservice cluster. By collecting and analyzing the log data of each application in the microservice cluster, the abnormal running status of the application is automatically and regularly detected, and a probability list of the root causes of the abnormalities is obtained in conjunction with the calling relationship of the microservice cluster, thereby reducing the dependence of operation and maintenance personnel on front-line R&D personnel, and at the same time reducing the losses caused by failures and the cost of manpower investigation.

[0005] According to a first aspect of the present invention, a method for determining abnormal applications of a microservice cluster is first provided, wherein the microservice cluster includes at least one application and an ingress gateway; the method includes:

[0006] Obtaining historical log data, current exception log data, and exception requests for accessing the microservice cluster via the ingress gateway of the application;

[0007] generating at least one historical log template according to the historical log data of the application, and generating at least one current log template according to the current abnormal log data of the application;

[0008] Comparing the historical log template with the current log template to obtain the difference between the applications;

[0009] Obtaining a target application called by the abnormal request and a difference degree of the target application;

[0010] Abnormal applications of the microservice cluster are determined based on the difference of the target application.

[0011] Optionally, the obtaining of historical log data and current abnormal log data of the application includes:

[0012] Collect log data of different applications in the microservice cluster;

[0013] Aggregating the log data of the different applications according to the application dimension to obtain log data corresponding to the same application;

[0014] The historical log data and the current abnormal log data are obtained from the log data corresponding to the same application.

[0015] Optionally, generating at least one historical log template according to the historical log data of the application, and generating at least one current log template according to the current abnormal log data of the application includes:

[0016] Classifying the historical log data and the current abnormal log data of the application;

[0017] For the historical log data of different categories, respectively extract the same logs of the historical log data to generate a historical log template;

[0018] For the current abnormal log data of different categories, the same logs of the current abnormal log data are respectively extracted to generate a current log template.

[0019] Optionally, the comparing the historical log template with the current log template to obtain the difference between the applications includes:

[0020] For any historical log template, determine the amount of historical log data corresponding to the historical log template;

[0021] Obtaining the contribution of the historical log template according to the proportion of the amount of historical log data corresponding to the historical log template in the amount of historical log data of the application;

[0022] For any current log template, determine the current abnormal log data volume corresponding to the current log template;

[0023] Obtaining a contribution degree of the current log template according to a proportion of the current abnormal log data volume corresponding to the current log template in the current abnormal log data volume of the application;

[0024] The difference degree of the application is obtained according to the contribution degree of the historical log template and the contribution degree of the current log template.

[0025] Optionally, obtaining the difference of the application according to the contribution of the historical log template and the contribution of the current log template includes:

[0026] The sum of the absolute values ​​of the differences between the contribution of the historical log template and the contribution of the current log template is calculated to obtain the difference of the application.

[0027] Optionally, obtaining the target application called by the abnormal request includes:

[0028] Obtaining a call chain diagram of the abnormal request;

[0029] The target application called by the abnormal request is obtained according to the call link graph.

[0030] Optionally, the determining the abnormal application of the microservice cluster based on the difference of the target application includes:

[0031] Obtaining a call depth of the target application according to a call chain diagram of the abnormal request;

[0032] Obtaining an abnormal probability of the target application according to the call depth of the target application and the difference of the target application;

[0033] The abnormal application of the microservice cluster is determined according to the abnormal probability of the target application.

[0034] According to a second aspect of the present invention, there is also provided a device for determining abnormal applications of a microservice cluster, characterized in that the device comprises:

[0035] A data acquisition module, used to acquire historical log data of the application, current abnormal log data, and abnormal requests for accessing the microservice cluster via the ingress gateway;

[0036] A log template generating module, configured to generate at least one historical log template according to the historical log data of the application, and to generate at least one current log template according to the current abnormal log data of the application;

[0037] A difference acquisition module, used for comparing the historical log template with the current log template to obtain the difference of the application;

[0038] A target application acquisition module, used to acquire the target application called by the abnormal request and the difference degree of the target application;

[0039] The abnormal application determination module is used to determine the abnormal application of the microservice cluster based on the difference of the target application.

[0040] According to the third aspect of the present invention, there is also provided an electronic device, characterized in that it includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, it implements the abnormal application determination method of the microservice cluster as described above.

[0041] According to a fourth aspect of the present invention, a computer-readable storage medium is further provided, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the abnormal application determination method of the microservice cluster as described above is implemented.

[0042] The abnormal application determination method of the microservice cluster provided by the embodiment of the present invention obtains the historical log data, current abnormal log data of the application, and the abnormal request to access the microservice cluster via the entry gateway, generates at least one historical log template according to the historical log data of the application, generates at least one current log template according to the current abnormal log data of the application, compares the historical log template with the current log template, and obtains the difference of the application; obtains the target application called by the abnormal request and the difference of the target application, and determines the abnormal application of the microservice cluster based on the difference of the target application. The embodiment of the present invention collects and centrally analyzes log data, automatically and regularly detects the abnormal running state of the application, and obtains the probability list of the abnormal root cause in conjunction with the call relationship of the microservice cluster, thereby obtaining the possible abnormal root cause of the microservice cluster. Compared with the traditional ELK solution that configures a log template for each application, it has greater flexibility and feasibility. Compared with the traditional link tracking system, there is no need to make system modifications. The application log is already available in most systems as the most basic troubleshooting means, and has greater practicality. It can greatly reduce the dependence of operation and maintenance personnel on front-line R&D personnel, and reduce the losses caused by failures and the cost of manpower troubleshooting.

[0043] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. In the drawings:

[0045] Figure 1It is a flowchart of the steps of a method for determining abnormal applications of a microservice cluster provided by an embodiment of the present invention;

[0046] Figure 2 This is an abnormal analysis diagram of a microservice cluster provided by an embodiment of the present invention;

[0047] Figure 3 It is a statistical flow chart of a log template provided by an embodiment of the present invention;

[0048] Figure 4 This is a log template generation flow chart provided by an embodiment of the present invention;

[0049] Figure 5 It is a request call graph of a microservice cluster provided by an embodiment of the present invention;

[0050] Figure 6 It is a structural diagram of an abnormal application determination device for a microservice cluster provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it can be understood by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable readers to better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present invention can also be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0052] ELK: An open source log collection component combination, which is a combination of three open source software: Elasticsearch (elastic search), Logstash (log storage), and Kibana.

[0053] Tracing: A link tracking system used to track the service details of each request, including time consumption / status, etc.

[0054] In traditional troubleshooting scenarios, developers usually rely on logs, the most basic troubleshooting method. In abnormal situations, logs usually have a large number of abnormal outputs. If a log clustering model for each application in a microservice cluster is built, the probability of possible abnormalities can be obtained by comparing the model differences. Logs are existing basic data, and only need to be collected and analyzed centrally, and the abnormal running status of the application is automatically and regularly detected. In conjunction with the call relationship of the microservice cluster, a probability list of the root cause of the abnormality is obtained. Operation and maintenance personnel can greatly reduce the dependence on front-line R&D personnel, and reduce the losses caused by failures and the cost of manpower troubleshooting.

[0055] Reference Figure 1 , shows a flowchart of a method for determining abnormal applications of a microservice cluster provided by an embodiment of the present invention, which may specifically include the following steps:

[0056] Step 101, obtaining historical log data of the application, current exception log data, and exception requests for accessing the microservice cluster via the ingress gateway.

[0057] Historical log data usually refers to all log data within a certain time range, and current exception log data usually refers to errors or exception logs that occurred within a certain time period (log data within the time period when the microservice cluster currently has an exception).

[0058] In a microservice cluster, all types of log data reported by all applications in the microservice cluster are collected to the log storage center, and the collected log data is aggregated according to the application dimension, the log data of different applications are isolated, and the log data of the same application are summarized. After that, the historical log data and current abnormal log data of each application can be obtained, for example, through query language, log analysis tools, etc.

[0059] The log data of different applications are isolated to facilitate separate analysis and processing, and the log data of the same application are aggregated to facilitate overall analysis and monitoring.

[0060] Among them, the log storage center is a place where log data is centrally stored and managed, and usually includes the following components: log storage: use a distributed storage system to store log data, message queue: use a message queue to buffer log data to ensure high throughput and reliability of data, log processing: use a log processing framework to clean, convert and aggregate log data.

[0061] Step 102: Generate at least one historical log template according to the historical log data of the application, and generate at least one current log template according to the current abnormal log data of the application.

[0062] The log clustering model is constructed by applying historical log data to obtain various log templates and indexes of various log templates as a reference system, where the index is equivalent to the ID of the log template, which is used to identify the unique log template. Specifically, the log clustering model is a model generated by analyzing the log data through a clustering algorithm. Its function is to aggregate similar log records together to form different clusters, that is, to classify the log data, and then extract representative log records for each cluster to form a log template. For example, clustering the historical log data, two clusters are obtained: Cluster 1: 100 log records, indicating that the user login is successful, and Cluster 2: 200 log records, indicating that the user login fails; then two historical log templates can be extracted: Log Template 1: User login is successful, Log Template 2: User login fails. Among them, 1 and 2 are the indexes of the log templates, the amount of log data corresponding to Log Template 1 is 200, and the amount of log data corresponding to Log Template 2 is 100. The same method is used to cluster the current abnormal log data to obtain two clusters. For each cluster, representative log records are extracted to form two current log templates. At this time, the amount of log data corresponding to log template 1 may be 120, and the amount of log data corresponding to log template 2 may be 230.

[0063] The log template obtained by clustering historical log data is the same as the log template obtained by clustering current log data, but the amount of log data corresponding to the log template obtained by clustering historical log data is different from the amount of log data corresponding to the log template obtained by clustering current log data. Therefore, possible abnormal changes in the application can be evaluated by the change in the amount of log data in the log template.

[0064] Step 103: compare the historical log template with the current log template to obtain the difference between the applications.

[0065] After clustering the log data and obtaining multiple log templates corresponding to each cluster and the amount of log data corresponding to each log template, the proportion of each log template can be obtained. The proportion of log templates obtained from historical log data is calculated by subtracting the proportion of log templates obtained from current abnormal data to obtain the difference of each log template, and then the difference of each log template is added to obtain the difference of the application.

[0066] Let's take a simple example to illustrate. Suppose there are two historical log templates: Log Template 1: User Login Success, Log Template 2: User Login Failure. The log data volume corresponding to Log Template 1 is 200, and the log data volume corresponding to Log Template 2 is 100. The two current day log templates: Log Template 1 corresponds to 120 log data, and Log Template 2 corresponds to 230 log data. The proportion of historical log templates is:

[0067] Log template 1: 200 / (200+100)=66.7%

[0068] Log template 2: 100 / (200+100)=33.3%

[0069] The current daily log template ratio is:

[0070] Log template 1: 120 / (120+230)=34.3%

[0071] Log template 2: 230 / (120+230)=65.7%

[0072] Then we can get the difference of each log template:

[0073] Difference of log template 1 = |66.7%-34.3%| = 32.4%

[0074] Difference of log template 2 = |33.3%-65.7%| = 32.4%

[0075] The total difference obtained by adding the differences of each log template is: 32.4%+32.4%=64.8%, and the total difference is the difference of the application.

[0076] Step 104: Obtain the target application called by the abnormal request and the difference between the target applications.

[0077] Abnormal requests can be identified through the gateway of the microservice cluster. As the entrance to the microservice cluster, the gateway is responsible for processing all requests entering the cluster. Therefore, it can capture and record detailed information of all requests, including request path, request parameters, response status code, etc. By analyzing this information, abnormal requests can be identified.

[0078] Then, the call chain diagram of the abnormal request can be obtained through the chain tracking system. According to the call chain diagram, we can find out which applications are called by the abnormal request, and the difference between these applications has been calculated previously.

[0079] Step 105: Determine abnormal applications of the microservice cluster based on the difference of the target application.

[0080] The call chain diagram of the abnormal request obtained by the chain tracking system can also obtain the call depth of each application. Then, according to the call depth of each application and the difference degree corresponding to each application, the root cause probability ranking of each application is obtained, that is, the root cause of the microservice cluster abnormality is the probability of each application. According to the root cause probability ranking, the application with a higher probability can be directly located for abnormality troubleshooting.

[0081] Through the above method, by collecting and centrally analyzing log data, and coordinating the call relationship of the microservice cluster to obtain a probability list of abnormal root causes, the abnormal root causes of complex application calls to the network can be quickly located, and the dependence of operation and maintenance personnel on front-line R&D personnel can be greatly reduced, reducing the losses caused by failures and the cost of manpower investigation.

[0082] In an embodiment of the present invention, by obtaining the historical log data, current abnormal log data of the application, and the abnormal request to access the microservice cluster via the entry gateway, at least one historical log template is generated according to the historical log data of the application, at least one current log template is generated according to the current abnormal log data of the application, and the historical log template is compared with the current log template to obtain the difference of the application; the target application called by the abnormal request and the difference of the target application are obtained, and the abnormal application of the microservice cluster is determined based on the difference of the target application. The embodiment of the present invention collects and centrally analyzes log data, automatically and regularly detects the abnormal running state of the application, and obtains the probability list of the abnormal root cause in conjunction with the call relationship of the microservice cluster, thereby obtaining the possible abnormal root cause of the microservice cluster. Compared with the traditional ELK solution that configures a log template for each application, it has greater flexibility and feasibility. Compared with the traditional link tracking system, there is no need to make system modifications. The application log is already available in most systems as the most basic troubleshooting means, and it has greater practicality. It can greatly reduce the dependence of operation and maintenance personnel on front-line R&D personnel, and reduce the losses caused by failures and the cost of manpower troubleshooting.

[0083] In an optional embodiment of the present invention, step 101 further includes the following steps:

[0084] S1011, collects log data of different applications in the microservice cluster;

[0085] S1012, aggregating log data of different applications according to the application dimension to obtain log data corresponding to the same application;

[0086] S1013, obtaining historical log data and current abnormal log data from log data corresponding to the same application.

[0087] Collect all kinds of log data reported by all applications in the microservice cluster to the log storage center. The log collection methods include log agent, in-application log library, application reporting, etc. Among them, log agent (Log Agent): deploy log agent (such as Fluentd (fluent log collector), Logstash (log storage), Filebeat (file collector), etc.) on the application server. The agent is responsible for collecting local log files and sending log data to the log storage center; in-application log library: integrate the log library (such as Log4j (log 4j), Logback (log backtracking), Serilog (sequence log), etc.) in the application code, and configure the log library to send log data directly to the log storage center; application reporting: the application directly sends log data to the log storage center through the HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol) / HTTPS (HyperText TransferProtocol Secure, Hypertext Transfer Protocol Secure) interface.

[0088] In the process of transmitting log data from the application to the log center, it is necessary to ensure the reliability and security of the data, use a reliable transmission protocol, encrypt the log data during transmission to prevent data leakage, compress the log data, and reduce the transmission bandwidth usage.

[0089] In the log storage center, log data can be aggregated according to the application dimension. When collecting logs, add an application ID to each log so that it can be aggregated according to the application dimension later. When storing logs, create different indexes or partitions according to the application ID to facilitate query and aggregation according to the application dimension. Use log processing frameworks (such as Logstash, Fluentd, Spark Streaming, etc.) to aggregate log data and generate application dimension statistics (such as log volume, error rate, response time, etc.). In this way, log data from different applications can be isolated and log data from the same application can be aggregated.

[0090] After the log data is aggregated, the historical exception data and current log data of each application can be obtained through query language, log analysis tools, etc.

[0091] In an optional embodiment of the present invention, step 102 further includes the following steps:

[0092] S1021, classifying the historical log data and current abnormal log data of the application;

[0093] S1022, for different categories of historical log data, extract the same logs of the historical log data respectively, and generate a historical log template;

[0094] S1023, for different categories of current abnormal log data, respectively extract the same logs of the current abnormal log data to generate a current log template.

[0095] The log data of different nodes of the same application are classified according to the application dimension. The log data of each application is used to calculate the topN log templates according to the similarity algorithm, converge the log data volume, and store the log templates of each application for subsequent comparison. Among them, the similarity algorithm can use the edit distance algorithm. The edit distance (Edit Distance), also known as the Levenshtein Distance, is a measurement method for measuring the difference between two strings. It represents the minimum number of single-character editing operations (insertion, deletion, replacement) required to convert one string into another. The smaller the edit distance, the more similar the two strings are.

[0096] In the embodiment of the present invention, the similarity of each line of log text can be calculated using Jaro-Winkler Distance, which is an improved edit distance algorithm that adds a weight for prefix matching on the basis of Jaro distance, making similarity calculation more accurate. Grouping and classification are performed based on the calculated similarity, and the log texts are grouped and classified based on a preset similarity threshold (such as 60%). If the similarity of two texts is greater than or equal to the threshold, they are classified into the same group.

[0097] For similar group contents (logs in the same group), use the HanLP word segmentation tool to segment each line of content in each group, perform intersection processing on the word sequences after word segmentation, extract common keywords, extract one of the log contents as a sample, and perform word segmentation processing, compare the word segmentation with the extracted common keywords, and retain the intersection of the two word segmentations to obtain the log template. For example, the log content in a group is:

[0098] User 1 successfully logged in

[0099] User 2 successfully logged in

[0100] Use HanLP to segment the contents of the group. The segmentation results are:

[0101] User / 1 / Login / Success

[0102] User / 2 / Login / Success

[0103] Perform intersection processing on the analyzed sequence to extract common keywords. The intersection result is:

[0104] User / Login / Success

[0105] Then, we extract one of the log contents as a sample and perform word segmentation processing to obtain: User / 1 / Login / Success. By comparing this result with User / Login / Success, we can obtain the log template: User <&> Login Successfully, where <&> is a placeholder. The purpose of extracting samples for comparison is to determine the position of the placeholder.

[0106] In the embodiment of the present invention, the historical log data is the log data of the day before the application and the log data of the hour before the application. At least one log template is generated by applying the log data of the day before the application. It is assumed that three log templates are generated: log template 1, log template 2 and log template 3, as a reference template set. Three log templates are generated by applying the log data of the hour before the application, and the three generated log templates are compared with the reference template set. If the similarity exceeds 80%, it is considered to be the same template as a healthy reference system.

[0107] The current abnormal log data can be taken from the current hour of the application, indicating that an abnormality occurred in the microservice cluster within the current hour. Three log templates are generated through the current hour's log data. The three generated log templates are compared with the benchmark template set. If the similarity exceeds 80%, they are considered to be the same template and are used as the detected object.

[0108] In an optional embodiment of the present invention, step 103 further includes the following steps:

[0109] S1031, for any historical log template, determining the amount of historical log data corresponding to the historical log template;

[0110] S1032, obtaining a contribution degree of the historical log template according to a proportion of the amount of historical log data corresponding to the historical log template in the amount of historical log data of the application;

[0111] S1033, for any current log template, determining the current abnormal log data volume corresponding to the current log template;

[0112] S1034, obtaining the contribution of the current log template according to the proportion of the current abnormal log data volume corresponding to the current log template in the current abnormal log data volume of the application;

[0113] S1035, obtaining the difference of the application according to the contribution of the historical log template and the contribution of the current log template.

[0114] For the log data corresponding to each different application, the historical log data and current abnormal log data corresponding to each application are obtained. The historical log data is still the log data of the application one day ago and the log data of the application one hour ago, and the current abnormal log data is the log data of the application one hour ago.

[0115] The log templates generated by applying the log data of the previous day are used as the benchmark template set, as shown in Table 1:

[0116] Table 1 Benchmark template set

[0117] index Log clustering benchmark template for the previous day 1 Log Template 1 2 Log Template 2 3 Log Template 3 ... …

[0118] Specifically, the log templates generated by applying the log data of the previous day include: log template 1, log template 2, log template 3, and the like.

[0119] The log template generated by applying the log data of the previous hour is used as the health reference system, as shown in Table 2:

[0120] Table 2 Health reference system

[0121] Index n Clustering reference template for the previous hour Number of logs Proportion (contribution) 3 Log Template 3 4390 C3 2 Log Template 2 2381 C2 ... ... xx Cn

[0122] Specifically, the log template log template 3 generated by applying the log data of the previous hour corresponds to 4390 log data, and the log template 2 corresponds to 2381 log data. Among them, the proportion of log template 3 is C3, that is, the contribution of log template 3 is C3, and the proportion of log template n is Cn, that is, the contribution of log template n is Cn. Taking log template n as an example, the calculation method of the proportion of log templates, that is, the contribution, is: Cn = the log data volume (number of logs) corresponding to log template n / the sum of the log data volumes corresponding to all log templates.

[0123] The log template generated by the current abnormal log data is used as the detected object, as shown in Table 3:

[0124] Table 3 Detected objects

[0125] Index n Current 1-hour clustering reference template Number of logs Proportion (contribution) 2 Log Template 2 3051 S2 1 Log Template 1 2521 S1 ... ... xx Sn

[0126] Specifically, the amount of log data corresponding to log template 2 generated by applying the current 1 hour's log data is 3051, and the amount of log data corresponding to log template 1 is 2521. Among them, the proportion of log template 2 is S2, that is, the contribution of log template 2 is S2, and the proportion of log template n is Sn, that is, the contribution of log template n is Sn.

[0127] The cluster reference template of the previous hour is used as the benchmark, and the difference in the proportion (contribution) of each log template in the template of the current hour is calculated as follows:

[0128]

[0129] Where n represents the log template index, m represents the number of log templates, and abs represents the absolute value.

[0130] Then, the total difference is obtained through the difference of each log template, which is the difference of the application.

[0131] In an optional embodiment of the present invention, S1035 further includes the following sub-steps:

[0132] S1035-1, calculating the sum of the absolute values ​​of the differences between the contribution of the historical log template and the contribution of the current log template, and obtaining the difference degree of application.

[0133] After obtaining the difference of each log template through the difference formula, the difference of each log template is added together to obtain the total difference, which is the difference of the application.

[0134] Preset a difference threshold, compare the difference of the application with the threshold, and if the difference of the application is greater than the threshold, it is determined that the application may have risk anomalies in the current hour. The threshold is mainly set to filter out applications with very small changes, and such applications do not need to be analyzed.

[0135] In an optional embodiment of the present invention, step 104 further includes the following sub-steps:

[0136] S1041, obtaining a call chain diagram of the abnormal request;

[0137] S1042, obtaining the target application called by the abnormal request according to the call link graph.

[0138] You can deploy a link tracking system in a microservice cluster, such as Zipkin (Zipkin link tracking system), Jaeger (Jaeger link tracking system) or SkyWalking (SkyWalking link tracking system), and integrate a link tracking client in each microservice. The link tracking client will record the call information during the call process of each microservice (each microservice is an independent application), including the call time, caller service, callee service, call result, etc. These call information will be sent to the link tracking system for storage and analysis. The link tracking system will generate a call link diagram based on the stored call information, showing the complete call path of the request in the microservice cluster.

[0139] Through the interface or API provided by the link tracking system, obtain the call link graph of the abnormal request, analyze the call link graph, and identify all applications called by the abnormal request.

[0140] Reference Figure 5 , shows a request call diagram of a microservice cluster provided by an embodiment of the present invention, which is as follows:

[0141] The customer accesses the microservice cluster through the access gateway, which captures and records detailed information of all requests, including request path, request parameters, response status code, etc. By analyzing this information, it can be identified that the abnormal request is request 1.

[0142] The applications involved in request 1 are: application A, application C, and application D, and all applications called by the abnormal request (request 1) are obtained.

[0143] In an optional embodiment of the present invention, step 105 further includes the following sub-steps:

[0144] S1051, obtaining the call depth of the target application according to the call link graph of the abnormal request;

[0145] S1052, obtaining an abnormal probability of the target application according to the call depth of the target application and the difference of the target application;

[0146] S1053: Determine abnormal applications of the microservice cluster according to the abnormal probability of the target application.

[0147] According to the call chain diagram of the abnormal request, the call depth of all applications called by the abnormal request can also be obtained. Figure 5 As shown, request 1 first calls application A, and then application A calls application C and application D. The call depth of application A is 1, and the call depth of application C and application C is 2.

[0148] Through the above steps, the difference of each application has been obtained, and then the call depth and root cause list of the difference of all applications called by the abnormal application can be obtained, as shown in Table 4:

[0149] Table 4 List of root causes

[0150] application Call Depth Log Difference Application A depth(A)=1 diff(A) Application C depth(C)=2 diff(C) Application D depth(D)=2 diff(D)

[0151] Specifically, the call depth of application A is 1, and its difference is diff(A); the call depth of application C is 2, and its difference is diff(C); and the call depth of application D is 1, and its difference is diff(D).

[0152] The abnormal root cause probability of each application can be obtained based on the application's call depth and difference, as follows:

[0153] Application anomaly root cause probability = (depth(app) / max_depth+diff(app) / sum_diff) / 2

[0154] Among them, max_depth indicates the maximum call depth, sum_diff indicates the sum of differences, and app indicates the application name. The application abnormality root cause probability is the probability that the root cause of the microservice cluster abnormality is this application.

[0155] Then, based on the probability of the root cause of the anomaly for each application, you can locate the application with the highest probability of the root cause of the anomaly and conduct troubleshooting.

[0156] This application collects and centrally analyzes log data, extracts current log templates and historical log templates, compares the current log template with the historical log model, and evaluates the probability of the root cause of application anomalies in combination with the call depth and call network of the microservice cluster. Compared with the traditional ELK solution that configures log templates for each application, this solution has greater flexibility and feasibility. Compared with the traditional link tracking system, this solution does not require system modification. Application logs are already available in most systems as the most basic troubleshooting method, and are relatively practical.

[0157] Reference Figure 2 , shows an abnormal analysis diagram of a microservice cluster provided by an embodiment of the present invention, as follows:

[0158] Collect all kinds of log data reported by all applications (business application 1, business application 2, business application 3) in the microservice cluster to the log storage center. The log storage center aggregates the collected log data according to the application dimension, isolates the log data of different applications, and summarizes the log data of the same application. The logs are analyzed by clustering algorithms to obtain the clustering model of the latest logs (logs of the previous hour, abnormal logs of the current hour), and possible abnormal changes of the application are obtained according to the changes in the model. Among them, the clustering model is a model generated after analyzing the log data through the clustering algorithm. Its function is to aggregate similar log records together to form different clusters (groups).

[0159] Every day, the application log template generated by applying historical (here we take the previous day and previous hour) log text data is used as a benchmark reference. The log template generated by applying the log data of the last N (N is determined according to the actual time period of the cluster anomaly, here we take 1) hours is compared with the log template generated by the historical log text data to obtain the difference.

[0160] Through the calling relationship of microservices (applications) and combined with the gateway entry exception request, the calling topology network (relationship network) is found, and the related application exception log model analysis in the topology network (relationship network) is performed to obtain the abnormal link path that may cause problems, thereby obtaining the abnormal root cause analysis report.

[0161] Reference Figure 3 , shows a statistical flow chart of a log template provided by an embodiment of the present invention, which is as follows:

[0162] Classify log data by application dimension, isolate log data of different applications, and aggregate log data of the same application;

[0163] According to the classified log data, for different categories of log data, n log templates are extracted, and the number of log records (log data volume) of each type of model is counted to obtain the distribution statistics (proportion, contribution) of each log template.

[0164] Reference Figure 4 , shows a log template generation flow chart provided by an embodiment of the present invention, which is as follows:

[0165] Assume that the original log text set in the figure is: 2023-11-03 10:29:37WARN Watchdog-Reconnecting, last destination was / 171.16.30.2:31046

[0166] java.util.concurrent.TimeoutException:Waited 3000milliseconds(plus4milliseconds,330400nanoseconds delay)for ClientCalls$GrpcFuture

[0167] 2023-11-03 15:09:01ERROR HikariPool-1-Error thrown while acquiringconnection from data source2023-11-03 15:58:40INFO query storage nodessuccess,http code:200

[0168] 2023-11-03 15:59:40ERROR dial tcp 172.16.10.1:33295connectex:Noconnection could be made because the target machine actively refused it.

[0169] 2023-11-03 15:59:42ERROR dial tcp 172.16.10.2:33295connectex:Noconnection could be made because the target machine actively refused it.

[0170] Compare each line of text in the original text set of the log in turn, use the edit distance algorithm (Jaro-Winkler Distance) to calculate the similarity, and group and classify the log text according to the preset similarity threshold (such as 60%). If the similarity of two texts is greater than or equal to the threshold, they are classified into the same group.

[0171] Using the editing algorithm to group and classify, a set of log text data can be obtained as follows:

[0172] ERROR dial tcp 172.16.10.1:33295connectex

[0173] ERROR dial tcp 172.16.10.2:33295connectex

[0174] Use HanLP to segment each line of the grouped content. For example, segment the second line of text data to get:

[0175] ERROR / nx

[0176] dail / nx

[0177] TCP / nx

[0178] 172.16.10.2 / m

[0179] : / w

[0180] 33295 / m

[0181] Connectex / nx

[0182] The segmented objects are processed by intersection, and the phrases after the intersection are retained. Then one of the log contents is extracted as a sample and segmented. After segmentation, it is compared with the intersection of the previous step. Only the segmentations including the intersection are retained, and the generated log template can be obtained:

[0183] ERROR dial tcp<&#&>:33295connectex

[0184] Among them, <&#&> is a placeholder.

[0185] Reference Figure 6 , showing a schematic diagram of the structure of an abnormal application determination device for a microservice cluster provided by an embodiment of the present invention, the device comprising:

[0186] The data acquisition module 201 is used to acquire the historical log data of the application, the current abnormal log data, and the abnormal request for accessing the microservice cluster via the ingress gateway;

[0187] A log template generating module 202, configured to generate at least one historical log template according to the historical log data of the application, and to generate at least one current log template according to the current abnormal log data of the application;

[0188] The difference acquisition module 203 is used to compare the historical log template with the current log template to obtain the difference of the application;

[0189] A target application acquisition module 204, used to acquire the target application called by the abnormal request and the difference degree of the target application;

[0190] The abnormal application determination module 205 is used to determine the abnormal application of the microservice cluster based on the difference of the target application.

[0191] In an optional embodiment of the present invention, the data acquisition module 201 includes:

[0192] A collection module, used to collect log data of different applications in the microservice cluster;

[0193] An aggregation module, used to aggregate the log data of different applications according to the application dimension to obtain log data corresponding to the same application;

[0194] The Japanese data acquisition module is used to acquire historical log data and current abnormal log data from the log data corresponding to the same application.

[0195] In an optional embodiment of the present invention, the log template generating module 202 includes:

[0196] A classification module, used for classifying the historical log data and the current abnormal log data of the application;

[0197] A historical log template generating module, used for respectively extracting the same logs of the historical log data of different categories and generating historical log templates;

[0198] The current log template generating module is used to extract the same logs of the current abnormal log data of different categories respectively, and generate the current log template.

[0199] In an optional embodiment of the present invention, the difference acquisition module 203 includes:

[0200] A first acquisition module, configured to determine, for any historical log template, the amount of historical log data corresponding to the historical log template;

[0201] A first contribution acquisition module, configured to obtain the contribution of the historical log template according to the proportion of the amount of historical log data corresponding to the historical log template in the amount of historical log data of the application;

[0202] A second acquisition module is used to determine, for any current log template, the current abnormal log data volume corresponding to the current log template;

[0203] A second contribution acquisition module, configured to obtain a contribution of the current log template according to a proportion of the current abnormal log data volume corresponding to the current log template in the current abnormal log data volume of the application;

[0204] The application difference acquisition module is used to obtain the difference of the application according to the contribution of the historical log template and the contribution of the current log template.

[0205] In an optional embodiment of the present invention, the application difference acquisition module includes:

[0206] The calculation module is used to calculate the sum of the absolute values ​​of the differences between the contribution of the historical log template and the contribution of the current log template to obtain the difference of the application.

[0207] In an optional embodiment of the present invention, the target application acquisition module 204 includes:

[0208] A call chain graph acquisition module, used to obtain the call chain graph of the abnormal request;

[0209] A third acquisition module is used to obtain the target application called by the abnormal request according to the call link graph;

[0210] In an optional embodiment of the present invention, the abnormal application determination module 205 includes:

[0211] A call depth acquisition module, used to obtain the call depth of the target application according to the call chain diagram of the abnormal request;

[0212] An abnormality probability determination module, used to obtain the abnormality probability of the target application according to the call depth of the target application and the difference degree of the target application;

[0213] A determination module is used to determine the abnormal application of the microservice cluster according to the abnormal probability of the target application.

[0214] In an embodiment of the present invention, by obtaining the historical log data, current abnormal log data of the application, and the abnormal request to access the microservice cluster via the entry gateway, at least one historical log template is generated according to the historical log data of the application, at least one current log template is generated according to the current abnormal log data of the application, and the historical log template is compared with the current log template to obtain the difference of the application; the target application called by the abnormal request and the difference of the target application are obtained, and the abnormal application of the microservice cluster is determined based on the difference of the target application. The embodiment of the present invention collects and centrally analyzes log data, automatically and regularly detects the abnormal running state of the application, and obtains the probability list of the abnormal root cause in conjunction with the call relationship of the microservice cluster, thereby obtaining the possible abnormal root cause of the microservice cluster. Compared with the traditional ELK solution that configures a log template for each application, it has greater flexibility and feasibility. Compared with the traditional link tracking system, there is no need to make system modifications. The application log is already available in most systems as the most basic troubleshooting means, and it has greater practicality. It can greatly reduce the dependence of operation and maintenance personnel on front-line R&D personnel, and reduce the losses caused by failures and the cost of manpower troubleshooting.

[0215] An embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for determining abnormal applications of a microservice cluster as described above is implemented.

[0216] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0217] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0218] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for determining abnormal applications of a microservice cluster as described above is implemented.

[0219] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0220] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0221] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0222] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0223] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for determining abnormal applications of a microservice cluster, characterized in that: The microservice cluster includes at least one application and an ingress gateway; the method includes: Obtaining historical log data, current exception log data, and exception requests for accessing the microservice cluster via the ingress gateway of the application; generating at least one historical log template according to the historical log data of the application, and generating at least one current log template according to the current abnormal log data of the application; Comparing the historical log template with the current log template to obtain the difference between the applications; Obtaining a target application called by the abnormal request and a difference degree of the target application; Abnormal applications of the microservice cluster are determined based on the difference of the target application.

2. The method according to claim 1, characterized in that The obtaining of the historical log data and current abnormal log data of the application includes: Collect log data of different applications in the microservice cluster; Aggregating the log data of the different applications according to the application dimension to obtain log data corresponding to the same application; The historical log data and the current abnormal log data are obtained from the log data corresponding to the same application.

3. The method according to claim 1, characterized in that The step of generating at least one historical log template according to the historical log data of the application, and generating at least one current log template according to the current abnormal log data of the application, comprises: Classifying the historical log data and the current abnormal log data of the application; For the historical log data of different categories, respectively extract the same logs of the historical log data to generate a historical log template; For the current abnormal log data of different categories, the same logs of the current abnormal log data are respectively extracted to generate a current log template.

4. The method according to claim 1 or 3, characterized in that: The comparing the historical log template with the current log template to obtain the difference between the applications includes: For any historical log template, determine the amount of historical log data corresponding to the historical log template; Obtaining the contribution of the historical log template according to the proportion of the amount of historical log data corresponding to the historical log template in the amount of historical log data of the application; For any current log template, determine the current abnormal log data volume corresponding to the current log template; Obtaining a contribution degree of the current log template according to a proportion of the current abnormal log data volume corresponding to the current log template in the current abnormal log data volume of the application; The difference degree of the application is obtained according to the contribution degree of the historical log template and the contribution degree of the current log template.

5. The method according to claim 4, characterized in that The obtaining the difference of the application according to the contribution of the historical log template and the contribution of the current log template includes: The sum of the absolute values ​​of the differences between the contribution of the historical log template and the contribution of the current log template is calculated to obtain the difference of the application.

6. The method according to claim 1, characterized in that The step of obtaining the target application called by the abnormal request includes: Obtaining a call chain diagram of the abnormal request; The target application called by the abnormal request is obtained according to the call link graph.

7. The method according to claim 1 or 6, characterized in that: The determining the abnormal application of the microservice cluster based on the difference of the target application includes: Obtaining a call depth of the target application according to a call chain diagram of the abnormal request; Obtaining an abnormal probability of the target application according to the call depth of the target application and the difference of the target application; The abnormal application of the microservice cluster is determined according to the abnormal probability of the target application.

8. A device for determining abnormal applications of a microservice cluster, characterized in that: The device comprises: A data acquisition module, used to acquire historical log data of the application, current abnormal log data, and abnormal requests for accessing the microservice cluster via the ingress gateway; A log template generating module, configured to generate at least one historical log template according to the historical log data of the application, and to generate at least one current log template according to the current abnormal log data of the application; A difference acquisition module, used for comparing the historical log template with the current log template to obtain the difference of the application; A target application acquisition module, used to acquire the target application called by the abnormal request and the difference degree of the target application; The abnormal application determination module is used to determine the abnormal application of the microservice cluster based on the difference of the target application.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for determining abnormal applications of a microservice cluster according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for determining abnormal applications of a microservice cluster according to any one of claims 1 to 7 is implemented.

Images