Abnormality detection method and device, equipment, storage medium and program product

Abstract

The invention provides an anomaly detection method which can be applied to the fields of financial science and technology and distributed technology. The method comprises the following steps: respectively carrying out feature aggregation on a plurality of obtained service call chain sets, and determining a plurality of statistical feature vectors; processing the plurality of statistical feature vectors through a trained first anomaly detection model to obtain a plurality of first anomaly scores, and performing screening in the plurality of service call chain sets based on anomaly degrees indicated by the plurality of first anomaly scores to obtain a candidate service call chain set; performing feature extraction on a plurality of candidate service call chains in the candidate service call chain set to obtain a plurality of service feature vectors; and processing the plurality of service feature vectors through a trained second anomaly detection model to obtain a plurality of second anomaly scores, and screening in the candidate service call chain set based on the anomaly degree indicated by the plurality of second anomaly scores to obtain an abnormal service call chain. The invention further provides an anomaly detection device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the fields of financial technology and distributed systems, and more specifically to an anomaly detection method, apparatus, device, medium, and program product. Background Technology

[0002] A call chain is a complete call path formed in a distributed service framework, where a business request starts from the entry service, goes through several service nodes, and is completed. However, complex call chains may encounter problems such as increased latency and abnormal error rates.

[0003] Currently, most anomaly detection relies on machine learning solutions that utilize call metrics from the call chain. However, these call metrics are mostly single-layer features, such as the response time of the call chain, which cannot accurately locate abnormal service nodes and cannot simultaneously support service-level anomaly detection and transaction-level anomaly location. Summary of the Invention

[0004] In view of the above problems, this application provides anomaly detection methods, apparatus, devices, storage media and program products to improve the accuracy and efficiency of call chain anomaly detection.

[0005] According to a first aspect of this application, an anomaly detection method is provided, comprising: performing feature aggregation on multiple sets of business call chains to determine multiple statistical feature vectors, wherein each set of business call chains has a matching service node, and the statistical feature vectors characterize the comprehensive performance of the set of business call chains; processing the multiple statistical feature vectors through a trained first anomaly detection model to obtain multiple first anomaly scores, and filtering the multiple sets of business call chains based on the degree of anomaly indicated by the multiple first anomaly scores to obtain a set of candidate business call chains; extracting features from multiple candidate business call chains in the set of candidate business call chains to obtain multiple business feature vectors, wherein the business feature vectors characterize the execution state of the candidate business call chains; and processing the multiple business feature vectors through a trained second anomaly detection model to obtain multiple second anomaly scores, and filtering the candidate set of business call chains based on the degree of anomaly indicated by the multiple second anomaly scores to obtain anomaly business call chains.

[0006] According to an embodiment of this application, the method further includes: forming a business call chain topology by taking multiple service nodes in the abnormal business call chain as nodes and the call relationship between the multiple service nodes as edges; marking the nodes in the business call chain topology as abnormal based on the abnormal service nodes indicated by the candidate business call chain set, thereby obtaining an abnormal business call chain topology, wherein the abnormal business call chain topology indicates the call path of the abnormal service nodes.

[0007] According to an embodiment of this application, the method further includes: determining the dependency strength between the abnormal service node and multiple adjacent nodes based on the call success rate, call frequency, and call duration indicated by the abnormal business call chain topology, wherein the dependency strength characterizes the degree of call dependency between the abnormal service node and adjacent nodes; filtering among multiple adjacent nodes based on the dependency strength to obtain multiple candidate root cause nodes; comparing the call data of the multiple candidate root cause nodes and the call data of the abnormal service node with preset benchmark data to obtain multiple node difference features; and filtering among the multiple candidate root cause nodes and the abnormal service node based on the degree and type of difference indicated by the multiple node difference features to obtain the root cause node.

[0008] According to embodiments of this application, the business call chain set includes multiple business call chains. For each business call chain set, feature aggregation is performed on the obtained business call chain set to determine the statistical feature vector, including: calculating the average response time, percentile response time, and maximum response time of the business call chain set based on the response time of multiple business call chains. The average response time characterizes the response efficiency of the service node matching the business call chain set, the percentile response time characterizes the processing performance of the service node matching the business call chain set, and the maximum response time characterizes the processing upper limit of the service node matching the business call chain set; obtaining the error rate of the business call chain set based on the number of call failures and the total number of calls of multiple business call chains. The error rate characterizes the processing stability of the service node matching the business call chain set; and obtaining the statistical feature vector based on at least one of the average response time, percentile response time, maximum response time, error rate, and average call frequency. The average call frequency characterizes the load intensity of the service node matching the business call chain set.

[0009] According to an embodiment of this application, the service feature vector includes at least one of response time, error identifier, and message size. The response time characterizes the calling speed of the candidate service call chain, the error identifier characterizes the calling availability of the candidate service call chain, and the message size characterizes the data transmission volume of the candidate service call chain.

[0010] According to an embodiment of this application, the abnormal service call chain includes multiple nodes. The method further includes: extracting features from the multiple nodes to obtain multiple node feature vectors, whereby the node feature vectors represent the execution state of the nodes; processing the multiple node feature vectors through a trained third anomaly detection model to obtain multiple third anomaly scores; and filtering among the multiple nodes based on the degree of anomaly indicated by the multiple third anomaly scores to obtain abnormal service nodes.

[0011] According to an embodiment of this application, the node feature vector includes at least one of node response time, node error identifier, and node message size. The node response time characterizes the node's calling speed, the node error identifier characterizes the node's calling availability, and the node message size characterizes the node's data transmission volume.

[0012] According to an embodiment of this application, the method further includes: performing consistency verification on the abnormal service nodes indicated by the candidate service call chain set based on the abnormal nodes; if the consistency verification passes, mapping the abnormal service nodes to a preset service call chain topology based on the service request identifier indicated by the abnormal service call chain, wherein the preset service call chain topology indicates the service node call path in response to the service request.

[0013] The second aspect of this application provides an anomaly detection device, comprising: a feature aggregation module, configured to perform feature aggregation on multiple sets of obtained business call chains respectively, and determine multiple statistical feature vectors, wherein each set of business call chains has a matching service node, and the statistical feature vectors characterize the comprehensive performance of the set of business call chains; a first anomaly detection module, configured to process the multiple statistical feature vectors through a trained first anomaly detection model to obtain multiple first anomaly scores, and filter the multiple sets of business call chains based on the degree of anomaly indicated by the multiple first anomaly scores to obtain a set of candidate business call chains; a feature extraction module, configured to extract features from multiple candidate business call chains in the set of candidate business call chains to obtain multiple business feature vectors, wherein the business feature vectors characterize the execution state of the candidate business call chains; and a second anomaly detection module, configured to process the multiple business feature vectors through a trained second anomaly detection model to obtain multiple second anomaly scores, and filter the candidate sets of business call chains based on the degree of anomaly indicated by the multiple second anomaly scores to obtain anomaly business call chains.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] Based on the anomaly detection method, apparatus, device, storage medium, and program product provided in the embodiments of this application, a multi-layer detection mechanism is designed. This mechanism obtains a statistical feature vector of a set of business call chains matching a service node through feature aggregation. The abnormal service node is located using this statistical feature vector, which reflects the overall performance of the set of business call chains. Then, feature extraction is performed only on candidate business call chains within the scope of the abnormal service node. The execution status reflected by the business feature vector is used to detect the abnormal call chain and determine the abnormal business call chain. This allows for the identification of abnormal service nodes and the accurate location of abnormal business call chains in a high-concurrency distributed environment. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of anomaly detection methods, apparatus, devices, storage media, and program products according to embodiments of this application.

[0020] Figure 2 A flowchart illustrating an anomaly detection method according to an embodiment of this application is shown schematically.

[0021] Figure 3 An example diagram illustrating a service call chain topology according to an embodiment of this application is shown.

[0022] Figure 4 A flowchart illustrating root cause analysis according to an embodiment of this application is shown schematically;

[0023] Figure 5 A flowchart illustrating statistical feature vector extraction according to an embodiment of this application is shown schematically.

[0024] Figure 6 A schematic diagram illustrating the structure of an anomaly detection device according to an embodiment of this application is shown; and

[0025] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing an anomaly detection method according to an embodiment of this application. Detailed Implementation

[0026] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] The embodiments of this application provide an anomaly detection method, which adopts a two-layer detection mechanism combining aggregate detection and single business call chain location to realize anomaly detection of business call chains. First, the set of business call chains matched with service nodes is used to locate abnormal service nodes by leveraging their comprehensive performance. Then, based on the narrowed scope of business call chains due to abnormal service nodes, the execution status of each business call chain is used to locate abnormal business call chains. This method simultaneously locates abnormal service nodes and abnormal business call chains, and improves the efficiency of anomaly detection in high-concurrency distributed environments.

[0031] Figure 1 The illustration shows an application scenario diagram of the anomaly detection method according to an embodiment of this application.

[0032] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0033] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] It should be noted that the anomaly detection method provided in this application embodiment can generally be executed by server 105. Correspondingly, the anomaly detection device provided in this application embodiment can generally be located in server 105. The anomaly detection method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the anomaly detection device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, network 104, and server 105 shown is merely illustrative. Any number of terminal devices, network 104, and server 105 can be used depending on implementation needs.

[0038] It should be noted that the methods and devices described and implemented in this application can be used for anomaly detection during business processing in the financial technology field, and can also be used in any field other than the financial technology field. The application fields of the methods and devices described and implemented in this application are not limited.

[0039] The following will be based on Figure 1 The described scene, through Figures 2-5The anomaly detection method according to the embodiments of this application will be described in detail.

[0040] Figure 2 A flowchart illustrating an anomaly detection method according to an embodiment of this application is shown schematically. Figure 3 This illustration schematically shows an example diagram of a service call chain topology according to an embodiment of this application. Figure 4 A flowchart illustrating root cause analysis according to an embodiment of this application is shown schematically.

[0041] like Figure 2 As shown, the anomaly detection method of this embodiment includes operations S210 to S240.

[0042] In operation S210, feature aggregation is performed on the obtained multiple sets of business call chains to determine multiple statistical feature vectors.

[0043] According to embodiments of this application, each of the multiple service call chain sets has a matching service node, the multiple service call chain sets correspond to multiple service nodes, and each service call chain set includes multiple service call chains.

[0044] Furthermore, a sliding time window can be used to determine the time window, and multiple business call chains within the time range indicated by the service node aggregation time window can be formed into multiple business call chain sets.

[0045] For example, firstly, a time window is determined with the difference between the current time and a preset time range as the starting point and the current time as the ending point; then, multiple business call chains containing the service node code within the current time range are filtered according to the service node code corresponding to each service node to form a set of business call chains.

[0046] Here, call chain data can be collected through the gateway layer.

[0047] According to embodiments of this application, statistical feature vectors characterize the overall performance of a set of business call chains, which can be obtained by calculating the mean of the call parameters in the set of business call chains, such as average response time, error rate, etc.

[0048] In operation S220, multiple statistical feature vectors are processed by the trained first anomaly detection model to obtain multiple first anomaly scores. Based on the degree of anomaly indicated by the multiple first anomaly scores, multiple sets of business call chains are filtered to obtain a set of candidate business call chains.

[0049] According to an embodiment of this application, a first anomaly detection model is used to detect service node anomalies based on multiple statistical feature vectors to obtain multiple corresponding first anomaly scores; each of the multiple first anomaly scores has a matching set of business call chains, which is used to indicate the degree of anomaly of the matching business call chain set; the candidate set of business call chains is used to indicate the abnormal service node 406.

[0050] For example, the first anomaly detection model can be an isolated forest model, a long short-term memory network, or a clustering algorithm, etc.; the selection of multiple service call chain sets based on the degree of anomaly indicated by multiple first anomaly scores can be based on the size of multiple first anomaly scores, arranging multiple first anomaly scores in descending order, and taking the service call chain set corresponding to the first anomaly scores in the top row (e.g., the first three) as the candidate service call chain set; the service node matched by the candidate service call chain set is the abnormal service node 406, and the number of candidate service call chain sets can be one or more, which is not limited in this embodiment.

[0051] In operation S230, feature extraction is performed on multiple candidate business call chains in the candidate business call chain set to obtain multiple business feature vectors.

[0052] According to an embodiment of this application, the candidate service call chain set includes multiple candidate service call chains, and the service feature vector characterizes the execution status of the candidate service call chain.

[0053] For example, the candidate service call chain can be a transaction call chain, and feature extraction of the candidate service call chain can include extracting the provider address, response time, error identifier and / or timestamp of the candidate service call chain.

[0054] In operation S240, multiple business feature vectors are processed by the trained second anomaly detection model to obtain multiple second anomaly scores. Based on the degree of anomaly indicated by the multiple second anomaly scores, the abnormal business call chain is obtained by filtering from the candidate business call chain set.

[0055] According to an embodiment of this application, a second anomaly detection model is used to perform call chain anomaly detection based on multiple business feature vectors to obtain multiple second anomaly scores; each of the multiple second anomaly scores has a matching candidate business call chain, which is used to indicate the degree of anomaly of the respective matching candidate business call chain.

[0056] For example, the second anomaly detection model can be an isolated forest model, a clustering algorithm, an autoencoder, or a long short-term memory network, etc.; the selection of candidate service call chains based on the degree of anomaly indicated by multiple second anomaly scores can be based on the multiple second anomaly scores arranged in descending order, and the candidate service call chains corresponding to the top-ranked (such as the first two) second anomaly scores are taken as the abnormal service call chains.

[0057] Through the above operations S210 to S240, feature aggregation and anomaly detection are performed on the business call chain according to time nodes based on the time window, so as to locate the abnormal service node 406 and locate and reduce the scope of anomaly detection in the business call chain to cope with the real-time detection of high-concurrency calls; anomaly detection is performed on the business transaction chain corresponding to a single business request by using business feature vectors to accurately locate abnormal business, and at the same time, service-level anomaly detection and business-level anomaly location are realized.

[0058] To ensure that the extracted business feature vectors can fully reflect the execution status of the candidate business call chain, in some embodiments, the business feature vectors include at least one of response time, error identifier, and message size.

[0059] According to the embodiments of this application, the response time characterizes the calling speed of the candidate service call chain, the error identifier characterizes the calling availability of the candidate service call chain, and the message size characterizes the data transmission volume of the candidate service call chain.

[0060] Furthermore, for each candidate business call chain, feature extraction is performed on the candidate business call chain to obtain the business feature vector. This vector can be obtained by extracting the node response time, node error identifier, and node message size of each service node in the corresponding call data of the candidate business call chain. The average node response time is calculated as the response time of the candidate business call chain, the most severe node error in the node error identifier is used as the error identifier of the candidate business call chain, and the average node message size is calculated as the message size of the candidate business call chain.

[0061] For example, the business feature vector includes response time, error identifier, and message size, and can be represented as:

[0062] OrderBusiness=[rt,error_flag,payload_size];

[0063] In the formula, OrderBusiness represents the business feature vector, rt represents the response time, error_flag represents the error flag, and payload_size represents the message size.

[0064] To accurately locate the abnormal service node 406, the abnormal business call chain is mapped to a call chain topology to facilitate observation of the abnormal service node 406's position in the call path and its relationship with other service nodes. In some embodiments, the method further includes:

[0065] First, a business call chain topology is formed by taking multiple service nodes in the abnormal business call chain as nodes and the call relationships between multiple service nodes as edges.

[0066] According to an embodiment of this application, the business call chain topology refers to a directed acyclic graph constructed based on the call data corresponding to the abnormal business call chain.

[0067] For example, multiple service nodes in an abnormal business call chain are used as nodes, with each node's ID corresponding to the service node's Span ID (service call representation). The call relationships between multiple service nodes are used as edges, with the direction of the edge being the calling node and the called node, forming a business call chain topology. Figure 3 The example business chain call topology shown includes nodes A, B, C, and D; nodes E and D; nodes F and D. The business call chain topology is identified by the business request ID (business request code) corresponding to the call chain, such as the transaction request ID. Edge attributes may include call frequency, average response time, and error rate.

[0068] Then, based on the abnormal service node 406 indicated by the candidate business call chain set, the nodes in the business call chain topology are marked as abnormal to obtain the abnormal business chain call topology.

[0069] According to an embodiment of this application, the abnormal service call chain topology indicates the call path of the abnormal service node 406.

[0070] For example, based on the Span ID of the abnormal service node 406 indicated by the candidate service call chain set, the corresponding node in the service call chain topology is marked as an abnormal node, thus obtaining the abnormal service chain call topology; based on this, the call position of the abnormal node in the service request and other nodes related to the abnormal node can be clearly identified from the abnormal service chain call topology.

[0071] Considering that traditional topology diagrams can only reflect static call relationships and cannot distinguish the strength of dependencies between nodes, they are prone to misclassifying nodes associated with abnormal nodes as root cause nodes 404. Therefore, to improve the accuracy of root cause analysis, some embodiments perform multi-level screening of root cause nodes 404 based on the obtained abnormal business call chain topology, combined with... Figure 4 The root cause analysis process shown includes the following:

[0072] First, based on the call success rate 409, call frequency 408, and call time 407 indicated by the abnormal business call chain topology, the dependency strength between the abnormal service node 406 and multiple adjacent nodes 401 is determined.

[0073] According to an embodiment of this application, the dependency strength characterizes the degree of call dependency between the abnormal service node 406 and its neighboring nodes. The greater the dependency strength, the stronger the dependency between the nodes.

[0074] Furthermore, the call success rate (409), call frequency (408), and call time (407) between nodes with call relationships in the abnormal business call chain topology are calculated respectively. The call success rate (409), call frequency (408), and call time (407) are weighted and summed to obtain the dependency strength.

[0075] Specifically, the call success rate (409) is obtained based on the ratio of the number of successful calls from the calling node to the called node within a preset time range to the total number of calls; the call frequency (408) is obtained based on the ratio of the average daily number of calls from the calling node to the called node within a preset time range to the total average daily number of calls from the calling node within a preset time range; and the call time (407), which indicates the stability of the call time, is obtained based on the difference between a preset value and the ratio of the standard deviation of the call time to the called node within a preset time range to the average call time.

[0076] For example, combined Figure 3 The success rate (409) is the ratio of the number of successful calls from node A to node B within 30 days to the total number of calls. The call frequency (408) is the ratio of the average daily number of calls from node A to node B within 30 days to the total average daily number of calls from node A within 30 days. The call duration (407) is the difference between 1 and the ratio of the standard deviation and the mean duration of calls from node A to node B within 30 days. The dependency strength between node A and node B... Represented as:

[0077] ;

[0078] In the formula, Indicates the success rate and time taken. The call took 407 seconds. This indicates a call frequency of 408. , , These represent weights, which can be 0.4, 0.3, and 0.3.

[0079] Then, based on the dependency strength, multiple candidate root cause nodes 405 are obtained by filtering among multiple neighboring nodes 401.

[0080] For example, the neighboring nodes of the abnormal service node 406 are sorted in descending order of dependency strength, and the neighboring nodes with dependency strength greater than a preset threshold are selected as candidate root cause nodes. For example, the neighboring nodes with dependency strength greater than 0.5 are selected as candidate root cause nodes.

[0081] Then, the call data 402 of multiple candidate root cause nodes and the call data 403 of abnormal service nodes are compared with the preset benchmark data to obtain the difference features of multiple nodes.

[0082] According to the embodiments of this application, the call data refers to the running data of the node during the call process, the baseline data refers to the non-abnormal running data of the node under the same service node, the same interface, and the same business scenario, and the node difference characteristics indicate the degree of difference between the call data and the baseline data.

[0083] Furthermore, the call data includes at least one of the following: CPU (processor) utilization timing sequence, memory usage timing sequence, and network latency sequence; the benchmark data includes at least one of the following: CPU utilization benchmark timing sequence, memory usage benchmark timing sequence, and network latency benchmark sequence.

[0084] For example, comparing the call data 402 of multiple candidate root cause nodes and the call data 403 of abnormal service nodes with preset benchmark data can be done by calculating the Euclidean distance between the CPU utilization time series and the CPU utilization benchmark time series, the memory usage time series and the memory usage benchmark time series, and the network latency series and the network latency benchmark series, and using the resulting multiple Euclidean distance matrices as node difference features.

[0085] Then, based on the degree and type of difference indicated by the difference characteristics of multiple nodes, the root cause node 404 is obtained by filtering among multiple candidate root cause nodes 405 and abnormal service nodes 406.

[0086] Furthermore, based on the difference type indicated by multiple node difference features, the node is filtered among multiple candidate root cause nodes 405 and abnormal service nodes 406. If the difference type corresponding to multiple candidate root cause nodes 405 and abnormal service nodes 406 is its own attribute feature, such as CPU utilization, then the node is the root cause node 404; otherwise, a significance check is performed on the difference features based on multiple nodes. If the node difference feature is a significant difference feature, then the corresponding node is the root cause node 404.

[0087] For example, an independent samples t-test is used to test the significance of differences among multiple nodes, and the P-value is calculated. If P < 0.05, then the difference is significant.

[0088] Figure 5 A flowchart illustrating statistical feature vector extraction according to an embodiment of this application is shown.

[0089] Single metrics are easily affected by random factors. To improve the accuracy of 406 error detection for abnormal service nodes, feature aggregation is performed on the business call chain set from multiple dimensions to comprehensively reflect the overall performance of the business call chain set and avoid misjudgments. In some embodiments, the business call chain set includes multiple business call chains. For each business call chain set, combined with... Figure 5 Operation S210 includes operations S510 to S530.

[0090] When operating S510, the average response time, percentile response time, and maximum response time of the set of business call chains are calculated based on the response time of multiple business call chains.

[0091] According to embodiments of this application, the average response time refers to the average time taken for all business call chains within the set to access service nodes, representing the response efficiency of service nodes matching the set of business call chains. The smaller the average response time, the higher the overall response efficiency of the service nodes for this group of requests. The percentile response time reflects the actual upper limit of the vast majority of business call chains, representing the processing performance of service nodes matching the set of business call chains. The maximum response time refers to the peak time taken for the group of business call chains to access service nodes, reflecting the longest waiting time for a single business call chain, representing the processing upper limit of service nodes matching the set of business call chains.

[0092] For example, the response time is extracted from the call data corresponding to multiple business call chains, and the average response time of the multiple business call chains is calculated as the average response time of the business call chain set; the 95th percentile response time of the multiple response times corresponding to the multiple business call chains is calculated as the percentile response time of the business call chain set; and the maximum value among the multiple response times is taken as the maximum response time of the business call chain set.

[0093] When operating S520, the error rate of the business call chain set is obtained based on the number of call failures and the total number of calls for multiple business call chains.

[0094] According to an embodiment of this application, the error rate refers to the proportion of service nodes that fail to access the service in the group of business call chains. It represents the processing stability of the service nodes that match the set of business call chains. The lower the error rate, the stronger the stability of the service nodes in processing the group of requests and the lower the probability of failure.

[0095] For example, the error rate of the set of business call chains is obtained based on the ratio of the number of failed calls to the total number of calls for multiple business call chains.

[0096] In operation S530, a statistical feature vector is obtained based on at least one of the following: average response time, percentile response time, maximum response time, error rate, and average call frequency 408.

[0097] According to the embodiments of this application, the average call frequency 408 refers to the average number of calls made by the group of business call chains to the service node per unit time, reflecting the "access density" of the group of call chains to the service node, and characterizing the load intensity of the service node matching the set of business call chains. The higher the frequency, the more requests of this type need to be processed by the service node at the same time, and the more concentrated the resource occupation.

[0098] For example, based on the average response time, percentile response time, maximum response time, error rate, and average call frequency (408), a statistical feature vector is obtained, represented as follows:

[0099] OrderService=[avg_rt,max_rt,p95_rt,error_rate,qps];

[0100] In the formula, OrderService represents the statistical feature vector, avg_rt represents the average response time, max_rt represents the maximum response time, p95_rt represents the 95th percentile response time, error_rate represents the error rate, and qps represents the average call frequency of 408.

[0101] For example, OrderService=[avg_rt=253ms,max_rt=280ms,p95_rt=280ms,error_rate]

[0102] =0.5, qps=4].

[0103] Next, taking the first anomaly detection model, Isolation Forest, as an example, we will further explain the processing of multiple statistical feature vectors. The specific process is as follows:

[0104] (1) Randomly select k samples from multiple statistical feature vectors (e.g., k=256).

[0105] (2) Randomly select a statistical feature vector from the sample.

[0106] (3) Generate a random threshold within the range of the eigenvalues ​​of the statistical feature vector and divide the sample into left and right subsets.

[0107] (4) Repeat steps (2) to (3) above until the subset contains only one sample or the tree depth reaches the upper limit log. k, resulting in a random tree.

[0108] (5) Repeat the above steps to construct N random trees (e.g., N=100) to form an isolated forest.

[0109] (6) For each statistical feature vector, calculate the average path length E(h(x)), and calculate the first anomaly score score(x,n) based on the average path length E(h(x)). The first anomaly score score(x,n) is expressed as:

[0110] score(x,n)=2 E(h(x)) / c(n) ;

[0111] In the formula, c(n) represents the expected average path length of the sample, x represents the statistical feature vector, and n represents the sample size; the score range of score(x,n) is [0,1]. The closer to 1, the higher the probability of an anomaly, and the closer to 0, the normal.

[0112] To ensure the reliability of 406 error detection for abnormal service nodes, a third detection model is used to perform secondary abnormal node detection based on node feature vectors, reducing the risk of missed or false detections in abnormal node detection. In some embodiments, the abnormal business call chain includes multiple nodes, and the method further includes:

[0113] First, feature extraction is performed on multiple nodes to obtain multiple node feature vectors.

[0114] According to an embodiment of this application, the node feature vector represents the execution state of the corresponding node.

[0115] Furthermore, the node feature vector includes at least one of node response time, node error identifier, and node message size. Node response time characterizes the node's call speed, that is, the total time interval from request initiation to node response. Node error identifier characterizes the node's call availability and is a binary identifier that marks whether a single call "completed normally". Node message size characterizes the node's data transmission volume and reflects the data transmission volume during the call process.

[0116] Based on this, the performance efficiency, operational reliability, and data transmission load of the business call chain are reflected by node response time, node error identifier, and node message size, respectively, so as to achieve an accurate characterization of the business call chain status.

[0117] For example, node response time, node error identifier, and node message size can be obtained by filtering the call data based on the node code (node ​​ID).

[0118] Then, the feature vectors of multiple nodes are processed by the trained third anomaly detection model to obtain multiple third anomaly scores. Based on the degree of anomaly indicated by the multiple third anomaly scores, the nodes are filtered to obtain anomaly service node 406.

[0119] According to an embodiment of this application, a third anomaly detection model is used to detect node anomalies based on multiple node feature vectors to obtain multiple corresponding third anomaly scores; each of the multiple third anomaly scores has a matching node, which is used to indicate the degree of anomaly of each matching node.

[0120] For example, the third anomaly detection model can be an isolated forest model, a long short-term memory network, or a clustering algorithm, etc. Filtering among multiple nodes based on the degree of anomaly indicated by multiple third anomaly scores can be done by arranging the multiple third anomaly scores in descending order and taking the node corresponding to the top-ranked (e.g., the first) third anomaly score as the anomaly node.

[0121] The abnormal node obtained from the above operations may be the same as or different from the abnormal service node 406 indicated by the candidate service call chain set. To ensure the accuracy and reliability of abnormal service node 406 detection and improve the visualization efficiency of abnormal service node 406, in some embodiments, the method further includes:

[0122] First, based on the abnormal node, a consistency check is performed on the abnormal service node 406 indicated by the candidate service call chain set.

[0123] According to an embodiment of this application, consistency verification refers to comparing whether the abnormal node and the abnormal service node 406 are the same.

[0124] For example, it can be compared whether the node ID of the abnormal node is the same as the node ID of the abnormal service node 406. If the node ID of the abnormal node is the same as the node ID of the abnormal service node 406, the consistency check is considered to have passed. If the node ID of the abnormal node is different from the node ID of the abnormal service node 406, the consistency check is considered to have failed.

[0125] Then, if the consistency check passes, based on the business request identifier indicated by the abnormal business call chain, the abnormal service node 406 is mapped to the preset business call chain topology.

[0126] According to embodiments of this application, each of the preset business call chain topologies has a matching business request identifier, and the preset business call chain topology indicates the service node call path in response to the business request.

[0127] For example, the preset business call chain topology can be determined in advance using historical call data of the corresponding business call chain. For instance, multiple service nodes in the business call chain can be used as nodes, and the call relationships between the multiple service nodes can be used as edges to form the business call chain topology. In practical applications, the corresponding business call chain topology can be obtained by calling the business request identifier indicated by the abnormal business call chain in the preset multiple business call chain topologies. The node in the corresponding business call chain topology is marked as abnormal according to the node ID of the abnormal service node 406.

[0128] If the consistency check fails, both the abnormal node and the abnormal service node 406 are considered abnormal and mapped to the preset business call chain topology. At the same time, an early warning instruction is generated and sent to the operation and maintenance personnel to remind them to further confirm the above consistency check results and abnormal node detection results.

[0129] Based on the above-described anomaly detection method, this application also provides an anomaly detection device. The following will be combined with... Figure 6 The device is described in detail.

[0130] Figure 6 A schematic block diagram of an anomaly detection device according to an embodiment of this application is shown.

[0131] like Figure 6 As shown, the anomaly detection device 600 of this embodiment includes a feature aggregation module 610, a first anomaly detection module 620, a feature extraction module 630, and a second anomaly detection module 640.

[0132] The feature aggregation module 610 is used to perform feature aggregation on the obtained multiple sets of business call chains respectively, and determine multiple statistical feature vectors. Each set of business call chains has a matching service node, and the statistical feature vectors characterize the comprehensive performance of the set of business call chains. In one embodiment, the feature aggregation module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0133] The first anomaly detection module 620 is used to process multiple statistical feature vectors through a trained first anomaly detection model to obtain multiple first anomaly scores. Based on the degree of anomaly indicated by the multiple first anomaly scores, it filters from multiple sets of business call chains to obtain a set of candidate business call chains. In one embodiment, the first anomaly detection module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0134] The feature extraction module 630 is used to extract features from multiple candidate service call chains in the candidate service call chain set, obtaining multiple service feature vectors. The service feature vectors represent the execution state of the candidate service call chains. In one embodiment, the feature extraction module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0135] The second anomaly detection module 640 is used to process multiple business feature vectors through a trained second anomaly detection model to obtain multiple second anomaly scores. Based on the degree of anomaly indicated by the multiple second anomaly scores, it filters from the candidate business call chain set to obtain an abnormal business call chain. In one embodiment, the second anomaly detection module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0136] According to an embodiment of this application, the device further includes a visualization module, which is used to form a business call chain topology with multiple service nodes in the abnormal business call chain as nodes and the call relationship between the multiple service nodes as edges; and to mark the nodes in the business call chain topology as abnormal based on the abnormal service node 406 indicated by the candidate business call chain set, thereby obtaining the abnormal business call chain topology, which indicates the call path of the abnormal service node 406.

[0137] According to an embodiment of this application, the device further includes a root cause analysis module. The root cause analysis module is used to determine the dependency strength between multiple nodes in the call path of the abnormal service node 406 based on the call success rate 409, call frequency 408, and call duration 407 indicated by the abnormal business call chain topology. The dependency strength characterizes the degree of call dependency between nodes. Based on the dependency strength, multiple candidate root cause nodes 405 are obtained by filtering among multiple nodes. The call data 402 of multiple candidate root cause nodes and the call data 403 of abnormal service nodes are compared with preset benchmark data to obtain multiple node difference features. Based on the degree and type of difference indicated by the multiple node difference features, the root cause node 404 is obtained by filtering among multiple candidate root cause nodes 405 and abnormal service nodes 406.

[0138] According to an embodiment of this application, the device includes a third anomaly detection module. The anomaly detection module is used to extract features from multiple nodes to obtain multiple node feature vectors. The node feature vectors represent the execution state of the nodes. The multiple node feature vectors are processed by a trained third anomaly detection model to obtain multiple third anomaly scores. Based on the degree of anomaly indicated by the multiple third anomaly scores, the device filters among the multiple nodes to obtain an anomaly service node 406.

[0139] According to embodiments of this application, any multiple modules among the feature aggregation module 610, the first anomaly detection module 620, the feature extraction module 630, and the second anomaly detection module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the feature aggregation module 610, the first anomaly detection module 620, the feature extraction module 630, and the second anomaly detection module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the feature aggregation module 610, the first anomaly detection module 620, the feature extraction module 630, and the second anomaly detection module 640 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0140] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing an anomaly detection method according to an embodiment of this application.

[0141] like Figure 7 As shown, an electronic device 700 according to an embodiment of this application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0142] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0143] According to embodiments of this application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0144] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0145] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0146] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the anomaly detection method provided in the embodiments of this application.

[0147] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0148] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0149] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0150] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. An anomaly detection method characterized by, The method comprises: respectively, a plurality of service call chain sets are obtained, and a plurality of statistical feature vectors are determined, wherein each of the plurality of service call chain sets has a matched service node, and the statistical feature vector represents the comprehensive performance of the service call chain set; processing the plurality of statistical feature vectors through the trained first anomaly detection model to obtain a plurality of first anomaly scores, and screening in the plurality of service call chain sets based on the anomaly degree indicated by the plurality of first anomaly scores to obtain a candidate service call chain set; feature extraction is performed on a plurality of candidate service call chains in the candidate service call chain set to obtain a plurality of service feature vectors, wherein the service feature vector represents the execution state of the candidate service call chain; processing the plurality of service feature vectors through the trained second anomaly detection model to obtain a plurality of second anomaly scores, and screening in the candidate service call chain set based on the anomaly degree indicated by the plurality of second anomaly scores to obtain an abnormal service call chain.

2. The method of claim 1, wherein, The method further comprises: forming a service call chain topology by taking a plurality of service nodes in the abnormal service call chain as nodes and taking the call relationship between the plurality of service nodes as edges; abnormal marking is performed on the nodes in the service call chain topology based on the abnormal service nodes indicated by the candidate service call chain set to obtain an abnormal service chain call topology, wherein the abnormal service call chain topology indicates the call path of the abnormal service node.

3. The method of claim 2, wherein, The method further comprises: determining the dependency strength between the abnormal service node and a plurality of adjacent nodes based on the call success rate, the call frequency and the call time indicated by the abnormal service call chain topology, wherein the dependency strength represents the call dependency degree between the abnormal service node and the adjacent nodes; screening in the plurality of adjacent nodes based on the dependency strength to obtain a plurality of candidate root cause nodes; performing feature comparison on the call data of the plurality of candidate root cause nodes and the call data of the abnormal service node with the preset baseline data to obtain a plurality of node difference features; screening in the plurality of candidate root cause nodes and the abnormal service node based on the difference degree and the difference type indicated by the plurality of node difference features to obtain a root cause node.

4. The method of claim 1, wherein, The service call chain set comprises a plurality of service call chains, and for each service call chain set, feature aggregation is performed on the obtained service call chain set to determine a statistical feature vector, which comprises: based on the response time of the plurality of service call chains, calculating the average response time, the percentile response time and the maximum response time of the service call chain set, wherein the average response time represents the response efficiency of the service node matched with the service call chain set, the percentile response time represents the processing performance of the service node matched with the service call chain set, and the maximum response time represents the processing upper limit of the service node matched with the service call chain set; based on the number of call failures and the total number of calls of the plurality of service call chains, obtaining the error rate of the service call chain set, wherein the error rate represents the processing stability of the service node matched with the service call chain set; An abnormal service invocation chain is obtained based on the candidate service invocation chain set, and the abnormal service invocation chain is used to indicate an abnormal service node in the candidate service invocation chain set.

5. The method of claim 1, wherein, The service feature vector includes at least one of a response time, an error identifier, and a message size, the response time representing a calling speed of the candidate service invocation chain, the error identifier representing a calling availability of the candidate service invocation chain, and the message size representing a data transmission amount of the candidate service invocation chain.

6. The method of claim 1, wherein, The method further includes: feature extraction is performed on the plurality of nodes to obtain a plurality of node feature vectors, the node feature vectors representing execution states of the nodes; the plurality of node feature vectors are processed by using the trained third anomaly detection model to obtain a plurality of third anomaly scores, and an abnormal service node is screened out from the plurality of nodes based on anomaly degrees indicated by the plurality of third anomaly scores.

7. The method of claim 6, wherein, The node feature vector includes at least one of a node response time, a node error identifier, and a node message size, the node response time representing a calling speed of the node, the node error identifier representing a calling availability of the node, and the node message size representing a data transmission amount of the node.

8. The method according to claim 6 or 7, characterized in that, The method further includes: based on the abnormal node, consistency verification is performed on an abnormal service node indicated by the candidate service invocation chain set; in a case where the consistency verification is passed, the abnormal service node is mapped to a preset service invocation chain topology based on a service request identifier indicated by the abnormal service invocation chain, the preset service invocation chain topology indicating a service node calling path responding to a service request.

9. An abnormality detection device characterized by comprising: The apparatus includes: a feature aggregation module configured to perform feature aggregation on the obtained plurality of service invocation chain sets respectively to determine a plurality of statistical feature vectors, the plurality of service invocation chain sets each having a matched service node, and the statistical feature vectors representing comprehensive performances of the service invocation chain sets; a first anomaly detection module configured to process the plurality of statistical feature vectors by using a trained first anomaly detection model to obtain a plurality of first anomaly scores, and to screen out a candidate service invocation chain set from the plurality of service invocation chain sets based on anomaly degrees indicated by the plurality of first anomaly scores; a feature extraction module configured to perform feature extraction on a plurality of candidate service invocation chains in the candidate service invocation chain set to obtain a plurality of service feature vectors, the service feature vectors representing execution states of the candidate service invocation chains; and a second anomaly detection module configured to process the plurality of service feature vectors by using a trained second anomaly detection model to obtain a plurality of second anomaly scores, and to screen out an abnormal service invocation chain from the candidate service invocation chain set based on anomaly degrees indicated by the plurality of second anomaly scores.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images