Performance bottleneck diagnosis method and device

Abstract

The invention provides a performance bottleneck diagnosis method and device, and belongs to the technical field of micro-services. According to the method, the key performance index reflecting the operation health state of the micro-service is utilized to evaluate the operation health state of the micro-service, and when the exception is found, the traffic mirroring strategy corresponding to the abnormal micro-service is determined. The traffic mirroring strategy is used for acquiring the mirroring traffic of the abnormal micro-service. According to the on-demand mirroring strategy, only when the performance bottleneck is detected, mirroring sampling is carried out on the flow of the abnormal micro-service causing the performance bottleneck, and compared with collection of full-amount logs or fixed-proportion logs, the sampling data size can be remarkably reduced, and therefore the storage cost and the calculation cost are reduced. And in the mode, mirror image sampling is only carried out on the traffic of the abnormal micro-service, so that normal traffic noise data can be reduced from the source, the noise influence is avoided, and the positioning efficiency and the positioning precision of the performance bottleneck root cause are improved.

Description

Technical Field

[0001] This application relates to the field of microservices technology, and in particular to a method and apparatus for diagnosing performance bottlenecks. Background Technology

[0002] With the widespread adoption of cloud-native technologies, microservice architecture has become the mainstream model for modern application development. Applications built using microservice architecture are called microservice systems. Microservice systems are used to break down an application into multiple functionally independent microservices that can be deployed independently. For example, an e-commerce system includes multiple microservices such as order services, payment services, inventory services, and security services. To ensure the normal operation of a microservice system, performance monitoring and bottleneck diagnosis are necessary for each microservice.

[0003] Currently, a combination of static monitoring and fixed-rendering strategies can be used for performance bottleneck diagnosis. Specifically, full logs or a fixed percentage of logs from the microservice system are collected. When a performance bottleneck occurs, the root cause of the bottleneck is determined based on a pre-defined fixed-rendering strategy and the collected full or fixed-rendering logs. For example, the root cause of the performance bottleneck might be an increase in overall P99 latency.

[0004] However, full log sampling or fixed-ratio log sampling generates massive amounts of data, increasing storage and computation costs. Furthermore, this massive data includes a large amount of normal request noise, which can easily mask key information about performance bottlenecks, leading to poor accuracy in locating the root cause of performance bottlenecks. Summary of the Invention

[0005] This application provides a performance bottleneck diagnosis method and apparatus for implementing an on-demand mirroring strategy. This means that only when a performance bottleneck is detected will specific traffic from the abnormal microservice causing the bottleneck be mirrored, thereby reducing the amount of sampled data and thus lowering storage and computing costs. Furthermore, it can reduce the impact of normal request noise on key information about the performance bottleneck, improving the accuracy of root cause localization.

[0006] In a first aspect, embodiments of this application provide a performance bottleneck diagnosis method, applied to a microservice system including multiple microservices, the method comprising:

[0007] Obtain the key performance indicators (KPIs) corresponding to each of the microservices; the KPIs of the microservices are used to reflect the operational health status of the microservices.

[0008] Based on the key performance indicators of each microservice, identify the abnormal microservice from among the multiple microservices;

[0009] Determine the traffic mirroring strategy corresponding to the abnormal microservice; the traffic mirroring strategy is used to obtain the mirrored traffic of the abnormal microservice.

[0010] Obtain the target mirrored traffic; the target mirrored traffic is the mirrored traffic obtained by performing the mirroring operation corresponding to the traffic mirroring strategy;

[0011] Based on the target mirrored traffic, determine the root cause of the performance bottleneck.

[0012] Optionally, determining the traffic mirroring strategy corresponding to the abnormal microservice includes:

[0013] Based on the time point of the abnormal event of the abnormal microservice and the context information of the abnormal microservice, obtain the upstream and downstream call chain of the abnormal microservice;

[0014] The context information indicates the association information for constructing the upstream and downstream call chains, and the link segment of the upstream and downstream call chains indicates the call link between two nodes in the upstream and downstream call chains; the nodes include microservices or interfaces.

[0015] The anomaly score of each link segment in the upstream and downstream call chain is determined, and the anomaly score of each link segment is correlated with the correlation between the two nodes of that link segment;

[0016] The traffic mirroring strategy is determined based on the anomaly score of each link segment.

[0017] Optionally, determining the anomaly score of each link segment in the upstream and downstream call chains includes:

[0018] The anomaly score for this link segment is calculated using the following formula:

[0019] score=w1·z_score+w2·correlation_shift

[0020] Wherein, score is the anomaly score of the link segment, z_score is the standardized deviation value, indicating the degree to which the actual value of the key performance indicator of the abnormal microservice deviates from the normal baseline value, correlation_shift is the standardized correlation value, indicating the change in correlation between the two nodes corresponding to the link segment before and after the time point of the abnormal event within a preset period, w1 is the first weight value, w2 is the second weight value, w1+w2=1, and both w1 and w2 are greater than 0.

[0021] Optionally, determining the traffic mirroring strategy based on the anomaly score of each link segment includes:

[0022] The link segments in the upstream and downstream call chains with the anomalies in the first k positions are taken as the mirror targets, and the mirror ratio corresponding to the mirror targets is obtained, where k is a positive integer;

[0023] The traffic mirroring strategy is obtained based on the mirroring target and the mirroring ratio.

[0024] Optionally, the method further includes:

[0025] If multiple abnormal microservices depend on the same upstream microservice in the upstream and downstream call chain, the upstream microservice is determined to be faulty.

[0026] Reduce the anomaly of the link segments between the upstream microservice and the multiple downstream microservices.

[0027] Optionally, after obtaining the key performance indicators corresponding to each of the microservices, the method further includes:

[0028] According to the preset sliding time window and step size, the indicator data corresponding to each type of key performance indicator of each microservice is aggregated to obtain multiple time-series aggregated data of each type of key performance indicator of each microservice.

[0029] Within each sliding time window, multiple sets of key performance indicators of the same type for the same microservice are aggregated to obtain the time-series aggregated data of the key performance indicators of the same type for the microservice within the sliding time window.

[0030] The step of identifying abnormal microservices from the plurality of microservices based on the key performance indicators of each microservice includes:

[0031] Based on multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice, abnormal microservices are identified from the multiple microservices.

[0032] Optionally, determining the abnormal microservice among the multiple microservices based on multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice includes:

[0033] Smoothing is performed on multiple time-series aggregated data of the same type of key performance indicators for the same microservice to obtain the smoothed trend value of the key performance indicator corresponding to the microservice.

[0034] The abnormal microservices are determined based on multiple time-series aggregated data and corresponding smoothing trend values ​​of each type of key performance indicator corresponding to each microservice.

[0035] For the same type of key performance indicators in the same microservice, perform the following operations:

[0036] Determine the residual between each time-series aggregated data point and its corresponding smoothed trend value in multiple time-series aggregated data points of the same type of key performance indicators for the same microservice;

[0037] Based on the residual between each time-series aggregated data and the corresponding smoothed trend value, and the pre-acquired residual threshold, determine whether the microservice is the abnormal microservice.

[0038] Optionally, the smoothing process for multiple time-series aggregated data of the same type of key performance indicators for the same microservice includes:

[0039] We use exponentially weighted moving averages to smooth multiple time-series aggregated data of the same type of key performance indicators for the same microservice.

[0040] Optionally, the residual threshold corresponding to each type of key performance indicator for each microservice is positively correlated with the historical residual of that type of key performance indicator for that microservice; the historical residual is the residual obtained before the current diagnosis.

[0041] Secondly, embodiments of this application provide a performance bottleneck diagnosis device applied to a microservice system, the microservice system comprising multiple microservices, the device comprising:

[0042] The acquisition unit is used to acquire the indicator data corresponding to the key performance indicators of each microservice; the key performance indicators of the microservice are used to reflect the operational health status of the microservice.

[0043] The first determining unit is used to determine the abnormal microservice from the plurality of microservices based on the indicator data corresponding to the key performance indicators of each microservice.

[0044] The second determining unit is used to determine the traffic mirroring strategy corresponding to the abnormal microservice; the traffic mirroring strategy is used to obtain the mirrored traffic corresponding to the abnormal microservice.

[0045] A traffic acquisition unit is used to acquire target mirrored traffic, wherein the target mirrored traffic is the mirrored traffic acquired by performing the mirroring operation corresponding to the traffic mirroring strategy;

[0046] A diagnostic unit is used to determine the root cause of the performance bottleneck based on the target mirrored traffic.

[0047] Thirdly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the method in any of the possible implementations of any of the above aspects.

[0048] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program (also referred to as code or instructions) that, when run on a computer, causes the computer to perform the method in any of the possible implementations of any of the above aspects.

[0049] Fifthly, embodiments of this application provide a chip system including one or more processors for calling and executing instructions stored in memory, causing the methods in any of the above aspects or possible implementations to be executed. The chip system may be composed of chips or may include chips and other discrete devices.

[0050] This application provides a performance bottleneck diagnosis method and apparatus. The method utilizes key performance indicators (KPIs) reflecting the operational health of a microservice to assess its health. When an anomaly is detected, a traffic mirroring strategy is determined for the abnormal microservice. This traffic mirroring strategy is used to obtain mirrored traffic from the abnormal microservice. This approach only mirrors the traffic of the abnormal microservice causing the performance bottleneck when one is detected. Compared to collecting full logs or a fixed percentage of logs, this on-demand mirroring strategy significantly reduces the amount of sampled data, thereby reducing storage and computational costs. Furthermore, since this method only mirrors the traffic of the abnormal microservice, it reduces noise data from normal traffic at the source, avoids the impact of noise, and improves the efficiency and accuracy of locating the root cause of the performance bottleneck. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0053] Figure 2 A flowchart of a performance bottleneck diagnosis method provided in an embodiment of this application;

[0054] Figure 3 A flowchart illustrating a method for determining a traffic mirroring strategy for an abnormal microservice, as provided in this application embodiment;

[0055] Figure 4 A flowchart of another performance bottleneck diagnosis method provided in this application embodiment;

[0056] Figure 5 This is a schematic diagram of a performance bottleneck diagnostic device provided in an embodiment of this application. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] A microservice system refers to a distributed application system built on cloud-native technologies and designed with a microservice architecture. A microservice system comprises multiple functionally independent microservices that work together to complete a full business process. For example, an e-commerce system is a microservice system, including multiple functionally independent microservices such as order service, payment service, inventory service, security service, and user service. Among them, the order service, payment service, and inventory service work together to complete the business process of a user placing an order.

[0059] To ensure the normal operation of a microservice system, it is necessary to monitor it. If a performance bottleneck is detected, it is necessary to diagnose the bottleneck and determine its root cause. A performance bottleneck refers to an insufficient processing capacity or response efficiency in a certain part of the microservice system, leading to limited performance, increased request latency, decreased throughput, or increased error rate of the entire microservice system.

[0060] Currently, a combination of static monitoring and a fixed mirroring strategy can be used for performance bottleneck diagnosis. Specifically, full logs or a fixed percentage of logs are collected from the microservice system. For example, only one log entry is retained for every 10 requests. When a performance bottleneck occurs, the root cause of the bottleneck is determined based on the pre-configured fixed traffic mirroring strategy and the fixed collection of full or fixed percentage logs.

[0061] Traffic mirroring strategies are sets of traffic replication and forwarding rules generated to address performance bottlenecks. Their purpose is to accurately replicate abnormal, related traffic to the diagnostic environment without interfering with real traffic, providing data support for root cause analysis of performance bottlenecks. Fixed traffic mirroring strategies refer to strategies with fixed key parameters. For example, the mirroring target is fixed, the mirroring ratio is fixed, and the strategy's effective time is fixed. For instance, for an e-commerce system, the mirroring target is set as: long-term mirroring of the gateway service and order service. The mirroring ratio is: a fixed 5% of traffic mirrored, not dynamically adjusted based on the severity of the anomaly. The strategy's effective time is: continuous mirroring 24 / 7.

[0062] However, current performance bottleneck diagnosis methods have the following problems: collecting full logs or a fixed percentage of logs generates massive amounts of data, increasing storage and computing costs. Furthermore, this massive data includes a large amount of normal request noise, which can mask critical anomalies, causing root cause analysis of performance bottlenecks to remain superficial and unable to pinpoint specific microservices, interfaces, or specific request patterns. In other words, collecting full logs or a fixed percentage of logs results in low accuracy in locating the root cause of performance bottlenecks. For example, only discovering an increase in p99 latency in the order service cannot pinpoint from the collected logs that the increase in p99 latency in the order service is caused by slow database queries in the downstream inventory service. Here, p99 latency refers to the latency of 99% of requests being less than or equal to a certain value within a specified time window. For example, within a specified time window, 99% of requests having a latency of less than or equal to 200ms.

[0063] In view of this, embodiments of this application provide a performance bottleneck diagnosis method. This method utilizes key performance indicators reflecting the operational health status of a microservice to assess its operational health. When an anomaly is detected, a traffic mirroring strategy corresponding to the anomaly microservice is determined. This traffic mirroring strategy is used to obtain mirrored traffic from the anomaly microservice. This method only mirrors the traffic of the anomaly microservice causing the performance bottleneck when it is detected. This on-demand mirroring strategy, compared to collecting full logs or a fixed percentage of logs, significantly reduces the amount of sampled data, thereby reducing storage and computational costs. Furthermore, this method only mirrors the traffic of the anomaly microservice, thus reducing normal request noise at the source, avoiding the impact of noise, and improving the efficiency and accuracy of locating the root cause of performance bottlenecks.

[0064] To enable those skilled in the art to better understand the performance bottleneck diagnosis method provided in this application, a detailed and complete description is provided below in conjunction with the accompanying drawings.

[0065] First, we introduce the application scenarios of the performance bottleneck diagnosis method provided in the embodiments of this application.

[0066] For example, Appendix Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, this application scenario includes a diagnostic system 101 and a microservice system 102. The diagnostic system 101 and the microservice system 102 are connected.

[0067] The microservice system 102 includes multiple microservices. For example, such as... Figure 1 As shown, the microservice system 102 includes microservice A, microservice B and microservice C.

[0068] In one example, the microservice system 102 also includes a traffic middleware for performing mirroring operations based on a traffic mirroring strategy and synchronizing mirrored traffic to shadow endpoints. For example, as... Figure 1 As shown, the traffic middleware is a service mesh or service proxy.

[0069] In this embodiment, the diagnostic system 101 is used to acquire the key performance indicators (KPIs) of each microservice in the microservice system 102; based on the KPIs of each microservice, it identifies abnormal microservices from multiple microservices; and determines the traffic mirroring strategy corresponding to the abnormal microservice. This traffic mirroring strategy is used to acquire mirrored traffic for the abnormal microservice. The diagnostic system 101 is also used to acquire target mirrored traffic and, based on the target mirrored traffic, determine the root cause of the performance bottleneck. The target mirrored traffic is the mirrored traffic acquired by executing the mirroring strategy corresponding to this traffic mirroring strategy.

[0070] For example, such as Figure 1 As shown, the diagnostic system 101 is used to: collect key performance indicator (KPI) data corresponding to microservice A, microservice B, and microservice C. Based on the collected KPI data, it identifies abnormal microservices from microservices A, B, and C. For example, the collected KPI data for microservice A includes: p99 latency of 280ms, error rate of 3.2%, and central processing unit (CPU) utilization of 65%. The KPI data for microservice B includes: p99 latency of 45ms, error rate of 0.05%, and CPU utilization of 50%. The KPI data for microservice C includes: p99 latency of 45ms, error rate of 0.05%, and CPU utilization of 50%. The diagnostic system 101 uses preset abnormal baselines, such as a p99 latency of 50ms, a CPU utilization baseline of 70%, and an error rate baseline of 0.1%. Specifically, if the p99 latency of a microservice is greater than or equal to 50ms, the microservice is determined to be abnormal; if the CPU utilization of a microservice is less than or equal to 70%, the microservice is determined to be abnormal; if the error rate of a microservice is greater than or equal to 0.1%, the abnormal microservice is determined to be microservice A. Then, the diagnostic system 101 determines the traffic mirroring strategy for microservice A. For example, the mirroring target is microservice A, the mirroring ratio is 8%, the strategy is effective for 15 minutes, and the gray-scale phase configuration is [1, 5, 8], that is, it observes in stages of 1%, 5%, and 8% for 5 minutes to avoid sudden load.

[0071] In this embodiment, the diagnostic system 101 is further configured to: distribute the determined traffic mirroring strategy to the traffic middleware of the microservice system 102, and have the traffic middleware execute the mirroring operation according to the traffic mirroring strategy. The diagnostic system 101 is also configured to: obtain mirrored traffic from a shadow endpoint. The shadow endpoint is a read-only endpoint used to receive the mirrored traffic obtained by the traffic middleware based on the traffic mirroring strategy. The diagnostic system 101 is further configured to: locate the root cause of performance bottlenecks by performing link tracing on the mirrored traffic.

[0072] It should be noted that the performance bottleneck diagnosis method provided in this application embodiment can also be applied to other application scenarios. For example, it can be applied to a multi-microservice system scenario. That is, a diagnosis system 101 is connected to multiple microservice systems to determine the performance bottleneck of each microservice system. This application embodiment is not specifically limited.

[0073] The performance bottleneck diagnosis method provided in the embodiments of this application is described below. For ease of understanding by those skilled in the art, the following diagnosis method is illustrated in the appendix. Figure 1 The application scenario shown is used as an example for illustrative explanation.

[0074] Appendix Figure 2 A flowchart of a performance bottleneck diagnosis method provided in this application embodiment is shown. The method includes the following:

[0075] S210 retrieves the key performance metrics data for each microservice.

[0076] Key performance indicators (KPIs) of a microservice are used to reflect its operational health. In this embodiment, KPIs include, but are not limited to, request latency, request success rate, error rate, throughput, and resource utilization.

[0077] Request latency refers to the time difference between a microservice receiving a request and returning a response, such as p99 latency or p95 latency. p95 latency means that within a specified time window, 95% of request latencies are less than or equal to a certain latency value. Request latency reflects the response efficiency of a microservice. If the request latency is stable within a baseline range (e.g., p99 latency less than or equal to 100ms), it indicates that the microservice is responding efficiently, and users experience no lag; that is, the microservice is functioning normally. If the latency suddenly increases, for example, if the p99 latency rises above the baseline range (from 50ms to 300ms), it indicates that the microservice's processing capacity has decreased, and user operations are experiencing lag; this microservice is an abnormal microservice.

[0078] The success rate of a microservice's request refers to the percentage of requests successfully processed by the microservice within a specified time window out of the total number of requests. The error rate refers to the percentage of requests that failed within the microservice within a specified time window out of the total number of requests. The success rate and error rate of a microservice reflect its availability and business correctness, and are direct indicators of service stability.

[0079] The throughput of a microservice refers to the total number of requests processed by the microservice per unit of time. Throughput is commonly expressed in Queries Per Second (QPS). For example, an order service with a QPS of 500 means it processes 500 order requests per second. Throughput reflects the carrying capacity of a microservice and is a metric for the matching of resource allocation with traffic.

[0080] Resource utilization of a microservice refers to the percentage of hardware / software resources used for the operation of the microservice. In this embodiment, resource utilization can be at least one of CPU utilization, memory utilization, and database connection pool utilization. For example, a CPU utilization of 85% for a microservice means that the microservice uses 85% of the total allocated CPU cores. Resource utilization reflects the resource support capability for the operation of a microservice.

[0081] In this embodiment of the application, the diagnostic system can accurately identify abnormal microservices from multiple microservices based on the indicator data corresponding to key performance indicators.

[0082] In one specific implementation, the diagnostic system can obtain key performance indicators (KPIs) for each microservice within the microservice system from various data sources. For example, it can obtain request latency, throughput, request success rate, or request error rate for each microservice from the microservice system's service broker or service mesh. It can also obtain application-layer performance data from an Application Performance Monitoring (APM) system. This APM data reflects the internal operational status of the microservices, such as the number of active threads in the thread pool. Finally, it can obtain resource utilization from basic monitoring settings. For instance, the diagnostic system can use Prometheus / OTel Collector to obtain key performance indicator KPIs from various data sources according to preset rules.

[0083] In this embodiment, a preset rule is used, for example, to collect indicator data once at preset collection intervals. The preset duration can be set by those skilled in the art as needed, for example, a preset duration of 10 seconds. Alternatively, indicator data can be collected in real time, but this embodiment is not limited to this.

[0084] S220 identifies abnormal microservices from multiple microservices based on the key performance indicators of the microservices.

[0085] The diagnostic system can identify abnormal microservices from multiple microservices based on the key performance indicators (KPIs) of each microservice. For example, the diagnostic system can pre-set anomaly rules and identify abnormal microservices based on these rules. For instance, anomaly rules may include at least one of the following: a sudden increase in microservice latency, an error rate exceeding an error rate baseline, and an abnormal decrease in throughput.

[0086] In this embodiment of the application, in order to improve the accuracy of identifying abnormal microservices, the diagnostic system can comprehensively consider the microservice's request latency, request success rate, error rate, throughput, and resource utilization.

[0087] For example, if the exception rules are: within a specified time window, p99 latency exceeds the latency baseline by 1.3 times, request success rate is below 99.9%, error rate exceeds 0.1%, throughput exceeds 60% or 200% of the historical average for the same period, and resource utilization exceeds 80%. Figure 1 As shown, the key performance indicators (KPIs) for microservice A are as follows: p99 latency is 180ms, request success rate is 98.5%, error rate is 1.5%, throughput is 1200 (historical average is 1000), and resource utilization is 85%. The KPIs for microservice B are as follows: p99 latency is 55ms, request success rate is 99.97%, error rate is 0.03%, throughput is 800 (historical average is 1000), and resource utilization is 85%. The KPIs for microservice C are as follows: p99 latency is 60ms, request success rate is 99.95%, error rate is 0.05%, throughput is 900 (historical average is 1000), and resource utilization is 70%. Microservice A has 4 abnormal KPIs, microservice B has 1 abnormal KPI, and microservice C has no abnormal KPIs.

[0088] Furthermore, due to the strong correlation between abnormal metrics in microservice A—namely, resource utilization exceeding 85% (resource overload) leading to increased P99 latency, which in turn causes request processing failures, resulting in an increase in error rate and a decrease in success rate—microservice A exhibits a complete causal chain where resource overload leads to performance degradation, and performance degradation leads to business anomalies. Microservice B, however, does not exhibit this business correlation; therefore, the abnormal microservice can be identified as microservice A.

[0089] The embodiments of this application can determine abnormal microservices from multiple microservices through the above-mentioned multi-source data fusion. Compared with the method of determining abnormal microservices through a single data source, this method can improve the accuracy of determining abnormal microservices.

[0090] The embodiments of this application can also identify abnormal microservices from multiple microservices in other ways, and the embodiments of this application are not limited thereto.

[0091] S230, determine the traffic mirroring strategy for the abnormal microservice, wherein the traffic mirroring strategy is used to obtain the mirrored traffic corresponding to the abnormal microservice.

[0092] In this embodiment of the application, the traffic mirroring policy includes: mirroring target, mirroring ratio, target interface, policy effective time, mirroring percentage during the gray-scale phase, and rollback threshold. Furthermore, the traffic mirroring policy may also include: minimizing or tokenizing the template name, allowing mirroring via Hypertext Transfer Protocol (HTTP) methods, shadow endpoints, RPS guardrails, policy version number, policy signature, or policy verification value.

[0093] In this application embodiment, different microservices employ different traffic mirroring strategies. For example, such as... Figure 1 As shown, the traffic mirroring strategy for microservice A is as follows: the mirroring target is microservice A and microservice B associated with microservice A, the mirroring ratio is set to 5%, and the rollback threshold is set to rollback after 65ms. The traffic mirroring strategy for microservice B is as follows: the mirroring target is microservice B and microservice A associated with microservice B, the mirroring ratio is set to 8%, and the rollback threshold is set to rollback after 70ms. The traffic mirroring strategy for microservice C is as follows: the mirroring target is microservice C and third-party interfaces associated with microservice C, the mirroring ratio is set to 10%, and the rollback threshold is set to rollback after 104ms. After the diagnostic system identifies the abnormal microservice, it can determine the corresponding traffic mirroring strategy based on the abnormal microservice. This on-demand traffic mirroring strategy provided in this embodiment can ensure that sufficient diagnostic information is captured while reducing the additional overhead caused by traffic mirroring.

[0094] Furthermore, the diagnostic system can dynamically generate traffic mirroring strategies corresponding to abnormal microservices based on their abnormal signals. For example, if the abnormal signal indicates an abnormal severity level, the diagnostic system pre-defines the mapping relationship between the severity level and the mirroring ratio to determine the mirroring ratio. For instance, if the severity level is minor, the mirroring ratio is 5%; if the severity level is normal, the mirroring ratio is 8%; and if the severity level is severe, the mirroring ratio is 10%. This dynamic adjustment of the traffic mirroring strategy helps improve the accuracy of locating the root cause of performance bottlenecks.

[0095] Furthermore, embodiments of this application can also determine traffic mirroring strategies based on other methods, as detailed below. Figure 3 As shown, the embodiments in this application will not be described in detail.

[0096] This traffic mirroring strategy is understandable; it's an on-demand mirroring approach, meaning that traffic from the abnormal microservice causing the bottleneck is only mirrored when a performance bottleneck is detected. This approach helps reduce the amount of data collected, lowering storage and computing costs.

[0097] S240, Obtain target mirrored traffic, wherein the target mirrored traffic is the mirrored traffic obtained by performing the mirroring operation corresponding to the traffic mirroring strategy.

[0098] In this embodiment, after determining the traffic mirroring strategy, the diagnostic system distributes the strategy to the traffic middleware (specifically, a service mesh or service proxy) of the microservice system. The traffic middleware performs the mirroring operation and sends the acquired target mirrored traffic to the shadow endpoint. The diagnostic system then obtains the target mirrored traffic from the shadow endpoint.

[0099] In one specific implementation, the traffic middleware can render traffic mirroring policies as virtual services or target rules. Virtual services define traffic routing and mirroring rules, while target rules define access policies for accessing the mirrored target service. Then, the rendered object is checked for compliance, including syntax correctness checks, resource relationship checks, and permission validity checks. Next, the traffic middleware performs phased mirroring according to the canary phase configuration and policy effective time. For example, if the traffic mirroring policy canary phase configuration is [1, 5, 8], and the policy effective time is 15 minutes, the traffic middleware can observe the policy in phases of 1%, 5%, and 8% for 5 minutes each. The traffic middleware monitors the key performance indicators corresponding to the source service and shadow endpoints in real time, and determines whether to trigger a rollback operation based on a preset rollback threshold. If a rollback operation is triggered, the current mirroring configuration is automatically revoked, and the system rolls back to the previous policy version.

[0100] In this embodiment, the traffic middleware can also be configured with concurrency / rate and circuit breaker mechanisms to avoid cascading effects. Configuring concurrency / rate refers to setting a hard limit on mirrored traffic to prevent shadow endpoint failures from negatively impacting the source service or traffic middleware. The circuit breaker mechanism is a protection mechanism that automatically suspends the forwarding of mirrored traffic to a shadow endpoint or abnormal microservice when it experiences frequent failures. For sensitive fields, minimization and tokenized template names can be implemented for privacy protection. Furthermore, if privacy protection fails, mirroring is stopped and an alert is issued, with audit logs retained.

[0101] For example, the traffic middleware can render traffic mirroring policies as Istio Virtual Services or Destination Rules, and perform pre-checks using istioctl analyze. Concurrency / rate and circuit breaking can be configured through Envoy Filter to avoid cascading effects. Sensitive fields are minimized or tokenized using Envoy Lua / WASM template names on the data plane.

[0102] To avoid the impact of mirroring requests on business status, in this embodiment of the application, idempotent methods such as Get can be mirrored; for creating or updating resources, it must be ensured that the shadow endpoint has no side effects, or the application can identify shadow endpoint requests and convert them into read-only paths; and retries can be disabled for the mirrored target.

[0103] S250 identifies the root cause of performance bottlenecks based on target mirror traffic.

[0104] In this embodiment of the application, the diagnostic system determines the root cause of the performance bottleneck based on the target mirror traffic.

[0105] In one example, the diagnostic system first reconstructs the complete call path of the request across microservices and the time distribution of each stage. Next, the system examines the request's parameter information to determine if the request is related to a specific input, thus eliminating interference from general requests. If the request is not related to a specific input, the system combines this with service resource usage at the time the mirrored traffic occurred to determine if the performance bottleneck is caused by insufficient resources, thereby ruling out performance bottlenecks caused by resource overload. If the performance bottleneck is not caused by insufficient resources, the system further statistically analyzes the error types and frequencies in the mirrored traffic to ultimately pinpoint the root cause of the performance bottleneck. This distributed elimination method narrows down the range of root causes layer by layer, avoiding the problem of noise masking key information and resulting in insufficient localization accuracy.

[0106] In another example, the diagnostic system can reconstruct the request waterfall chart and identify critical paths using distributed tracing logs as the main framework. The request waterfall chart is a visual chart with time on the horizontal axis and the call sequence of microservices on the vertical axis. It visually displays the call order, start and end times, and duration of each stage of a single request within the microservices. The critical path refers to the call chain with the longest total duration in the request waterfall chart. It is the core link that determines the upper limit of the overall request response time. An increase in the duration of any stage on this path will directly lead to an increase in overall request latency, and it is the core focus of the diagnostic module in the document for locating performance bottlenecks.

[0107] In the critical path, the N longest-lasting segments are identified as the core bottleneck units, where N is a positive integer. One segment corresponds to one independent call. For example, in the call chain of microservice A processing a request, there are three segments: microservice A receives the user request, taking 10ms; microservice A calls microservice B, taking 200ms; and microservice B calls the database, taking 50ms. If N=1, then the core bottleneck unit is the call from microservice A to microservice B.

[0108] The diagnostic system aggregates error codes or exception stacks to determine the error rate of core bottleneck units. The failure rate of a core bottleneck unit is calculated as: (Number of failed requests in the core bottleneck unit / Total number of requests) × 100%. The diagnostic system also uses resource utilization to eliminate the root causes of resource insufficiency.

[0109] Furthermore, if the core bottleneck unit involves database calls, the root cause of the query layer can be identified by analyzing key slow information and index hit rate in conjunction with the database slow query logs. Key slow information includes, but is not limited to, execution time, number of rows scanned, and lock wait time.

[0110] Furthermore, if the diagnostic system includes a historical problem database, this database stores historical root cause cases. Template / keyword-based similarity comparisons can be performed on the historical problem database to accelerate the localization process. The system outputs the top N1 root causes, where N1 is a positive integer, for example, N1=3.

[0111] Furthermore, the historical problem database also includes the confidence level corresponding to the historical root causes, and the diagnostic system can also output the confidence level of the root causes.

[0112] This application provides a performance bottleneck diagnosis method. This method utilizes key performance indicators reflecting the operational health of a microservice to assess its health status. When an anomaly is detected, a traffic mirroring strategy corresponding to the anomaly microservice is determined. This traffic mirroring strategy is used to obtain mirrored traffic from the anomaly microservice. This method only mirrors the specific traffic of the anomaly microservice causing the performance bottleneck when it is detected. This on-demand mirroring strategy, compared to collecting full logs or a fixed percentage of logs, reduces the amount of sampled data, thereby reducing storage and computational costs. Furthermore, this method only mirrors the specific traffic of the anomaly microservice, thus reducing noise data in normal traffic at the source, avoiding the impact of noise, and improving the efficiency and accuracy of locating the root cause of the performance bottleneck.

[0113] In one specific implementation, step S230 can be further refined to construct the upstream and downstream call chain of the abnormal microservice based on the abnormal signal and context information of the abnormal microservice, and adaptively adjust the traffic mirroring strategy according to the upstream and downstream call chain to avoid the problem that the traffic mirroring strategy cannot adjust the monitoring granularity and scope with real-time changes in load, error rate, etc., thereby causing resource waste in the microservice system during the stable period and difficulty in capturing key information during the high-incidence period of the microservice system anomalies.

[0114] Appendix Figure 3 A flowchart of a method for determining a traffic mirroring strategy for an abnormal microservice is provided in an embodiment of this application. The method includes steps S310 to S340.

[0115] S310: Obtain the upstream and downstream call chain of the abnormal microservice based on the time point and context information of the abnormal event of the abnormal microservice.

[0116] The context information is used to construct the association information of upstream and downstream call chains. The link segments of the upstream and downstream call chains indicate the call link units between two nodes in the upstream and downstream call chains; the nodes include microservices or interfaces. In the embodiments of this application, the context information is a request corresponding to the time point of the abnormal event, from the request entry point to the call relationship of all downstream dependencies and the call sequence.

[0117] In this embodiment, the diagnostic system can obtain multi-source data for a preset time period before and after the time point T of the abnormal event. For example, it can obtain multi-source data for T ± Δ (e.g., Δ represents 3 time periods). The multi-source data includes, but is not limited to, service mesh telemetry data, distributed tracing data, and APM application performance data. The diagnostic system obtains context information from the multi-source data. Based on this context information, the diagnostic system can reconstruct the complete upstream and downstream call chain. Specifically, it restores the call relationship and timing from the entry point to all downstream dependencies of a single request into a directed acyclic graph (DAG), which represents the upstream and downstream call chain of the abnormal microservice.

[0118] S320, determine the anomaly score of each link segment in the upstream and downstream call chain. The anomaly score of each link segment is used to describe the correlation between the two nodes in that link segment.

[0119] The diagnostic system can determine the anomaly score of a link segment based on the correlation between the two nodes in each link segment. The anomaly score of a link segment is positively correlated with the correlation between the two nodes; the more correlated the two nodes are, the higher the anomaly score of the link segment. Specifically, the correlation between upstream node A and downstream node B refers to the impact of performance changes in upstream node A on downstream node B.

[0120] In one specific implementation, the anomaly score of each link segment in the upstream and downstream call chain is calculated using formula (1):

[0121] score=w1·z_score+w2·correlation_shift(1)

[0122] Here, `score` represents the anomaly score of the link segment, `z_score` is the standardized deviation value, indicating the degree to which the actual value of the key performance indicator of the abnormal microservice deviates from the normal baseline value, `correlation_shift` is the standardized correlation value, indicating the change in correlation between the two nodes corresponding to the link segment over a preset period before and after the time point of the abnormal event, `w1` is the first weight value, `w2` is the second weight value, `w1+w2=1`, and both `w1` and `w2` are greater than 0. This fusion of single-point deviation and inter-node correlation changes can avoid misjudgment based on a single dimension and improve the accuracy of root cause localization.

[0123] In another specific implementation, if multiple downstream microservices simultaneously fail in the upstream and downstream call chain, and all of these downstream microservices depend on the same upstream microservice, the upstream microservice is identified as the faulty component; this reduces the anomaly score of the link segments corresponding to the multiple downstream microservices. For example, regarding... Figure 1 The upstream and downstream call chains shown are: upstream microservice S → microservice A, microservice B, and microservice C. If the anomaly score A for the S → A link segment is 1.67, the anomaly score B for the S → B link segment is 1.57, and the anomaly score C for the S → C link segment is 1.48, a weighting factor can be set. For example, if the weighting factor is 0.3, score A is adjusted to 0.5, score B to 0.47, and score C to 0.44. After weighting, the anomaly scores of downstream link segments are reduced. This method helps avoid the diagnostic system misjudging downstream links with linked anomalies as independent anomalies, ensuring that subsequent image acquisition targets focus on upstream microservice S. This method can reduce invalid images and waste of diagnostic resources.

[0124] S330 determines the mirror target and mirror ratio based on the anomaly score of each link segment.

[0125] In one specific implementation, the link segments in the upstream and downstream call chains where the exception occurs within the first k positions are used as mirror targets, and the mirror ratio corresponding to the mirror targets is obtained. Here, k is a positive integer.

[0126] Specifically, for the mirrored target, the mirroring ratio can be determined according to a preset ratio rule. For example, the preset ratio rule is that if the mirrored target is under high load, the mirroring ratio is low, and if the mirrored target is under low load, the mirroring ratio is high. For example, the mirroring ratio = min((abnormal request ratio × 1.2), (1 - current service resource utilization / threshold)).

[0127] S340. Determine the traffic mirroring strategy based on the mirroring target, mirroring ratio, and context information.

[0128] In one example, the diagnostic system can determine the policy activation time based on the duration of the anomaly in the context information. The policy activation time is positively correlated with the anomaly duration; for example, policy activation time = 1.5 × anomaly duration. This approach ensures that the policy activation time covers the anomaly fluctuation cycle, thereby improving the accuracy of root cause identification.

[0129] Furthermore, if the context information shows a sudden drop in the image traffic of an abnormal microservice, a traffic threshold trigger condition can be added. When the image traffic drops to the traffic threshold, the image operation will be automatically stopped, thereby avoiding the waste of resources caused by long-term image processing.

[0130] In this embodiment, the diagnostic system can derive a traffic mirroring strategy based on the mirroring target and mirroring ratio, as well as other strategies generated based on context information. This traffic mirroring strategy can minimize the additional overhead caused by traffic mirroring while capturing sufficient diagnostic information.

[0131] In summary, this embodiment of the application determines the mirroring target and mirroring ratio by constructing upstream and downstream call chains and anomaly scores for each link segment, and then adaptively generates a traffic mirroring strategy by combining context information. This ensures that the generated traffic mirroring strategy can capture sufficient diagnostic information while minimizing the additional overhead brought by traffic mirroring.

[0132] Since performance bottlenecks in microservice systems often manifest as instantaneous or sporadic fluctuations, this application provides another performance bottleneck diagnosis method to further improve the accuracy of root cause localization. In this method, key performance indicators (KPIs) for each microservice are collected at preset intervals (referred to as raw data). The raw data is then aggregated for each type of KPI according to a preset sliding time window and step size, resulting in multiple time-series aggregated data for each type of KPI. A detailed analysis follows with reference to the accompanying drawings.

[0133] Appendix Figure 4 This application provides a flowchart of another performance bottleneck diagnosis method, which is applied to... Figure 1 The following is an illustrative example of the application scenario shown: This method includes the following:

[0134] The S410 acquires key performance metrics data for each microservice by pre-setting a collection time interval.

[0135] The preset collection duration is set by those skilled in the art as needed. For example, the preset collection duration is 10 seconds, that is, every 10 seconds, the key performance indicators corresponding to each microservice are obtained.

[0136] S420 aggregates the data of each type of key performance indicator for each microservice according to the preset sliding time window and step size, and obtains multiple time-series aggregated data of each type of key performance indicator for each microservice.

[0137] Within each sliding time window, multiple sets of key performance indicators of the same type for the same microservice are aggregated to obtain the time-series aggregated data of that type of key performance indicator for that microservice within that sliding time window.

[0138] For example, for microservice A, its key performance indicator is p99 latency. The sliding time window is 60 seconds with a step size of 10 seconds, meaning there is a 10-second interval between two sliding time windows. The diagnostic system monitors the p99 latency of microservice A, collecting data every 10 seconds. In sliding time window 1 [00:00-01:00], the collected metric data are 58ms, 62ms, 65ms, 59ms, 63ms, and 61ms. Therefore, the time-series aggregated data of the p99 latency of microservice A in sliding time window 1 can be (58+62+65+59+63+61)÷6=61.33ms. During the sliding time window 2 [00:10-01:10], the collected indicator data are 62ms, 65ms, 59ms, 63ms, 61ms, and 64ms. The time series aggregated data generated during the sliding time window 2 can be (62+65+59+63+61+64)÷6=62.33ms.

[0139] It is understandable that the aggregated time-series data can avoid noise caused by instantaneous or occasional jitter and better reflect the actual performance level of microservices within the sliding time window.

[0140] In one specific implementation, the diagnostic system can first perform data cleaning and standardization on the raw collected indicator data. Specifically, this involves handling missing values, filtering outliers, and / or standardizing the indicator data to obtain standardized data. Using standardized data for aggregation can improve aggregation accuracy.

[0141] Specifically, the diagnostic system can perform forward imputation on the indicator data, filling in missing data with the most recent valid values ​​and marking the data source; and perform 99% pruning and noise reduction on the indicator data, removing 99% of extreme values ​​to avoid sudden abnormal interference. Furthermore, the diagnostic system can also perform robust standardization on the same indicator data within a sliding time window using the median and absolute median difference. See formulas (2) to (3) for specific processing methods:

[0142] (2)

[0143] (3)

[0144] Furthermore, to align with the standard deviation σ of the normal distribution, the diagnostic system also uses formula (4) for scaling:

[0145] (4)

[0146] Where x* represents the standardized time-series aggregated data, and x represents the unstandardized time-series aggregated data. MAD is the absolute median difference. median is the median of the same indicator data within the sliding time window.

[0147] S430 identifies abnormal microservices from multiple microservices based on multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice.

[0148] Specifically, the following processing is performed on the same type of key performance indicators for the same microservice:

[0149] Step 1: Smooth the multiple time-series aggregated data of the key performance indicator to obtain the smoothed trend value of the key performance indicator.

[0150] In this embodiment, the diagnostic system uses an exponentially weighted moving average to smooth multiple time-series aggregated data of the same type of key performance indicators for the same microservice.

[0151] For example, the time-series aggregated data is smoothed using formula (5).

[0152] (5)

[0153] Where α is a number greater than 0 and less than 1, for example, α = 0.3, t is the current sliding time window, s(t) is the smoothed value within the current sliding time window, R is the time-series aggregated data within the current sliding time window, and s(t-1) is the smoothed value corresponding to the previous sliding time window. In this embodiment, t is an integer greater than 1, and the initial smoothed value is preset.

[0154] By smoothing the data, it becomes less susceptible to occasional fluctuations and better reflects the performance trends of microservices.

[0155] Step 2: Determine the residuals of each time series aggregate data point and its corresponding smoothed trend value in multiple time series aggregate data points for the key performance indicator.

[0156] The formula for calculating the residual is shown in formula (6):

[0157] r(t) = standardized value - s(t) (6)

[0158] Where r(t) is the residual, and the standardized value is the value of the time series aggregated data after normalization, which is used to eliminate the problem of poor determination accuracy caused by the difference in magnitude between the time series aggregated data and the smoothed value s(t).

[0159] Step 3: Based on the residual between each time series aggregated data and the corresponding smoothed trend value, and the pre-acquired residual threshold, determine whether the microservice is an abnormal microservice.

[0160] Among them, the residual threshold corresponding to each type of key performance indicator for each microservice is positively correlated with the historical residual of that type of key performance indicator for that microservice; the historical residual is the residual that was obtained before the current diagnosis.

[0161] For example, the residual threshold θ = residual mean + 3 × residual standard deviation. Or, the residual threshold θ = median(r) + 3 × 1.4826 × MAD(r). Where r is the set of historical residuals.

[0162] In this embodiment of the application, a true anomaly is considered to occur only when two or more consecutive windows exceed the threshold, thus avoiding false alarms due to occasional fluctuations.

[0163] Furthermore, after identifying the abnormal microservice, an abnormal event can be generated. In this embodiment, the abnormal event includes: the identifier of the abnormal microservice, the type of abnormal indicator, the severity score of the abnormality, and the time point and context information of the abnormal event.

[0164] In this embodiment of the application, abnormal events can be simultaneously written to a time-series database for historical analysis and to a message queue for real-time processing.

[0165] In one specific implementation, the diagnostic system also involves fault tolerance mechanisms or service level agreement constraints, including but not limited to: for each sliding time window's microservice metrics, the total time consumed from data input to anomaly score calculation must be less than or equal to a preset timeout, such as 1 second; in case of collection failure, the most recent valid value is used and tagged, without generating anomaly events to avoid invalid anomaly alarms; if message delivery fails and still fails after M retries, local disk persistence is downgraded, i.e., data is delivered to downstream modules to prevent diagnostic data loss. Here, M is a positive integer, for example, M=3.

[0166] S440, determine the traffic mirroring strategy corresponding to the abnormal microservice.

[0167] The diagnostic system is also used to receive abnormal events, which can be analyzed using preset rules or machine learning models to identify services or interfaces that may have performance problems, such as detecting sudden increases in latency, error rates exceeding thresholds, and abnormal decreases in throughput.

[0168] Furthermore, in this embodiment, the diagnostic system takes multi-source data from three sliding time windows before T and three sliding time windows after T, based on the sliding time window T where the abnormal event occurs. It then obtains context information and reconstructs the complete upstream and downstream call chain. Specifically, it restores the call relationships and timing of a single request from the entry point to all downstream dependencies into a directed acyclic graph, calculates the score of each link segment in the abnormal chain, and finally mirrors the top k segments with the highest scores to obtain the traffic mirroring strategy.

[0169] Furthermore, the policy fault tolerance mechanism set in this application embodiment includes, but is not limited to: policy generation time being less than or equal to a preset generation time (e.g., 1 second) to ensure real-time response and avoid delays in diagnosis; budget conflicts are pruned according to severity to prune mirror targets. The preset conflict severity is divided into severity s1 corresponding to the abnormal microservice itself, severity S2 corresponding to the core dependencies of the abnormal microservice, and severity s3 corresponding to the non-core dependencies of the abnormal microservice, based on their correlation with the root cause of the performance bottleneck. s1 has the highest severity, and s3 has the lowest severity; when a service fails, the previous version policy is maintained and marked as read-only to prevent fault propagation from interrupting mirror operations.

[0170] S450, Obtain target mirrored traffic, wherein the target mirrored traffic is the mirrored traffic obtained by performing the mirroring operation corresponding to the traffic mirroring strategy.

[0171] For details, please refer to S240, which will not be repeated here.

[0172] S460 identifies the root cause of performance bottlenecks based on target mirror traffic.

[0173] For details, please refer to S250, which will not be repeated here.

[0174] The performance bottleneck diagnosis method provided in this application uses three mechanisms—sliding time window aggregation, smoothing, and residual dynamic threshold—to identify abnormal microservices. This can avoid interference from instantaneous or occasional jitters on the diagnosis results from the data level, thereby improving the accuracy of root cause localization of performance bottlenecks.

[0175] In addition, this application embodiment also provides a performance bottleneck diagnostic device.

[0176] Appendix Figure 5 This is a schematic diagram of a performance bottleneck diagnostic device provided in an embodiment of this application. The device 500 includes:

[0177] The acquisition unit 501 is used to acquire the indicator data corresponding to the key performance indicators of each microservice; the key performance indicators of the microservice are used to reflect the operational health status of the microservice.

[0178] The first determining unit 502 is used to determine the abnormal microservice from the plurality of microservices based on the indicator data corresponding to the key performance indicators of each microservice.

[0179] The second determining unit 503 is used to determine the traffic mirroring strategy corresponding to the abnormal microservice; the traffic mirroring strategy is used to obtain the mirrored traffic corresponding to the abnormal microservice.

[0180] Traffic acquisition unit 504 is used to acquire target mirrored traffic, wherein the target mirrored traffic is the mirrored traffic acquired by performing the mirroring operation corresponding to the traffic mirroring strategy;

[0181] The diagnostic unit 505 is used to determine the root cause of the performance bottleneck based on the target mirror traffic.

[0182] Optionally, determining the traffic mirroring strategy for the abnormal microservice includes:

[0183] Based on the time point and context information of the abnormal event of the abnormal microservice, obtain the upstream and downstream call chain of the abnormal microservice;

[0184] The context information indicates the association information for constructing the upstream and downstream call chains, and the link segment of the upstream and downstream call chains indicates the call link unit between two nodes in the upstream and downstream call chains; the nodes include microservices or interfaces.

[0185] Determine the anomaly score of each link segment in the upstream and downstream call chain. The anomaly score of each link segment is used to describe the correlation between the two nodes in that link segment.

[0186] The traffic mirroring strategy is determined based on the anomaly score of each link segment.

[0187] Optionally, determining the anomaly score of each link segment in the upstream and downstream call chains includes:

[0188] The anomaly score for this link segment is calculated using the following formula:

[0189] score=w1·z_score+w2·correlation_shift

[0190] Wherein, score is the anomaly score of the link segment, z_score is the standardized deviation value, indicating the degree to which the actual value of the key performance indicator of the abnormal microservice deviates from the normal baseline value, correlation_shift is the standardized correlation value, indicating the change in correlation between the two end nodes corresponding to the link segment within a specified time window before and after the time point of the abnormal event, w1 is the first weight value, w2 is the second weight value, w1+w2=1, and both w1 and w2 are greater than 0.

[0191] Optionally, determining the traffic mirroring strategy based on the anomaly score of each link segment includes:

[0192] The link segments in the upstream and downstream call chains where the anomaly is located in the first k positions are taken as the mirror targets, and the mirror ratio corresponding to the mirror targets is obtained; where k is a positive integer;

[0193] The traffic mirroring strategy is obtained based on the mirroring target and the mirroring ratio.

[0194] Optionally, the first determining unit 502 is further configured to:

[0195] If multiple downstream microservices simultaneously fail in the upstream and downstream call chain, and all of the downstream microservices depend on the same upstream microservice, the upstream microservice is determined to be faulty.

[0196] Reduce the anomaly score of the link segments corresponding to the upstream service and the multiple downstream microservices.

[0197] Optionally, the device 500 further includes: an aggregation unit for:

[0198] According to the preset sliding time window and step size, each type of key performance indicator of each microservice is processed to obtain multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice.

[0199] Within each sliding time window, multiple sets of key performance indicators of the same type for the same microservice are aggregated to obtain the time-series aggregated data of the key performance indicators of the same type for the microservice within the sliding time window.

[0200] The step of identifying abnormal microservices from the plurality of microservices based on the key performance indicators of each microservice includes:

[0201] Based on multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice, abnormal microservices are identified from the multiple microservices.

[0202] Optionally, determining the abnormal microservice among the multiple microservices based on multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice includes:

[0203] Smoothing is performed on multiple time-series aggregated data of the same type of key performance indicators for the same microservice to obtain the smoothed trend value of the key performance indicator corresponding to the microservice.

[0204] The abnormal microservices are determined based on multiple time-series aggregated data and corresponding smoothing trend values ​​of each type of key performance indicator corresponding to each microservice.

[0205] For the same type of key performance indicators in the same microservice, perform the following operations:

[0206] Determine the residual between each time-series aggregated data point and its corresponding smoothed trend value in multiple time-series aggregated data points of the same type of key performance indicators for the same microservice;

[0207] Based on the residual between each time-series aggregated data and the corresponding smoothed trend value, and the pre-acquired residual threshold, determine whether the microservice is the abnormal microservice.

[0208] Optionally, the smoothing process for multiple time-series aggregated data of the same type of key performance indicators for the same microservice includes:

[0209] We use exponentially weighted moving averages to smooth multiple time-series aggregated data of the same type of key performance indicators for the same microservice.

[0210] Optionally, the residual threshold corresponding to each type of key performance indicator for each microservice is positively correlated with the historical residual of that type of key performance indicator for that microservice; the historical residual is the residual obtained before the current diagnosis.

[0211] This application provides a performance bottleneck diagnosis device. This device utilizes key performance indicators reflecting the operational health status of a microservice to assess its health. When an anomaly is detected, it determines the traffic mirroring strategy corresponding to the abnormal microservice. This traffic mirroring strategy is used to obtain mirrored traffic from the abnormal microservice. This on-demand mirroring strategy, which only mirrors specific traffic from the abnormal microservice causing the performance bottleneck when a performance bottleneck is detected, significantly reduces the amount of sampled data compared to full logs or logs sampled at a fixed ratio, thereby reducing storage and computational costs. Furthermore, this method only mirrors specific traffic from the abnormal microservice, reducing noise data from normal traffic at the source, avoiding the impact of noise, and improving the efficiency and accuracy of locating the root cause of the performance bottleneck.

[0212] According to the method provided in the embodiments of this application, this application also provides a chip system, which includes one or more processors for calling and executing instructions stored in memory, thereby causing the method described in the embodiments of this application to be executed. The chip system may be composed of chips or may include chips and other discrete devices.

[0213] The chip system may include input circuits or interfaces for transmitting information or data, and output circuits or interfaces for receiving information or data.

[0214] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the various steps or processes executed by the network device or terminal device in any of the foregoing method embodiments.

[0215] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the various steps or processes executed by the network device or terminal device in any of the foregoing method embodiments.

[0216] The computer-readable storage medium may be the aforementioned volatile memory or non-volatile memory, or it may include both volatile memory and non-volatile memory.

[0217] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0218] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated.

[0219] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

Claims

1. A method for diagnosing performance bottlenecks, characterized in that, Applied to a microservice system comprising multiple microservices, the method includes: Obtain the key performance indicators (KPIs) corresponding to each of the microservices; the KPIs of the microservices are used to reflect the operational health status of the microservices. Based on the key performance indicators of each microservice, identify the abnormal microservice from among the multiple microservices; Determine the traffic mirroring strategy corresponding to the abnormal microservice; the traffic mirroring strategy is used to obtain the mirrored traffic of the abnormal microservice. Obtain the target mirrored traffic; the target mirrored traffic is the mirrored traffic obtained by performing the mirroring operation corresponding to the traffic mirroring strategy; Based on the target mirrored traffic, determine the root cause of the performance bottleneck.

2. The method according to claim 1, characterized in that, The step of determining the traffic mirroring strategy corresponding to the abnormal microservice includes: Based on the time point of the abnormal event of the abnormal microservice and the context information of the abnormal microservice, obtain the upstream and downstream call chain of the abnormal microservice; The context information indicates the association information for constructing the upstream and downstream call chains, and the link segment of the upstream and downstream call chains indicates the call link between two nodes in the upstream and downstream call chains; the nodes include microservices or interfaces. The anomaly score of each link segment in the upstream and downstream call chain is determined, and the anomaly score of each link segment is correlated with the correlation between the two nodes of that link segment; The traffic mirroring strategy is determined based on the anomaly score of each link segment.

3. The method according to claim 2, characterized in that, Determining the anomaly score of each link segment in the upstream and downstream call chains includes: The anomaly score for this link segment is calculated using the following formula: score=w1·z_score+w2·correlation_shift; Wherein, score is the anomaly score of the link segment, z_score is the standardized deviation value, indicating the degree to which the actual value of the key performance indicator of the abnormal microservice deviates from the normal baseline value, correlation_shift is the standardized correlation value, indicating the change in correlation between the two nodes corresponding to the link segment before and after the time point of the abnormal event within a preset period, w1 is the first weight value, w2 is the second weight value, w1+w2=1, and both w1 and w2 are greater than 0.

4. The method according to claim 3, characterized in that, The step of determining the traffic mirroring strategy based on the anomaly score of each link segment includes: The link segments in the upstream and downstream call chains with the anomalies in the first k positions are taken as the mirror targets, and the mirror ratio corresponding to the mirror targets is obtained, where k is a positive integer; The traffic mirroring strategy is obtained based on the mirroring target and the mirroring ratio.

5. The method according to claim 3, characterized in that, The method further includes: If multiple abnormal microservices depend on the same upstream microservice in the upstream and downstream call chain, the upstream microservice is determined to be faulty. Reduce the anomaly of the link segments between the upstream microservice and the multiple downstream microservices.

6. The method according to claim 1, characterized in that, After obtaining the key performance indicators corresponding to each of the microservices, the method further includes: According to the preset sliding time window and step size, the indicator data corresponding to each type of key performance indicator of each microservice is aggregated to obtain multiple time-series aggregated data of each type of key performance indicator of each microservice. Within each sliding time window, multiple sets of key performance indicators of the same type for the same microservice are aggregated to obtain the time-series aggregated data of the key performance indicators of the same type for the microservice within the sliding time window. The step of identifying abnormal microservices from the plurality of microservices based on the key performance indicators of each microservice includes: Based on multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice, abnormal microservices are identified from the multiple microservices.

7. The method according to claim 6, characterized in that, The step of determining the abnormal microservices among the multiple microservices based on multiple time-series aggregated data of each type of key performance indicator corresponding to each microservice includes: Smoothing is performed on multiple time-series aggregated data of the same type of key performance indicators for the same microservice to obtain the smoothed trend value of the key performance indicator corresponding to the microservice. The abnormal microservices are determined based on multiple time-series aggregated data and corresponding smoothing trend values ​​of each type of key performance indicator corresponding to each microservice. For the same type of key performance indicators in the same microservice, perform the following operations: Determine the residual between each time-series aggregated data point and its corresponding smoothed trend value in multiple time-series aggregated data points of the same type of key performance indicators for the same microservice; Based on the residual between each time-series aggregated data and the corresponding smoothed trend value, and the pre-acquired residual threshold, determine whether the microservice is the abnormal microservice.

8. The method according to claim 7, characterized in that, The smoothing process for multiple time-series aggregated data of the same type of key performance indicators for the same microservice includes: We use exponentially weighted moving averages to smooth multiple time-series aggregated data of the same type of key performance indicators for the same microservice.

9. The method according to claim 7, characterized in that, The residual threshold corresponding to each type of key performance indicator for each microservice is positively correlated with the historical residual of that type of key performance indicator for that microservice; the historical residual is the residual obtained before the current diagnosis.

10. A performance bottleneck diagnostic device, characterized in that, Applied to a microservice system, the microservice system comprising multiple microservices, the device includes: The acquisition unit is used to acquire the indicator data corresponding to the key performance indicators of each microservice; the key performance indicators of the microservice are used to reflect the operational health status of the microservice. The first determining unit is used to determine the abnormal microservice from the plurality of microservices based on the indicator data corresponding to the key performance indicators of each microservice. The second determining unit is used to determine the traffic mirroring strategy corresponding to the abnormal microservice; the traffic mirroring strategy is used to obtain the mirrored traffic corresponding to the abnormal microservice. A traffic acquisition unit is used to acquire target mirrored traffic, wherein the target mirrored traffic is the mirrored traffic acquired by performing the mirroring operation corresponding to the traffic mirroring strategy; A diagnostic unit is used to determine the root cause of the performance bottleneck based on the target mirrored traffic.

Images